Takeaways From a Refreshing Rejection of Junk Science

If you’ve ever watched a procedural crime drama or your local news, you’ve probably seen those little numbered tents marking the location of evidence at a crime scene.  After a shooting, those evidence markers often note the location of bullet casings that were left behind.  Many defendants (real and fictional) have been convicted, in part, based on testimony from so-called firearms and toolmark experts who told the jury that a bullet casing recovered from the crime scene “matched” a gun associated with the accused.  But, based on a careful and thorough opinion by a D.C. judge, this familiar scenario may require some serious rewrites.

Last fall, in United States v. Tibbs, 2016-CF1-19431, D.C. Superior Court judge Todd E. Edelman ruled that the prosecution’s firearms and toolmark expert could not testify that a bullet casing and gun “matched”, or that the gun in question was the “source” of the bullet casing—what is known as a “source attribution” statement.  Rather, the expert would only be allowed to testify that the firearm “cannot be excluded” as the source of the casing.  In other words: “This gun might have fired that bullet, but I cannot say for sure.”  In a criminal case, where guilt must be proven beyond a reasonable doubt, this distinction is of enormous significance.

Firearms and toolmark identification is an area of forensics based on the theory that tools used in the manufacture of firearms leave distinct markings, called “individual characteristics,” on the internal components of the gun, such as the barrel and firing pin.  When a gun is fired, the ammunition comes into contact with the internal components of the gun, and the tool markings are then transferred to the softer components of the bullet.  Experts in this field have often opined that no two firearms will produce the same microscopic markings on bullets and casings, and thus they believe they can use the toolmarks left on spent ammunition to positively identify the gun that fired it.

In the Tibbs case, police recovered a .40 caliber bullet casing from the scene of a homicide.  The government alleged that shortly after the shooting, police observed Mr. Tibbs discarding a .40 caliber semi-automatic pistol.  The police recovered that gun and the government’s firearms and toolmark expert compared the gun to the casing from the scene.  The expert’s report concluded that the casing had been fired by the recovered gun based on microscopic examination of “breechface marks and firing pin aperture marks.”  The defense moved to exclude the expert’s opinion under Daubert and Rule of Evidence 702, arguing that the expert’s conclusion was not supported by reliable principles and methods.

In its landmark Daubert ruling, nearly 30 years ago, the Supreme Court designated trial judges as “gatekeepers” who must “ensure that any and all scientific testimony or evidence admitted is not only relevant, but reliable” Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579, 589 (1993).  The elements of the Daubert inquiry are now reflected in Federal Rule of Evidence 702, which requires that expert testimony must be “the product of reliable principles and methods.”

The Tibbs defense faced a steep uphill battle.  Although the general scientific community has long questioned the accuracy and reliability of this evidence, for example here and here, testimony by firearms and toolmark identification experts routinely admitted in state and federal criminal courts all across the country—frequently over defendants’ objections—both before and after Daubert.  In 2019, a federal court in Nevada observed that no federal court has found the method of firearms and toolmark examination used in Mr. Tibbs’s case to be unreliable.  A few courts have imposed some degree of restriction on firearms and toolmark identification testimony, but this has been the exception rather than the rule.

Judges, understandably, have great respect for precedent and the determinations of their fellow jurists.  Most trial judges lack any scientific training, and when asked to make a call as to whether a forensic opinion is sufficiently reliable to be admitted into evidence, they often defer to what other courts did in (apparently) similar circumstances.

What makes Judge Edelman’s opinion so remarkable is that he resisted the temptation to fall in line.  He first stressed that there is no “grandfathering” provision in Rule 702—evidence should not be admitted merely because other courts have done so in the past.  He observed: “[T]he case law in this area follows a pattern in which holdings supported by limited analysis are nonetheless subsequently deferred to by one court after another.  This pattern creates the appearance of an avalanche of authority; on closer examination, however, these precedents ultimately stand on a flimsy factual foundation.”  Judge Edelman was also troubled that most of the published cases addressing firearms and toolmark identification testimony “involved no hearing on the admissibility of the evidence or only a cursory analysis of the relevant issues.”  Judge Edelman took the opposite approach, concluding that due to the lack of persuasive authority on the issue, in D.C. or elsewhere, he would treat the question as one of first impression.

Judge Edelman held a multi-day Daubert hearing, during which multiple expert witnesses testified for each side.  These witnesses addressed not only the evidence in the Tibbs case, but also the strengths and weaknesses of the published studies of firearms and toolmark identification, including the studies’ design and methodology.  Ultimately, Judge Edelman concluded that the government expert’s proffered “source attribution” statement was not admissible, largely for three reasons.  His analysis of each of these points provides valuable guidance for defense attorneys seeking to keep out unreliable firearms and toolmark identification opinions in their own case.

1. Lack of An Established Error Rate. Courts have generally accepted that firearms toolmark identification analysis has a very low error rate. Judge Edelman disagreed for three reasons.  First, basic design flaws in the studies, such as failure to account for possible test-taking biases and non-representative self-selection of study volunteers, undermine the value of their results.  Second, all but two of the published studies in the field use “closed” or “set-based” structure, which limit the reliability of the reported error rate.  Finally, many studies did not count “inconclusive” findings by the examiner in their error rate, which potentially underestimated the actual rate of error by as much as 34%.

These critiques of study design and execution are technical and complicated. Thus, they are much more likely to be effective in a pre-trial Daubert challenge than as lines of cross-examination at trial.  Defense attorneys would be wise to engage experts in study design and research methods early in the case.

2.Lack of an Objective Standard for Identification. The operative standard for firearms and toolmark examiners is the Association of Firearms and Toolmark Examiners (“AFTE”) “theory of identification,” which permits “opinions of common origin to be made when the unique surface of two toolmarks are in ‘sufficient agreement.’” Judge Edelman found that the AFTE standard was not only subjective (which is not automatically disqualifying under Daubert), but also had no defined criteria to guide examiners’ judgment.  In that sense, the AFTE theory of identification is more subjective than other types of subjective opinions that are admissible under Daubert because the expert’s opinion is bounded by objective criteria (for example, real estate assessments or medical standards of care).  Judge Edelman found that this factor weighed against the admission of the firearms and toolmark identification evidence because the “so-called standard…is merely unconstrained subjectivity masquerading as objectivity.”

This point is more straightforward than the first and more easily grasped by jurors.  Although defense attorneys should emphasize this during a pre-trial Daubert challenge, if the expert’s opinion is ultimately admitted, attorneys should make the lack of objective criteria a focal point of their cross-examination and argue that the expert’s opinion should therefore be given little weight.

3.Lack of Acceptance in the Wider Scientific Community. All previously published opinions about the AFTE method have concluded that it is generally accepted in the relevant scientific community. However, those courts usually limit their understanding of the “relevant” scientific community to the specific niche population of firearms and toolmark examiners themselves.

This is problematic because practitioners of any discipline will tend to accept its validity if they are using the challenged techniques and methods in their own work.  Judge Edelman astutely pointed out that “if Daubert…[and] Rule 702 are to have any meaning at all, courts must not confine the relevant scientific community to the specific group of practitioners dedicated to the validity of the theory—in other words, to those whose professional standing and financial livelihoods depend upon the challenged discipline.”  Accepting the AFTE method because firearms and toolmark examiners believe in it would be like assessing the efficacy of essential oils by consulting the people who are selling them on Facebook.

Judge Edelman cited scientific reports published independent of AFTE to show that the wider scientific community does not necessarily accept the theory that an examiner can reliably conclude that a particular firearm fired a particular casing or bullet.  Defense attorneys would be smart to retain experts from outside the discipline of criminal forensics to interpret these kinds of publications and effectively challenge the prevailing wisdom about the credibility that should be afforded to firearms and toolmark identifications.

* * *

In this day and age, when science seems under attack and is too often dismissed as “fake news”, Judge Edelman’s opinion is a rare victory for science in the courtroom.  Defense attorneys should make every effort to ensure that other courts take notice.