STANDARDS AND METHODS FOR FIREARMS RESEARCH
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
102 103 104 105 106 107 108 109 110 111 112 113 114 115
All research must follow some basic standards to be accepted by the
community of scholars in a field—firearms research is no different. These
standards are well known to scientists, although all of them are not achieved
in every research effort and meeting these minimal standards does not
guarantee that the completed research will be judged to be a contribution to
knowledge. These are necessarily minimal standards. Meeting them does
not guarantee a piece of research is sufficiently sound to warrant acceptance
of its findings. Another National Research Council committee (2002)
recently described the scientific process in terms of “six interrelated but not
necessarily ordered, principles of inquiry” (pp. 3-5):
• Pose significant questions that can be investigated empirically.
• Link research to relevant theory.
• Use methods that permit direct investigation of the question.
2For example, the 1986 Firearms Owners Protection Act, also known as the McClure-
Volkmer Act, forbids the federal government from establishing any “system of registration of
firearms, firearm owners, or firearms transactions or distribution.”
• Provide a coherent and explicit chain of reasoning.
• Replicate and generalize across studies.
• Disclose research to encourage professional scrutiny and critique.
While any group of scholars might modify this list, it poses some commonly
accepted standards that our committee used to begin its evaluation of the
literature on firearm violence. In so doing we have sought to ensure that
often controversial research issues are subjected to these minimum standards
for research and to encourage future research in this area to strive for
greater rigor.
The committee also noted that certain research strategies are very prevalent
in firearms research. These include various interrupted time-series approaches
(before-after studies) and the use of case-control techniques. Because
these are so frequently utilized in this area of research, we provide an
analysis of their use. In Appendix D there is a discussion of the difficulties
of before-after type studies, and in Chapter 7 there is one on case-control
designs. For advances to be made in firearm violence research, researchers
must be careful to use these techniques and approaches with due recognition
of their limitations and carefully consider the effect of research design
on findings.
In our analysis of the use of these methods in firearms research, we
found too often that the conclusions reached require the acceptance of
assumptions that are at best implausible. For example, many studies (e.g.,
Duggan, 2001; Kaplan and Geling, 1998; Kleck and Patterson, 1993; Miller
et al., 2002) of the relationship between the access to firearms and firearm
violence are conducted with the state as the unit of analysis (a measure of
the rate of firearm ownership is correlated with the rate of firearm violence).
These results are used to advance the argument that an individual’s
probability of access to firearms explains that individual’s probability of
committing a violent crime with a weapon. While the problems associated
with such cross-level interpretations are well known (the “ecological fallacy”;
that is, inferences about individual behavior cannot be drawn from
aggregate data about a group; Robinson, 1950), these authors and many
who use their work to advance various firearms policies all too frequently
draw inferences that cannot be supported by their analysis. Similarly in
interrupted-time-series designs, the length of the series and the well-known
problems associated with nonexperimental and quasi-experimental designs
(see Campbell and Stanley, 1966) are frequently not given the attention
required for the work to be judged acceptable. Throughout this report we
hold all the research we reviewed to these reasonable standards. Especially
in areas of research in which there is much public controversy, it is vital that
such standards be maintained.
Using the conventional standards of science, we have reviewed the data
and research on firearms and have suggested ways by which these data and
studies can be improved. Our readers will judge how well we have done
this. We hope they will bring to that assessment the same standards of
evidence that we applied in our work.