STANDARDS AND METHODS FOR FIREARMS RESEARCH

All research must follow some basic standards to be accepted by the

community of scholars in a field—firearms research is no different. These

standards are well known to scientists, although all of them are not achieved

in every research effort and meeting these minimal standards does not

guarantee that the completed research will be judged to be a contribution to

knowledge. These are necessarily minimal standards. Meeting them does

not guarantee a piece of research is sufficiently sound to warrant acceptance

of its findings. Another National Research Council committee (2002)

recently described the scientific process in terms of “six interrelated but not

necessarily ordered, principles of inquiry” (pp. 3-5):

• Pose significant questions that can be investigated empirically.

• Link research to relevant theory.

• Use methods that permit direct investigation of the question.

2For example, the 1986 Firearms Owners Protection Act, also known as the McClure-

Volkmer Act, forbids the federal government from establishing any “system of registration of

firearms, firearm owners, or firearms transactions or distribution.”

• Provide a coherent and explicit chain of reasoning.

• Replicate and generalize across studies.

• Disclose research to encourage professional scrutiny and critique.

While any group of scholars might modify this list, it poses some commonly

accepted standards that our committee used to begin its evaluation of the

literature on firearm violence. In so doing we have sought to ensure that

often controversial research issues are subjected to these minimum standards

for research and to encourage future research in this area to strive for

greater rigor.

The committee also noted that certain research strategies are very prevalent

in firearms research. These include various interrupted time-series approaches

(before-after studies) and the use of case-control techniques. Because

these are so frequently utilized in this area of research, we provide an

analysis of their use. In Appendix D there is a discussion of the difficulties

of before-after type studies, and in Chapter 7 there is one on case-control

designs. For advances to be made in firearm violence research, researchers

must be careful to use these techniques and approaches with due recognition

of their limitations and carefully consider the effect of research design

on findings.

In our analysis of the use of these methods in firearms research, we

found too often that the conclusions reached require the acceptance of

assumptions that are at best implausible. For example, many studies (e.g.,

Duggan, 2001; Kaplan and Geling, 1998; Kleck and Patterson, 1993; Miller

et al., 2002) of the relationship between the access to firearms and firearm

violence are conducted with the state as the unit of analysis (a measure of

the rate of firearm ownership is correlated with the rate of firearm violence).

These results are used to advance the argument that an individual’s

probability of access to firearms explains that individual’s probability of

committing a violent crime with a weapon. While the problems associated

with such cross-level interpretations are well known (the “ecological fallacy”;

that is, inferences about individual behavior cannot be drawn from

aggregate data about a group; Robinson, 1950), these authors and many

who use their work to advance various firearms policies all too frequently

draw inferences that cannot be supported by their analysis. Similarly in

interrupted-time-series designs, the length of the series and the well-known

problems associated with nonexperimental and quasi-experimental designs

(see Campbell and Stanley, 1966) are frequently not given the attention

required for the work to be judged acceptable. Throughout this report we

hold all the research we reviewed to these reasonable standards. Especially

in areas of research in which there is much public controversy, it is vital that

such standards be maintained.

Using the conventional standards of science, we have reviewed the data

and research on firearms and have suggested ways by which these data and

studies can be improved. Our readers will judge how well we have done

this. We hope they will bring to that assessment the same standards of

evidence that we applied in our work.