Inaccurate Response

In fact, it is widely thought that inaccurate response biases the estimates

of defensive gun use. Self-report surveys on possibly deviant behaviors

invariably yield some false reports. Responses are miscoded, and respondents

may misunderstand the questions or may not correctly remember

or interpret the event. In addition to these unintentional errors, respondents

may also exaggerate or conceal certain information.

The literature speculates widely on the nature of reporting errors in

the firearms use surveys.5 Some argue that reporting errors cause the

estimates derived from the NCVS to be biased downward.6 Kleck and

Gertz (1995) and Kleck (2001a), for example, speculate that NCVS respondents

doubting the legality of their behaviors or more generally fearing

government intrusion may be inclined to provide false reports to

government officials conducting nonanonymous interviews. Furthermore,

Smith (1997) notes that NCVS respondents are not directly asked about

firearms use but instead are first asked whether they defended themselves,

and then they are asked to describe in what ways. Indirect questions may

lead to incomplete answers.

Others argue that the estimates from the NSDS and other firearms use

surveys are upwardly biased. Cook and Ludwig (1998), Hemenway

(1997a), and Smith (1997), for example, suggest that the firearms use

surveys do not effectively bound events that occur in prior interviews and

thus may result in “memory telescoping.” That is, respondents in the NSDS

are more likely to report events that occurred prior to the observation

window of interest. Furthermore, McDowall et al. (2000) speculate that

preemptive uses recorded in the NSDS but not generally covered in the

NCVS (which focuses on victims) are susceptible to a greater degree of

subjectivity and thus inaccurate reporting.

A number of other general arguments have been raised as to why these

surveys might be inaccurate. Some suggest that respondents may forget or

conceal events that do not lead to adverse outcomes (Kleck and Gertz,

1995; Kleck, 2001a), while others suggest that respondents may exaggerate

or conceal events due to social stigma. Some have even suggested that

respondents may strategically answer questions to somehow influence the

ongoing public debate (Cook et al., 1997). Finally, Hemenway (1997b)

raises what amounts to a mechanical, rather than behavioral, concern

5See Kleck (2001a) for a detailed review of the various hypotheses about inaccurate reporting

in gun use questionnaires.

6Kleck argues that the NCVS is well designed and uses state-of-the-art survey sampling

techniques for measuring victimization, but for exactly those reasons it is not well designed

for measuring defensive gun use.

regarding why the DGU estimates may be generally biased upward. For

any rare event, in fact for any event with less than 50 percent probability,

there are more respondents who can give false positive than false negative

reports. Suppose, for example, in a sample of 1,000 respondents, the true

prevalence rate is 1 percent; that is, 10 respondents used a gun defensively.

Then 990 may provide false positive reports, while only 10 may provide

false negative reports. Even small fractions of false positive reports may

lead to substantial upward biases. Cook et al. (1997) further suggest that

by focusing on victims, the NCVS reduces the scope of the false positive

problem.

Although the rare events problem may be well known and documented

in epidemiological studies of disease, it is uncertain whether this same

phenomena affects inferences on defensive gun uses as well. People may

have reasons to conceal or exaggerate defensive gun uses that may not

apply when studying rare diseases. In fact, what is known about accurate

reporting of other crime-related activities provides some evidence to the

contrary. Validation studies on the accuracy of self-reports of illicit drug

use among arrestees, for example, suggest that for this somewhat rare but

illegal activity, the numbers of false reports of use are far less than the

numbers of false reports of abstinence: self-reports of drug use are biased

downward (Harrison, 1995).

Although theories abound, it is not possible to identify the prevalence

of defensive gun use without knowledge on inaccurate reporting. Kleck and

Gertz (1995) and others suggest that estimates from the NCVS are biased

downward, arguing that respondents are reluctant to reveal information to

government officials, and that indirect questions may yield inaccurate reports.

Hemenway (1997a) and others suggest that estimates from the NSDS

are biased upward, arguing that memory telescoping, self-presentation biases,

and the rare events problem more generally lead the numbers of false

positive reports to substantially exceed the numbers of false negative reports.

It is not known, however, whether Kleck’s, Hemenway’s, or some

other assumptions are correct. The committee is not aware of any factual

basis for drawing conclusions one way or the other about reporting errors.

Nonresponse

While inaccurate response has received a great deal of speculative attention,

the problem of nonresponse has hardly been noticed.7 Nonresponse

is a problem in survey sampling, but it is especially problematic in the

firearms use phone surveys like the NSDS. Although not completely re-

7Both Duncan (2000b) and Hemenway (1997a) recognize the potential problems created

by nonresponse in the firearms use surveys.

vealed by Kleck and Gertz (1995), the response rate in the NSDS appears to

lie somewhere between 14 and 61 percent.8 The response rate in the NCVS

survey is substantially higher, at around 95 percent.

Survey data are uninformative about the behavior of nonrespondents.

Thus, these data do not identify prevalence unless one makes untestable

assumptions about nonrespondents. A simple example illustrates the

problem. Suppose that 1,000 individuals are asked whether they used a

firearm defensively during the past year but that 500 do not respond, so

the nonresponse rate is 50 percent. If 5 of the 500 respondents used guns

defensively during the past year, then the prevalence of defensive gun use

among respondents is 5/500 = 1 percent. However, true prevalence among

the 1,000 surveyed individuals depends on how many of the nonrespondents

used a firearm. If none did, then true prevalence is 5/1,000 =

0.5 percent. If all did, then true prevalence is [(5 + 500)/1,000] = 50.5

percent. If between 0 and 500 nonrespondents used a firearm defensively,

then true prevalence is between 0.5 and 50.5 percent. Thus, in this

example, nonresponse causes true prevalence to be uncertain within a

range of 50 percent.

Prevalence rates can be identified if one makes sufficiently strong assumptions

about the behavior of nonrespondents. In the DGU literature,

nonresponse is assumed to be random, thus implying that that prevalence

among nonrespondents is the same as prevalence among respondents. The

committee is not aware of any empirical evidence that supports the view

that nonresponse is random or, for that matter, evidence to the contrary.

External Validity

A number of scholars have suggested that results from the NSDS and

other firearms use surveys are difficult to reconcile with analogous statistics

8Kleck and Gertz report that 61 percent of contacts with persons for the NSDS resulted in a

completed interview. Presumably, however, there were also many households in the original

sampling scheme that were not contacted. For example, using data from the National Study

of Private Firearms Ownership (NSPFO), a national phone survey designed to elicit information

about firearms ownership and use, Cook and Ludwig (1998) report that 29,917 persons

were part of the original sampling scheme, of which 15,948 were determined to be ineligible

(phones not working, not residential, etc.), 3,268 were determined to be eligible, and the

remaining 10,701 were unknown (e.g., no answer, answering machine, busy, etc.). Of the

3,268 that were known to be eligible, 2,568 provided complete interviews, for a response rate

of 79 percent among contacted households. The 10,701 with unknown eligibility status must

also be accounted for. If none of these households was actually eligible, than the true response

rate would be 79 percent. If, however, all of these are eligible, then the true rate would be 18

percent [2,568/(10,701 + 3,268)]. Thus, the response rate in the NSPOF lies between 18 and

79 percent. If the response rates are consistent across the two surveys, the lower bound

response rate for the NSDS would be 14 percent [ (0.61/0.79)*0.18].

on crime and injury found in other data. For example, Hemenway (1997a)

points out that results from the NSDS imply that firearms are used defensively

in every burglary committed in occupied households and in nearly 60

percent of rapes and sexual assaults committed against persons over 18

years of age; that defensive gun users thought they wounded or killed

offenders in 207,000 incidents, yet only 100,000 people are treated in

emergency rooms for nonfatal firearms injuries; and that hundreds of thousands

of persons almost certainly would have been killed if they had not

used a firearm defensively, implying that nearly all potentially fatal attacks

are successfully defended against (Cook and Ludwig, 1998). Cook and

Ludwig (1998), Hemenway (1997a), and others argue that these and other

similar comparisons lead to “completely implausible conclusions” and go

on to suggest that these inconsistencies “only buttress the presumption of

massive overestimation” of defensive gun uses in the NSDS (Hemenway,

1997a:1444).

Although potentially troubling, the strong conclusion drawn about the

reliability and accuracy of the DGU estimates seems premature. In some

cases, it may be that the comparison statistic is subject to error. The reported

prevalence of rape in the NCVS, for example, is believed to be

biased substantially downward (National Research Council, 2003). More

importantly, however, evidence on the apparent biases of the estimated

incident rates, wounding rates, and counts of averted injuries does not

directly pertain to the accuracy of the DGU estimates. Kleck and Gertz

(1995), in fact, note that victimization estimates drawn using the NSDS, a

survey designed to measure firearms use rather than victimization, are subject

to potential reporting errors in unknown directions. Cook and Ludwig

(1998) find evidence of reporting errors of crime in the firearms use surveys,

with many respondents reporting that crime was involved on one

hand, yet that no crime was involved on the other. Likewise, questions

about whether a respondent thought he wounded or killed the offender and

those eliciting subjective information on what would have happened had a

gun not been used are also subject to substantial reporting biases. As noted

by Kleck and Gertz (1998), respondents may be inclined to “remember

with favor their marksmanship” and may tend to exaggerate the seriousness

of the event.

In addition to invalid response errors, sampling variability may also

play an important role in these conditional comparisons. Inferences drawn

from the relatively small subsamples of persons who report using firearms

defensively (N = 213 in the NSDS) are subject to high degrees of sampling

error. Using data from the National Study of Private Firearms Ownership,

a survey similar to the NSDS, Cook and Ludwig (1998), for example,

estimate that firearms were used defensively in 322,000 rapes (rape, attempted

rape, sexual assault) but report a 95 percent confidence interval of

 [12,000 to 632,000].9 The lower bound interval estimate would imply that

firearms are used defensively in less than 3 percent of all rapes and sexual

assaults (Kleck, 2001a).

Replication and Recommendations

As indicated above, the estimated numbers of defensive gun uses found

using the NSDS have been reproduced (i.e., are statistically indistinguishable)

in many other surveys. Kleck (2001a:270) suggests that replication

provides ample evidence of the validity of the findings in the NSDS survey:

The hypothesis that many Americans use guns for self-protection each

year has been repeatedly subjected to empirical test, using the only feasible

method for doing so, survey of representative samples of the populations.

The results of nineteen consecutive surveys unanimously indicate

that each year huge numbers of Americans (700,000 or more) use guns

for self-protection. Further, the more technically sound the survey, the

higher the defensive gun use estimates. The entire body of evidence cannot

be rejected based on the speculation that all surveys share biases that, on

net, cause an over estimation of defensive gun use frequency because,

ignoring fallacious reasoning, there is no empirical evidence to support

this novel theory. At this point, it is fair to say that no intellectually

serious challenge has been mounted to the case for defensive gun use

being very frequent.

Certainly, the numerous surveys reveal some phenomena. In light of the

differences in coverage and potential response errors, however, what exactly

these surveys measure remains uncertain. Ultimately, the committee

found no comfort in numbers: the existing surveys do not resolve the ongoing

questions about response problems and do not change the fact that

different subpopulations are queried. Mere repetition does not eliminate

bias (Rosenbaum, 2001; Hemenway, 1997a).

However, the committee strongly agrees with the main sentiment expressed

by Kleck and others. Evidence from self-reported surveys will invariably

be subject to concerns over reporting errors and other biases. Still, we

can hope to have a greater degree of confidence in the survey results by

relying on replications and survey sampling experiments that serve to effectively

reduce the degree of uncertainty about the true prevalence rate. The

objective of these experiments should be consistency of results in a variety of

sampling designs. Replications and experiments should disrupt aspects of the

original study to check whether the prevalence estimate is reproduced or

altered under different survey designs. Effective replications will vary the

9Kleck and Gertz (1995) do not report confidence intervals for these conditional estimates.

nature of the potential biases in order to explicitly reduce, rather than increase,

the prospects of reproducing the original results (Rosenbaum, 2001).

These ideas are not new to this controversial literature. McDowall et al.

(2000) do exactly this type of experimental evaluation by holding certain

factors constant—namely, the sampling methodology—but varying the content

of the questionnaire. Other similar experiments or replications or both

could be used to vary the nature of memory telescoping, social presentation

bias, and other plausible factors that might influence reporting behaviors.

In fact, Cook and Ludwig (1998), Smith (1997), Kleck (2000), and many

others make numerous recommendations for experiments or replications.

The committee strongly believes that these types of studies can and

should be undertaken. Without reliable information, researchers will continue

to be forced to make unsubstantiated assumptions about the validity

of responses and thus about the prevalence of defensive gun use.

The committee recommends a systematic and rigorous research program

to (1) clearly define and understand what is being measured, (2)

understand inaccurate response in the national use surveys, and (3) develop

methods to reduce reporting errors to the extent possible. Well-established

survey sampling methods can and should be brought to bear to evaluate the

response problems. Understanding response will be useful for not only

explaining the striking gap in DGU estimates but, more importantly, understanding

defensive gun use.

EFFICACY OF SELF-DEFENSE WITH A FIREARM

Accurate measurement on the extent of firearms use is the first step for

beginning a constructive dialogue on how firearms are used in American

society. Invariably, however, attention will turn to the more important and

difficult questions about the consequences of using a firearm for self-defense.

How effective are firearms at preventing injury and crime? Would

gun users have been better off (on average) using alternative defensive

strategies? How does the efficacy of self-defense vary by circumstance (e.g.,

abilities of victim and perpetrator, location of crime, weaponry)?

Answering these questions is essential for evaluating the costs and benefits

of firearms to society. For example, if using a firearm defensively is no

more effective than basic avoidance techniques, then defensive gun use

would have no relative benefit. In contrast, if firearms are more effective at

resisting crime and injury than alternative methods, then civilian ownership

and the use of firearms may play a vital role in the nation’s ability to deter

and fight crime. Of course, the benefits of defensive gun use must ultimately

be weighed against the potential costs that may arise if firearms are involved

in the final stages of violent criminal encounters: defensive gun use

may lead to relatively higher risks of injury and death to victims or offendUSE

OF FIREARMS TO DEFEND AGAINST CRIMINALS 115

ers. Finally, both the benefits and costs must be evaluated within the context

of offender weaponry. If criminals were not armed, would firearms be

more or less useful for protecting potential victims? If the efficacy of selfdefense

depends on the number of firearms in society, then partial equilibrium

analyses that hold offender weaponry fixed may not answer the right

questions.

Empirical Evidence

While the literature on self-defense has been preoccupied with the basic

measurement questions, a handful of studies assess the efficacy of defensive

gun use.10 Using data from the NCVS, Kleck (2001b) compares the probability

of injury and crime by different defensive actions. The results, summarized

in Table 5-2, suggest that respondents who use firearms are less likely to be

injured and lose property than those using other modes of protection. For

example, while the overall rate of injury in robbery is 30.2, only 12.8 percent

of those using a firearm for self-protection were injured. Ziegenhagen and

Brosnan (1985) draw similar conclusions about the efficacy of armed (although

not firearm) resistance when summarizing 13 city victim surveys.

Using a multivariate regression analysis, Kleck and DeLone (1993) confirm

these basic cross-tabular findings.11 Defense with a firearm is associated with

TABLE 5-2 Probability of Injury and Loss Among Victims by Means of

Self-Protection

Robbery Assault

Method Injury Loss Injury

With gun 12.8 15.2 27.9

All self-protection 34.0 52.8 58.1

No self-protection 23.6 83.6 55.2

All incidents 30.2 69.9 57.4

SOURCE: Adapted from Kleck (2001b:289, Table 7.1).

10A number of studies use samples of data collected from crimes reported to police. Police

records are presumed to understate resistance in general and defensive gun use in particular

(Kleck, 2001a; Kleck and DeLone, 1993). More importantly, these surveys cannot reveal

successful forms of resistance that are not reported to the police at all.

11The committee is not aware of other multivariate analyses of the effects of resistance with

a firearm on crime and injury. Researchers have, however, evaluated the effects of armed

resistance. Using data from the NCVS, Kleck and Sayles (1990) conclude that rapes are less

likely to be completed if the victim uses armed resistance. Lizotte (1986) draws similar conclusions

using data from city victim surveys.

fewer completed robberies and less injury. Two forms of self-defense, namely

using force without a weapon and trying to get help or attract attention, are

associated with higher injury rates than taking no self-protective action.

The results suggest interesting associations: victims who use guns defensively

are less likely to be harmed than those using other forms of selfprotection.

Whether these findings reflect underlying causal relationships

or spurious correlations remains uncertain. Much of the existing evidence

reports simple bivariate correlations, without controlling for any confounding

factors. Kleck and DeLone (1993) rely on multivariate linear regression

methods that implicitly assume that firearms use, conditional on observed

factors, is statistically independent of the unobserved factors influencing

the outcomes, as would be the case in a classical randomized experiment.12

Is this exogenous selection assumption reasonable? Arguably, the decisions

to own, carry, and use a firearm for self-defense are very complex, involving

both individual and environmental factors that are related to whether a

crime is attempted, as well as the outcomes of interest.13 The ability of a

person to defend himself or herself, attitudes toward violence and crime,

emotional well-being, and neighborhood characteristics may all influence

whether a person uses a firearm and the resulting injury and crime. Thus, in

general, it is difficult to be confident that the control variables account for

the numerous confounding factors that may result in spurious correlations.

Furthermore, the committee is not aware of any research that considers

whether the finding is robust to a variety of methodological adjustments.

Without an established body of research assessing whether the findings are

robust to the choice of covariates, functional form, and other modeling

assumptions, it is difficult to assess the credibility of the research to date.

The most obvious and fundamental limitation, however, is that the

data on defensive gun uses are, as described above, potentially error ridden.

Without reliable information on the prevalence of defensive gun use, researchers

are forced to make implausible and unsubstantiated assumptions

about the accuracy of self-reported measures of resistance. For example,

Kleck, one of the most vocal critics of DGU estimates derived from the

NCVS, assumes these data are fully accurate when measuring the efficacy

of resistance (Kleck, 2001b; Kleck and DeLone, 1993).

12Kleck and DeLone (1993) account for basic demographic characteristics of the victim

(e.g., race, gender, age, income, and education) and some details on the event (e.g, whether

the offender had a gun).

13Not only does the potential of unobserved factors create biases of unknown magnitude,

but it is also difficult to determine the direction of these biases. If, as suggested by the

National Research Council (1993:266), persons who use firearms were better prepared in

general to defend against crime, then the estimated associations would be biased upward. In

contrast, if firearms are used in more dangerous situations, then the estimated associations

would be biased downward (Kleck, 2001b:292).

The response problems described above, however, cannot be ignored.

To the contrary, these measurement problems may lead to substantial biases

in unknown directions. If, for example, respondents are inclined to

report being victimized when a crime is “successful” but conceal unsuccessful

crimes, the estimated efficacy of resistance will be biased downward. In

contrast, if respondents, concerned about being perceived as inept, are

inclined to report successful forms of resistance but conceal ineffective

forms, the estimated efficacy of self-defense will be biased upward. Without

better information on the nature and extent of response problems, it is

impossible to know whether and how the estimated associations between

defensive gun use, crime, and injury are biased. If, as Kleck and Gertz

(1995) suggest, the NCVS misses over 2 million defensive uses per year,

then biases caused by reporting errors may be substantial.