Inaccurate Response
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
102 103 104 105 106 107 108 109 110 111 112 113 114 115
In fact, it is widely thought that inaccurate response biases the estimates
of defensive gun use. Self-report surveys on possibly deviant behaviors
invariably yield some false reports. Responses are miscoded, and respondents
may misunderstand the questions or may not correctly remember
or interpret the event. In addition to these unintentional errors, respondents
may also exaggerate or conceal certain information.
The literature speculates widely on the nature of reporting errors in
the firearms use surveys.5 Some argue that reporting errors cause the
estimates derived from the NCVS to be biased downward.6 Kleck and
Gertz (1995) and Kleck (2001a), for example, speculate that NCVS respondents
doubting the legality of their behaviors or more generally fearing
government intrusion may be inclined to provide false reports to
government officials conducting nonanonymous interviews. Furthermore,
Smith (1997) notes that NCVS respondents are not directly asked about
firearms use but instead are first asked whether they defended themselves,
and then they are asked to describe in what ways. Indirect questions may
lead to incomplete answers.
Others argue that the estimates from the NSDS and other firearms use
surveys are upwardly biased. Cook and Ludwig (1998), Hemenway
(1997a), and Smith (1997), for example, suggest that the firearms use
surveys do not effectively bound events that occur in prior interviews and
thus may result in “memory telescoping.” That is, respondents in the NSDS
are more likely to report events that occurred prior to the observation
window of interest. Furthermore, McDowall et al. (2000) speculate that
preemptive uses recorded in the NSDS but not generally covered in the
NCVS (which focuses on victims) are susceptible to a greater degree of
subjectivity and thus inaccurate reporting.
A number of other general arguments have been raised as to why these
surveys might be inaccurate. Some suggest that respondents may forget or
conceal events that do not lead to adverse outcomes (Kleck and Gertz,
1995; Kleck, 2001a), while others suggest that respondents may exaggerate
or conceal events due to social stigma. Some have even suggested that
respondents may strategically answer questions to somehow influence the
ongoing public debate (Cook et al., 1997). Finally, Hemenway (1997b)
raises what amounts to a mechanical, rather than behavioral, concern
5See Kleck (2001a) for a detailed review of the various hypotheses about inaccurate reporting
in gun use questionnaires.
6Kleck argues that the NCVS is well designed and uses state-of-the-art survey sampling
techniques for measuring victimization, but for exactly those reasons it is not well designed
for measuring defensive gun use.
regarding why the DGU estimates may be generally biased upward. For
any rare event, in fact for any event with less than 50 percent probability,
there are more respondents who can give false positive than false negative
reports. Suppose, for example, in a sample of 1,000 respondents, the true
prevalence rate is 1 percent; that is, 10 respondents used a gun defensively.
Then 990 may provide false positive reports, while only 10 may provide
false negative reports. Even small fractions of false positive reports may
lead to substantial upward biases. Cook et al. (1997) further suggest that
by focusing on victims, the NCVS reduces the scope of the false positive
problem.
Although the rare events problem may be well known and documented
in epidemiological studies of disease, it is uncertain whether this same
phenomena affects inferences on defensive gun uses as well. People may
have reasons to conceal or exaggerate defensive gun uses that may not
apply when studying rare diseases. In fact, what is known about accurate
reporting of other crime-related activities provides some evidence to the
contrary. Validation studies on the accuracy of self-reports of illicit drug
use among arrestees, for example, suggest that for this somewhat rare but
illegal activity, the numbers of false reports of use are far less than the
numbers of false reports of abstinence: self-reports of drug use are biased
downward (Harrison, 1995).
Although theories abound, it is not possible to identify the prevalence
of defensive gun use without knowledge on inaccurate reporting. Kleck and
Gertz (1995) and others suggest that estimates from the NCVS are biased
downward, arguing that respondents are reluctant to reveal information to
government officials, and that indirect questions may yield inaccurate reports.
Hemenway (1997a) and others suggest that estimates from the NSDS
are biased upward, arguing that memory telescoping, self-presentation biases,
and the rare events problem more generally lead the numbers of false
positive reports to substantially exceed the numbers of false negative reports.
It is not known, however, whether Kleck’s, Hemenway’s, or some
other assumptions are correct. The committee is not aware of any factual
basis for drawing conclusions one way or the other about reporting errors.
Nonresponse
While inaccurate response has received a great deal of speculative attention,
the problem of nonresponse has hardly been noticed.7 Nonresponse
is a problem in survey sampling, but it is especially problematic in the
firearms use phone surveys like the NSDS. Although not completely re-
7Both Duncan (2000b) and Hemenway (1997a) recognize the potential problems created
by nonresponse in the firearms use surveys.
vealed by Kleck and Gertz (1995), the response rate in the NSDS appears to
lie somewhere between 14 and 61 percent.8 The response rate in the NCVS
survey is substantially higher, at around 95 percent.
Survey data are uninformative about the behavior of nonrespondents.
Thus, these data do not identify prevalence unless one makes untestable
assumptions about nonrespondents. A simple example illustrates the
problem. Suppose that 1,000 individuals are asked whether they used a
firearm defensively during the past year but that 500 do not respond, so
the nonresponse rate is 50 percent. If 5 of the 500 respondents used guns
defensively during the past year, then the prevalence of defensive gun use
among respondents is 5/500 = 1 percent. However, true prevalence among
the 1,000 surveyed individuals depends on how many of the nonrespondents
used a firearm. If none did, then true prevalence is 5/1,000 =
0.5 percent. If all did, then true prevalence is [(5 + 500)/1,000] = 50.5
percent. If between 0 and 500 nonrespondents used a firearm defensively,
then true prevalence is between 0.5 and 50.5 percent. Thus, in this
example, nonresponse causes true prevalence to be uncertain within a
range of 50 percent.
Prevalence rates can be identified if one makes sufficiently strong assumptions
about the behavior of nonrespondents. In the DGU literature,
nonresponse is assumed to be random, thus implying that that prevalence
among nonrespondents is the same as prevalence among respondents. The
committee is not aware of any empirical evidence that supports the view
that nonresponse is random or, for that matter, evidence to the contrary.
External Validity
A number of scholars have suggested that results from the NSDS and
other firearms use surveys are difficult to reconcile with analogous statistics
8Kleck and Gertz report that 61 percent of contacts with persons for the NSDS resulted in a
completed interview. Presumably, however, there were also many households in the original
sampling scheme that were not contacted. For example, using data from the National Study
of Private Firearms Ownership (NSPFO), a national phone survey designed to elicit information
about firearms ownership and use, Cook and Ludwig (1998) report that 29,917 persons
were part of the original sampling scheme, of which 15,948 were determined to be ineligible
(phones not working, not residential, etc.), 3,268 were determined to be eligible, and the
remaining 10,701 were unknown (e.g., no answer, answering machine, busy, etc.). Of the
3,268 that were known to be eligible, 2,568 provided complete interviews, for a response rate
of 79 percent among contacted households. The 10,701 with unknown eligibility status must
also be accounted for. If none of these households was actually eligible, than the true response
rate would be 79 percent. If, however, all of these are eligible, then the true rate would be 18
percent [2,568/(10,701 + 3,268)]. Thus, the response rate in the NSPOF lies between 18 and
79 percent. If the response rates are consistent across the two surveys, the lower bound
response rate for the NSDS would be 14 percent [ (0.61/0.79)*0.18].
on crime and injury found in other data. For example, Hemenway (1997a)
points out that results from the NSDS imply that firearms are used defensively
in every burglary committed in occupied households and in nearly 60
percent of rapes and sexual assaults committed against persons over 18
years of age; that defensive gun users thought they wounded or killed
offenders in 207,000 incidents, yet only 100,000 people are treated in
emergency rooms for nonfatal firearms injuries; and that hundreds of thousands
of persons almost certainly would have been killed if they had not
used a firearm defensively, implying that nearly all potentially fatal attacks
are successfully defended against (Cook and Ludwig, 1998). Cook and
Ludwig (1998), Hemenway (1997a), and others argue that these and other
similar comparisons lead to “completely implausible conclusions” and go
on to suggest that these inconsistencies “only buttress the presumption of
massive overestimation” of defensive gun uses in the NSDS (Hemenway,
1997a:1444).
Although potentially troubling, the strong conclusion drawn about the
reliability and accuracy of the DGU estimates seems premature. In some
cases, it may be that the comparison statistic is subject to error. The reported
prevalence of rape in the NCVS, for example, is believed to be
biased substantially downward (National Research Council, 2003). More
importantly, however, evidence on the apparent biases of the estimated
incident rates, wounding rates, and counts of averted injuries does not
directly pertain to the accuracy of the DGU estimates. Kleck and Gertz
(1995), in fact, note that victimization estimates drawn using the NSDS, a
survey designed to measure firearms use rather than victimization, are subject
to potential reporting errors in unknown directions. Cook and Ludwig
(1998) find evidence of reporting errors of crime in the firearms use surveys,
with many respondents reporting that crime was involved on one
hand, yet that no crime was involved on the other. Likewise, questions
about whether a respondent thought he wounded or killed the offender and
those eliciting subjective information on what would have happened had a
gun not been used are also subject to substantial reporting biases. As noted
by Kleck and Gertz (1998), respondents may be inclined to “remember
with favor their marksmanship” and may tend to exaggerate the seriousness
of the event.
In addition to invalid response errors, sampling variability may also
play an important role in these conditional comparisons. Inferences drawn
from the relatively small subsamples of persons who report using firearms
defensively (N = 213 in the NSDS) are subject to high degrees of sampling
error. Using data from the National Study of Private Firearms Ownership,
a survey similar to the NSDS, Cook and Ludwig (1998), for example,
estimate that firearms were used defensively in 322,000 rapes (rape, attempted
rape, sexual assault) but report a 95 percent confidence interval of
[12,000 to 632,000].9 The lower bound interval estimate would imply that
firearms are used defensively in less than 3 percent of all rapes and sexual
assaults (Kleck, 2001a).
Replication and Recommendations
As indicated above, the estimated numbers of defensive gun uses found
using the NSDS have been reproduced (i.e., are statistically indistinguishable)
in many other surveys. Kleck (2001a:270) suggests that replication
provides ample evidence of the validity of the findings in the NSDS survey:
The hypothesis that many Americans use guns for self-protection each
year has been repeatedly subjected to empirical test, using the only feasible
method for doing so, survey of representative samples of the populations.
The results of nineteen consecutive surveys unanimously indicate
that each year huge numbers of Americans (700,000 or more) use guns
for self-protection. Further, the more technically sound the survey, the
higher the defensive gun use estimates. The entire body of evidence cannot
be rejected based on the speculation that all surveys share biases that, on
net, cause an over estimation of defensive gun use frequency because,
ignoring fallacious reasoning, there is no empirical evidence to support
this novel theory. At this point, it is fair to say that no intellectually
serious challenge has been mounted to the case for defensive gun use
being very frequent.
Certainly, the numerous surveys reveal some phenomena. In light of the
differences in coverage and potential response errors, however, what exactly
these surveys measure remains uncertain. Ultimately, the committee
found no comfort in numbers: the existing surveys do not resolve the ongoing
questions about response problems and do not change the fact that
different subpopulations are queried. Mere repetition does not eliminate
bias (Rosenbaum, 2001; Hemenway, 1997a).
However, the committee strongly agrees with the main sentiment expressed
by Kleck and others. Evidence from self-reported surveys will invariably
be subject to concerns over reporting errors and other biases. Still, we
can hope to have a greater degree of confidence in the survey results by
relying on replications and survey sampling experiments that serve to effectively
reduce the degree of uncertainty about the true prevalence rate. The
objective of these experiments should be consistency of results in a variety of
sampling designs. Replications and experiments should disrupt aspects of the
original study to check whether the prevalence estimate is reproduced or
altered under different survey designs. Effective replications will vary the
9Kleck and Gertz (1995) do not report confidence intervals for these conditional estimates.
nature of the potential biases in order to explicitly reduce, rather than increase,
the prospects of reproducing the original results (Rosenbaum, 2001).
These ideas are not new to this controversial literature. McDowall et al.
(2000) do exactly this type of experimental evaluation by holding certain
factors constant—namely, the sampling methodology—but varying the content
of the questionnaire. Other similar experiments or replications or both
could be used to vary the nature of memory telescoping, social presentation
bias, and other plausible factors that might influence reporting behaviors.
In fact, Cook and Ludwig (1998), Smith (1997), Kleck (2000), and many
others make numerous recommendations for experiments or replications.
The committee strongly believes that these types of studies can and
should be undertaken. Without reliable information, researchers will continue
to be forced to make unsubstantiated assumptions about the validity
of responses and thus about the prevalence of defensive gun use.
The committee recommends a systematic and rigorous research program
to (1) clearly define and understand what is being measured, (2)
understand inaccurate response in the national use surveys, and (3) develop
methods to reduce reporting errors to the extent possible. Well-established
survey sampling methods can and should be brought to bear to evaluate the
response problems. Understanding response will be useful for not only
explaining the striking gap in DGU estimates but, more importantly, understanding
defensive gun use.
EFFICACY OF SELF-DEFENSE WITH A FIREARM
Accurate measurement on the extent of firearms use is the first step for
beginning a constructive dialogue on how firearms are used in American
society. Invariably, however, attention will turn to the more important and
difficult questions about the consequences of using a firearm for self-defense.
How effective are firearms at preventing injury and crime? Would
gun users have been better off (on average) using alternative defensive
strategies? How does the efficacy of self-defense vary by circumstance (e.g.,
abilities of victim and perpetrator, location of crime, weaponry)?
Answering these questions is essential for evaluating the costs and benefits
of firearms to society. For example, if using a firearm defensively is no
more effective than basic avoidance techniques, then defensive gun use
would have no relative benefit. In contrast, if firearms are more effective at
resisting crime and injury than alternative methods, then civilian ownership
and the use of firearms may play a vital role in the nation’s ability to deter
and fight crime. Of course, the benefits of defensive gun use must ultimately
be weighed against the potential costs that may arise if firearms are involved
in the final stages of violent criminal encounters: defensive gun use
may lead to relatively higher risks of injury and death to victims or offendUSE
OF FIREARMS TO DEFEND AGAINST CRIMINALS 115
ers. Finally, both the benefits and costs must be evaluated within the context
of offender weaponry. If criminals were not armed, would firearms be
more or less useful for protecting potential victims? If the efficacy of selfdefense
depends on the number of firearms in society, then partial equilibrium
analyses that hold offender weaponry fixed may not answer the right
questions.
Empirical Evidence
While the literature on self-defense has been preoccupied with the basic
measurement questions, a handful of studies assess the efficacy of defensive
gun use.10 Using data from the NCVS, Kleck (2001b) compares the probability
of injury and crime by different defensive actions. The results, summarized
in Table 5-2, suggest that respondents who use firearms are less likely to be
injured and lose property than those using other modes of protection. For
example, while the overall rate of injury in robbery is 30.2, only 12.8 percent
of those using a firearm for self-protection were injured. Ziegenhagen and
Brosnan (1985) draw similar conclusions about the efficacy of armed (although
not firearm) resistance when summarizing 13 city victim surveys.
Using a multivariate regression analysis, Kleck and DeLone (1993) confirm
these basic cross-tabular findings.11 Defense with a firearm is associated with
TABLE 5-2 Probability of Injury and Loss Among Victims by Means of
Self-Protection
Robbery Assault
Method Injury Loss Injury
With gun 12.8 15.2 27.9
All self-protection 34.0 52.8 58.1
No self-protection 23.6 83.6 55.2
All incidents 30.2 69.9 57.4
SOURCE: Adapted from Kleck (2001b:289, Table 7.1).
10A number of studies use samples of data collected from crimes reported to police. Police
records are presumed to understate resistance in general and defensive gun use in particular
(Kleck, 2001a; Kleck and DeLone, 1993). More importantly, these surveys cannot reveal
successful forms of resistance that are not reported to the police at all.
11The committee is not aware of other multivariate analyses of the effects of resistance with
a firearm on crime and injury. Researchers have, however, evaluated the effects of armed
resistance. Using data from the NCVS, Kleck and Sayles (1990) conclude that rapes are less
likely to be completed if the victim uses armed resistance. Lizotte (1986) draws similar conclusions
using data from city victim surveys.
fewer completed robberies and less injury. Two forms of self-defense, namely
using force without a weapon and trying to get help or attract attention, are
associated with higher injury rates than taking no self-protective action.
The results suggest interesting associations: victims who use guns defensively
are less likely to be harmed than those using other forms of selfprotection.
Whether these findings reflect underlying causal relationships
or spurious correlations remains uncertain. Much of the existing evidence
reports simple bivariate correlations, without controlling for any confounding
factors. Kleck and DeLone (1993) rely on multivariate linear regression
methods that implicitly assume that firearms use, conditional on observed
factors, is statistically independent of the unobserved factors influencing
the outcomes, as would be the case in a classical randomized experiment.12
Is this exogenous selection assumption reasonable? Arguably, the decisions
to own, carry, and use a firearm for self-defense are very complex, involving
both individual and environmental factors that are related to whether a
crime is attempted, as well as the outcomes of interest.13 The ability of a
person to defend himself or herself, attitudes toward violence and crime,
emotional well-being, and neighborhood characteristics may all influence
whether a person uses a firearm and the resulting injury and crime. Thus, in
general, it is difficult to be confident that the control variables account for
the numerous confounding factors that may result in spurious correlations.
Furthermore, the committee is not aware of any research that considers
whether the finding is robust to a variety of methodological adjustments.
Without an established body of research assessing whether the findings are
robust to the choice of covariates, functional form, and other modeling
assumptions, it is difficult to assess the credibility of the research to date.
The most obvious and fundamental limitation, however, is that the
data on defensive gun uses are, as described above, potentially error ridden.
Without reliable information on the prevalence of defensive gun use, researchers
are forced to make implausible and unsubstantiated assumptions
about the accuracy of self-reported measures of resistance. For example,
Kleck, one of the most vocal critics of DGU estimates derived from the
NCVS, assumes these data are fully accurate when measuring the efficacy
of resistance (Kleck, 2001b; Kleck and DeLone, 1993).
12Kleck and DeLone (1993) account for basic demographic characteristics of the victim
(e.g., race, gender, age, income, and education) and some details on the event (e.g, whether
the offender had a gun).
13Not only does the potential of unobserved factors create biases of unknown magnitude,
but it is also difficult to determine the direction of these biases. If, as suggested by the
National Research Council (1993:266), persons who use firearms were better prepared in
general to defend against crime, then the estimated associations would be biased upward. In
contrast, if firearms are used in more dangerous situations, then the estimated associations
would be biased downward (Kleck, 2001b:292).
The response problems described above, however, cannot be ignored.
To the contrary, these measurement problems may lead to substantial biases
in unknown directions. If, for example, respondents are inclined to
report being victimized when a crime is “successful” but conceal unsuccessful
crimes, the estimated efficacy of resistance will be biased downward. In
contrast, if respondents, concerned about being perceived as inept, are
inclined to report successful forms of resistance but conceal ineffective
forms, the estimated efficacy of self-defense will be biased upward. Without
better information on the nature and extent of response problems, it is
impossible to know whether and how the estimated associations between
defensive gun use, crime, and injury are biased. If, as Kleck and Gertz
(1995) suggest, the NCVS misses over 2 million defensive uses per year,
then biases caused by reporting errors may be substantial.