Abstract
Purpose of Review
Firearm policies have the potential to alleviate the public health burden of firearm violence, yet it is unclear which policies are effective. The current review aims to summarize studies that use synthetic control methods to overcome previous methodological limitations when examining the impacts of firearm policies.
Recent Findings
Evidence from studies using synthetic control methods find compelling evidence that purchasing licensing laws for all individuals (e.g., permit-to-purchase) have a preventive effect on firearm deaths. Otherwise, the effects of other firearm policies targeting firearm availability, ownership, sales, and use varied across studies and contexts.
Summary
Synthetic control evaluations find heterogenous effects of firearm policies, suggesting that previous inconsistent findings might reflect their varying impacts across regions rather than methodological limitations alone. Future research should aim to exploit the complementary biases of synthetic control methods to triangulate evidence across evaluation approaches and understand why firearm policies have differential impacts.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Worldwide, more than 250,000 people die from firearm injuries each year [1]. These deaths are predominantly caused by lethal violence as homicide and suicide account for over 90% of firearm deaths. Firearm violence is increasingly recognized as a critical global public health problem, placing serious social and economic costs on societies in addition to the most serious cost of loss of life [1–3]. The burden of firearm mortality varies both between and within countries. Brazil and the United States (US) account for 32% of all firearm deaths globally [2], and rates are 12 times higher in the US compared to other high-income countries [4]. There is also substantial variation within the US, with increasing trends in some states (e.g., Missouri) and decreasing in others (e.g., California) [5]. Thus, firearm deaths are shaped by complex factors that differ by region, including policies and laws that governments use to regulate access to, and use of, firearms [6].
A growing number of studies examine the wide range of firearm policies that often aim to reduce avoidable firearm deaths (Table 1), particularly in the US. While there remains a lack of clarity on which policies effectively prevent firearm deaths [6–10] and which increase deaths [11], some consistent findings have emerged. First, the simultaneous adoption of multiple laws that target different elements of firearms regulation has led to reductions in firearm-related deaths in some countries [6], such as Australia [12, 13]. Second, more restrictive firearm policies are generally associated with decreased firearm deaths, as are stronger background checks and permit-to-purchase laws [6, 14]. Third, limitations in the evidence can be attributed in part to difficulties in providing valid estimates of the impacts of firearm policies [15].
Firearm laws are heterogeneous in formulation, implementation, and enforcement [16]. Comparing firearm laws across regions and periods can be difficult because law changes are not always immediately implemented or enforced, and public response may be delayed [8]. Other challenges include difficulties in identifying appropriate control groups and fulfilling the modeling assumptions underlying the common analytical techniques used to evaluate firearm policies, such as interrupted time series (ITS) and difference-in-difference (DiD) designs [17, 18]. Moreover, a recent simulation study showed that the existing literature suffers from sensitivity to modeling specifications and commonly used modeling approaches in gun policy research have high rates of false positives [15, 19]. Subsequent recommendations have called for future research to use alternative evaluation approaches to avoid further hindrance from the limited power of traditional significance testing methods [15].
Synthetic control methodology (SCM) offers an alternative to traditional evaluation methods and can overcome key limitations in the firearm policy literature: (i) the lack of comparable intervention and control groups, and (ii) fragility of findings to modeling specifications. SCM is a data-driven technique for evaluating the impacts of population-level interventions [20–23], such as firearm policies, and has been described as “the most important innovation in the evaluation literature in the last fifteen years” [24]. The method moves away from traditional significance testing methods by using a data-driven algorithm to identify an optimal weighted control unit—a “synthetic control”—based on pre-intervention data from available control units, referred to as the “donor pool” [20–22]. The method aims to construct a well-matched counterfactual for between-group studies by minimizing systematic differences between the intervention and control units’ pre-intervention outcome trends (and covariates) [25]. The approach uses optimization to determine the weights for each potential control and variable importance from the covariates to construct the synthetic control unit. Any post-intervention differences between the intervention and synthetic control unit should be attributable to the intervention itself, provided the synthetic control matches the intervention unit on a long series of pre-intervention outcomes, and no other firearm policies are implemented during the study period [21]. SCM neither relies on traditional significance testing nor requires strict modeling assumptions about the shape of the intervention effect over time, and thus addresses some of the current limitations in the firearm policy literature. The method was initially developed for interventions implemented in a single unit (e.g., the introduction of a national firearm law) [20, 21] but has since been generalized to more complex data structures (e.g., multiple intervention units with staggered adoption) [26, 27]. Given the potential and relative infancy of the method, this review aims to summarize and critically evaluate the evidence SCM has generated on firearm policy effects.
Methods
We systematically searched the literature from January 2015 to November 2021, when the searches were performed. We searched for published studies in 5 databases: Embase, Pubmed, Proquest, PsycINFO, and Ovid Medline. Following a scoping search for empirical studies that used synthetic control methodology to examine firearm policies, we searched for articles that contained the following terms in the title and/or abstract: (“gun” or “guns” or handgun* or firearm*) AND (“legislation” or “law” or “laws” or “statute” or “statutes” or “regulation” or “policy” or “jurisprudence”) AND synthetic control*. We also conducted directed searches of Google Scholar and the RAND Corporation, examined bibliographies of relevant articles, and included papers previously known to the authors.
We identified 88 articles from databases, 1 article from the RAND Corporation [31], and 4 articles previously known to the authors [23, 32–33]. Articles were de-duplicated, screened, and evaluated for relevance. For this narrative review, we excluded 3 theses and dissertations [34–36]. This resulted in 17 empirical studies and two commentaries on the included empirical papers [37, 38]. Four methodological papers were also identified but excluded [23, 31, 32, 33].
Methodological Observations
Included studies that used synthetic control methodology (SCM) to examine firearm policies are summarized in Table 2. Among the 17 studies, only one was conducted outside of the US (in Australia) [39••]. This was also the only study to evaluate a national firearm policy [39••]. All 17 studies applied the original single-intervention-unit SCM [20, 21], irrespective of the number of intervention units (range 1–33). Studies with more than one intervention unit were exclusively evaluations of firearm policies across US states. The most common inference method was the in-place placebo test [21] (used in 16 of the 17 studies). Four studies also used unverified inference methods, including conventional t-tests to test for differences in the treated and synthetic control outcomes [40•, 41, 42] and segmented regression to identify structural breaks in the synthetic control outcomes [32]. Only a handful of academics published papers using SCM to examine firearm policies, often authoring more than one included study. For example, some authors appeared in a quarter of all studies identified and one academic co-authored seven included studies. This is likely due to SCM not yet being widely understood and applied in public health research [23, 25], and thus only a handful of academics are well versed in both methodology and topic.
SCM was generally applied with methodological rigor. Evaluations were generally sensitive to contamination effects and appropriately excluded controls that may have been influenced by the intervention from the donor pool. For example, all 16 US studies excluded states (i.e., controls) from the donor pool that had similar policies enacted, or not enacted if evaluating a policy repeal, during the study period (Table 2). The only non-US study evaluating Australia’s 1996 Gun Buyback program did not specifically exclude other countries (i.e., controls) with gun buyback programs but did restrict the donor pool to countries with population sizes greater than 500,000 [39••]. Four additional studies excluded controls that had sparse and/or missing data [39••, 41, 46, 49•]. A few studies conducted sensitivity analyses to check the robustness of findings to donor pool restrictions [43•, 44].
All but one study matched on and adjusted for a comprehensive list of covariates in the synthetic control models (Table 2). Most covariates were selected and included as they were state-level characteristics hypothesized and/or evidenced to predict the outcome(s) under evaluation, namely homicide and suicide. Common covariates thus included measures of population sociodemographics (age, sex, race, ethnic distribution), geography (population density, urbanicity, metropolitan statistical areas), and economics (e.g., poverty, unemployment rate, median household income, Gini coefficient, high school education), as well as alcohol consumption and proxy measures of gun availability. Less common covariates included rate of religious adherence [51••], number of law enforcement employees [51••], and violent crime rates [44, 46, 47•, 48]. One study selected covariates which further improved synthetic control model fit [43•], and three studies did not fully explain why covariates were included [46, 50, 52•]. The only study to not directly adjust for covariates was the national evaluation of Australia’s 1996 Gun Buyback program, which described common covariates but did not include them in the synthetic control models due to unbalanced (missing) data among control countries in the donor pool [39••]. The authors’ further argued that, because the causal process of homicide is unknown, it is preferential to match on the pre-intervention trend in the outcome series rather than covariates [39••].
Despite this, there was often poor information on included matching variables, including if they were time constant or varying, their time intervals and period, and their variable importance weights (determined by the optimization). Several studies did not achieve a good match on pre-intervention trends, which we identified to be mainly due to volatile and/or sparse outcomes. Although most studies tried to minimize related biases by including robustness checks or alternative evaluation methods [43•, 44, 49•], other studies did not comprehensively address poor synthetic control fit [40•].
Empirical Evidence
Policies Targeting Availability
We identified only one study that used SCM to evaluate policies that target firearm availability. The study examined the nationwide rollout of Australia’s Gun Buyback Program, which ran for 12 months from October 1996 to September 1997 and was estimated to retrieve around 650,000 guns [39••]. Australia was compared to a weighted control unit derived from 28 WHO-reporting countries with similar population sizes and minimal missing data. The study found significant reductions in homicide rates and no effect on their negative control outcome of motor vehicle fatalities following the 1996 gun buyback program [39••]. However, there was also no significant effect on suicide rates. The authors argued that this absence of effect was due to the 1996 program being limited to military-style assault rifles and shotguns.
Policies Targeting Ownership
There were several evaluations of different policies that targeted firearm ownership. This included four studies evaluating purchasing licenses (permit-to-purchase laws) [45, 49•, 50, 51••], three studies evaluating restrictions on individuals (firearm seizures and misdemeanor violence prohibition laws) [41, 43•, 47•], and one study evaluating safe storage (child access prevention laws) [40•]. An additional study evaluated Massachusetts Gun Law Reform in 1998, which made unprecedented changes to state firearm laws through 23 legislative changes [52•]. These primarily placed restrictions on gun ownership through individual restrictions (minimum age, misdemeanor violence prohibition), firearm safety training, safety storage, and reporting lost and stolen firearms requirements. This evaluation found a reduction in total firearm suicide rates for several years and a sustained reduction in firearm suicide rates after placing additional restrictions on gun ownership [52•]. Overall, there was consistent evidence of a preventive effect of purchasing licenses on firearm mortalities but inconsistent evidence on the effectiveness of restricting ownership for specific individuals and safe storage requirements (see Fig. 1).
Purchasing Licenses
Permit-to-purchase (PTP) law requires a prospective firearm buyer to apply for a license directly to a government authority (e.g., local law enforcement agency) that vets the application and initiates a background check. Four studies used SCM to evaluate the impacts of PTP laws on firearm suicide and homicide rates. Studies consistently found reductions in firearm suicide and homicide rates following the introduction of PTP laws and increases after the law was repealed [45, 49•, 50, 51••]. The introduction of PTP laws was associated with a 28–40% decrease in firearm homicides and a 15–28% decrease in firearm suicides, while repealing PTP laws was associated with a 47% increase in homicides and 16–24% increase in suicides [45, 49•, 50, 51••]. In addition, two studies analyzed the repeal of Missouri’s PTP law on firearm suicide rates among adolescents and young adults specifically and reported similar increases among this age group [45, 49•]. This literature is limited by only examining the impact of PTP laws in two states: the enactment of Connecticut’s PTP law in 1995 [45, 50, 51••] and the repeal of Missouri’s PTP law in 2007 [45, 49•, 51••].
Restrictions on Individuals
There was mixed evidence on the effectiveness of firearm seizure laws by state, though evidence trended towards reductions in firearm deaths. In 1999, Connecticut became the first state to enact firearm seizure legislation following a mass shooting at the state lottery headquarters. Two examinations of Connecticut’s firearm seizure law found associated reductions in firearm suicide and homicide rates [41, 47•]. In addition, the reduction in firearm suicide rates was more pronounced following the Virginia Tech mass shooting, which led to a fivefold increase in the number of guns seized [47•]. Indiana’s firearm seizure law, enacted in 2005, was also associated with a decrease in firearm suicide rates [47•]. However, a second study identified an increase in firearm homicide rates following Indiana’s firearm seizure law [41]. An additional examination of California’s enactment of a misdemeanor violence prohibition (MVP) policy, alongside the enactment of a comprehensive background check, found no change in either firearm homicides or suicides [43•]. The reason for varying effects across states is unclear but may be linked to differences in the laws themselves or their implementation. For example, Connecticut’s law authorizes the seizure of firearms from individuals deemed a threat to themselves or others, whereas Indiana’s law requires the individual to both be a threat and mentally ill or violently unstable. Such legislative differences may be vital in determining whether firearm seizure laws effectively prevent firearm deaths, especially homicides.
Safe Storage
We identified one SCM evaluation of the safe storage requirement of child access prevention (CAP) laws across 22 US states on youth firearm suicide rates [40•]. The findings were mixed, with CAP laws being associated with lower youth firearm suicide rates in only nine out of 22 states. These results suggest that for some (but not all) US states, there was a preventive effect of CAP laws on firearm suicides among 0- to 18-year olds. However, these findings are subject to potential methodological limitations due to sparse and volatile data (see “Discussion” section).
Policies Targeting Sales
We identified five studies that applied SCM to examine policies targeting firearm sales, specifically the impacts of comprehensive background checks (CBC) [43•, 44, 46, 48, 51••]. There were heterogeneous effects of CBC laws across states for background checks and null effects for firearm deaths. Delaware and Oregon’s CBC laws were associated with increases in the number of backgrounds checks, but no effect was seen in Washington and Colorado [44, 48], even though the law was implemented during a similar period (2013–2015). The authors suggested that the discrepancy in effects may be due to low compliance and/or insufficient enforcement of the laws in the states will null findings. When Washington’s CBC law was enacted (2014), there was a well-documented “I will not comply” rally in the state capital [53], while many county law enforcement officials in Colorado reported that they would not enforce its CBC law [54]. McCourt et al. [51••] found that the CBC laws in Maryland and Pennsylvania were not associated with reductions in firearm deaths. Castillo-Carniglia et al. [43•] reached the same conclusion about Calfornia’s CBC law. A further study found that the repeal of CBC laws in Indiana and Tennessee did not appear to reduce firearm suicides or homicides [46]. All three studies cite widespread non-compliance and infrequent enforcement of CBC laws in these states as a plausible explanation to why CBC laws are not consistently associated with reductions in firearm mortality rates [55, 56]. Together, the evidence from SCM studies suggests that CBC laws alone (i.e., in the absence of PTP laws), and when poorly enforced and complied with, may be insufficient for preventing firearm deaths.
Policies Targeting Use
Two policies that targeted firearm use were evaluated using SCM. Three studies examined the impact of concealed carry laws [42, 49•, 57••], while two studies examined the impact of expanding an individual’s right to use firearms in self-defense [58••, 59]. The enactment of these policies made state firearm legalization more permissive. The estimated effects of both policies were heterogeneous across states. Stand your ground (SYG) laws were associated with robust increases in firearm homicide in some states (e.g., Florida) but not others (e.g., Indiana), and there was inconsistent evidence on whether relaxing concealed carry laws resulted in higher violence rates.
Concealed Carry
A comprehensive US-wide analysis of 33 states found that the enactment of “shall issue” right-to-carry (RTC) laws was associated with an increase in violent crime rates [57••]. But the authors found no robust association between enacting RTC laws and homicide and property crime. A subsequent commentary was published in response to this study and questioned the robustness of these findings [38]. The commentary replicated their analysis and found that it was more common to see reductions rather than increases in violent crime after enacting RTC laws. Another US-wide analysis also found limited evidence of the impact of RTC laws on homicide, with only one (New Mexico) out of eight states showing an increase in homicide and firearm homicide rates following the move from prohibited to RTC status [42]. A single-state analysis of Missouri, however, did report a 32% increase in youth firearm suicides when the minimum legal age to obtain a concealed carry permit was lowered to 19 years [49•]. A similar increase was seen for non-firearm suicide, questioning whether the observed effect resulted from this legal change.
Self-defense
Two studies examined the impact of expanding an individual’s right to use guns in self-defense by enacting SYG laws. A US-wide examination of 14 states reported heterogeneous effects of SYG across states, which varied in magnitude [58••]. Three (Florida, Alabama, Michigan) out of 14 states showed increases in firearm death rates (excluding suicide) that ranged from 13 to 24%. Florida had the largest increase in firearm deaths of 24% and was the only state to show an increase in homicide rates (of 13%). Moreover, an examination of Florida alone reported increases in firearm homicides among adolescents in Florida following the enactment of its SYG law, especially among African American adolescents [59].
Discussion
Methodological Value for Firearm Research
Robust evaluation designs are particularly important for firearm research given the controversy surrounding firearm policy and limitations in the current evidence base due to methodological inconsistencies [60]. Our review suggests that SCM has added methodological value to the firearm literature. First, SCM was used as an alternative method to complement more traditional evaluation approaches, such as DiD and ITS designs. For instance, nine of the 17 included studies used SCM to evaluate firearm policies alongside other analytical approaches and SCM broadly agreed with these other approaches [42, 43•, 45, 46, 47•, 50, 57••, 58••, 59]. Thus, SCM often served as a tool to assess the robustness of findings to different modeling approaches and replicate previous or concurrent examinations of firearm policies.
Second, SCM has been used to examine firearm policies in sparse data contexts where other evaluations would have been underpowered. Because SCM does not rely on formal frequentist inference, it has less stringent data requirements than other evaluation approaches like ITS analyses [23, 61]. A rigorous ITS analysis may require 20–30 pre-intervention data points to model underlying trends and seasonality [17]. SCM does not necessarily need as many pre-intervention data points (yearly data are usually sufficient). Most included studies (n = 14) relied on yearly outcome data, which would typically lead to poor statistical power in ITS evaluations.
Third, SCM was used to evaluate more complex and/or combined interventions, which often changed during the period after the first intervention was implemented. For example, Bhatt et al. [49•] evaluated the impacts of introducing three changes to firearm laws in Missouri: the repeal of PTP laws in 2007, and the lowering of the legal age to obtain a CCW from 23 to 21 years in 2010 and then to 21 to 19 years in 2014. The staggered introduction of multiple interventions in one state (i.e., same intervention unit) would be challenging to analyze using the more traditional method of ITS given the need for an impact model specifying the shape of the intervention effect over time, making it difficult to evaluate multiple interventions that rolled out gradually. SCM, however, offers a more flexible approach for evaluating these “messier” interventions since time-varying impacts can be estimated without specifying an impact model. The impact of firearm laws can therefore be assessed without fragile modeling assumptions [23].
Summary of the Empirical Evidence
The examinations of firearm policies using SCM primarily contributed to the literature by bolstering existing findings. The studies echo much of the inconsistent evidence in firearm research and the heterogeneous effects of firearm policies (see Fig. 1). This suggests that inconsistencies in the current evidence base are not solely due to modeling misspecifications [6, 60]—as SCM counteracts many previous concerns—but may also reflect the varying nature of the impacts of firearm policies themselves. Firearm policies are rarely homogenously formulated, implemented, and enforced across contexts [16]. In addition, changes to firearm policies do not occur in a vacuum but against a backdrop of several types of existing laws and contributing factors, such as the built environment, economic trends, population characteristics, media attention, culture, and law enforcement. Firearm policies most likely interact with these pre-existing laws and characteristics to shape heterogeneous effects across contexts. While it is difficult to tease apart whether inconsistent evidence is due to fragility to modeling approaches or effect heterogeneity, SCM has helped strengthen the latter interpretation for specific firearm policies. For example, SYG self-defense laws have been found to increase homicides in some states (e.g., Florida) but not others (e.g., Indiana) across evaluation designs: SCM [58••], ITS [62], and DiD [11]. This suggests that the variation is not simply caused by methodological issues related to specific methods and may instead capture effect heterogeneity across regions (and time) [63].
Although the SCM evidence indicated heterogeneous effects of firearm policies, there was compelling evidence that purchasing license (PTP) laws have a preventive impact on firearm homicides and suicides (Fig. 1). Reductions in firearm deaths have been observed after the introduction of PTP laws, and increases have been observed when they are repealed. These findings replicate existing literature [6], suggesting that laws that regulate firearm purchases and ownership for all individuals effectively prevent firearm deaths. Our review identified few controversial findings. A SCM analysis of 22 US states found mixed evidence of a preventative effect of CAP laws on youth suicide, with most states showing null effects [40•]. This finding contradicts previous literature, and the author argues that CAP laws may be less effective at reducing youth firearm suicides than prior research suggests [64–66]. However, these findings were based on synthetic controls with questionable validity due to sparse and volatile outcome data (e.g., Rhode Island). The null findings identified by this study may therefore represent inadequate synthetic control fit rather than true null effects of CAP laws in these states.
Recommendations
To exploit the potential of synthetic control methodology in firearm research fully, we make several recommendations. First, SCM should be more widely applied to examine firearm policies beyond the US context. Second, the added value of SCM for examining firearm research pivots around the method being usable with small datasets and not requiring strong modeling assumptions. SCM can thus address biases in alternative evaluation methods, including DiD and ITS [19]. In addition, SCM can identify time-varying effects and can therefore be used to study policy changes that are gradually implemented over time.
Third, future studies should exploit the increasing number of generalizations that have been formulated in recent years. Notably, SCM has now been extended for evaluating multiple and staggered interventions (e.g., generalized and augmented synthetic control methodologies [26, 27]), which would have been informative analytical tools for several of the US-wide studies identified in this review [40•, 42, 57••, 58••]. Among these included studies, Gius [40•] examined the staggered adoption of CAP laws in 22 states, Guettabi [58••] of SYG laws in 14 states, and Donohue [57••] of RTC laws in 33 states across the US. These studies applied single-unit SCM, separately running SCM for each state that adopted the laws and then averaging these estimates [27, 57••]. This approach is computationally unwieldy and not well understood [27]. It potentially incurs bias due to overlapping donor pools as control units often appear in more than one synthetic control, and there is a lack of clarity on how to handle treated units with poor synthetic control fit [27]. Gius [40•] excluded states that showed poor synthetic control fit to address this issue but it is unclear if this introduces its own selection bias. Both generalized and augmented SCM offer the opportunity to evaluate similar firearm policies adopted at the same and/or different times across different units, such as states [26, 27]. These extensions simultaneously estimate an intervention effect at the unit (e.g., state) and aggregate level (e.g., national). Additionally, these extensions strengthen causal inference and overcome inference issues of single-unit SCM as the methods provide uncertainty estimates that conform to more conventional statistical inference, such as confidence intervals using the parametric bootstrap procedure [26].
Finally, as synthetic control methods gain traction and popularity, we caution against SCM being inappropriately and inconsistently applied and underline the importance of standardized reporting. While SCM was generally used with methodological rigor, a handful of studies poorly specified which variables (and their temporal information) were used in the matching process, did not adequately address synthetic control fit, and used unverified methods for statistical inference [40•, 41, 42]. Future research that encounters poor pre-intervention fit due to sparse outcome data should consider matching on smoothed data or using recently proposed bias-correction methods [27, 67, 68]. In addition, future studies should avoid using conventional t-tests to test for differences between the intervention and synthetic control unit because it does not account for the construction of the synthetic control weights [69, 70]. To guard against potential biases, we emphasize the importance of standardizing the application of the method and identified a unified approach to statistical inference, such as placebo tests [21] or new inferential methods developed for SCM [69–73]. We point future firearm research towards recent tutorials that guide researchers through key assumptions and requirements of synthetic control evaluations [23, 61], and provide an overview of key limitations to consider below.
Limitations of SCM for Firearm Research
SCM is an increasingly flexible method that can be used to evaluate firearm policies implemented in discrete areas, such as state or national firearm legislation [23, 61]. Nevertheless, there are key limitations that need to be considered before applying SCM. Similarly to most controlled before-after study designs, evaluators should determine whether the following biases are potent: (1) anticipation bias where the treated area responds to the firearm policy before it is officially enacted; (2) contamination/spillover effects of the firearm policy to control units (e.g., neighboring areas); (3) co-interventions that occur at the same time as the firearm policy (e.g., a recording change to the outcome data); and (4) time-varying confounders in the post-intervention period that do not equivalently affect the treated and synthetic control unit (e.g., COVID-19 pandemic disproportionally impacting the treated area) [32].
Additionally, there are a number of practical and methodological issues to consider when determining the appropriateness of SCM [23]. The more common issues for firearm research include the problem of the convex hull, sparse and noisy outcome data, and poor synthetic control fit (including overfitting). While some of these issues can be overcome [23], the convex hull is an essential condition of SCM which requires the pre-intervention data of the treated unit to fall within the range of the donor pool. This condition cannot always be met. For example, an evaluation of a national firearm policy on gun deaths in Brazil may not be possible as firearm death rates typically exceed other national rates around the world. Sparse and missing outcome data may also present obstacles when examining the impact of policies on infrequent events, such as unintentional firearm deaths or child firearm suicide, as such data may prevent the identification of a good fitting synthetic control to the underlying trends.
Conclusions
Synthetic control methods can meaningfully contribute to the field by offering a complementary evaluation approach for examining firearm policies. To date, SCM has primarily been used to replicate existing findings and the evidence generated from SCM studies continue to show heterogenous effects of firearm policies. This suggests that the current evidence is not purely inconsistent due to methodological limitations in conventional approaches. Although methodological limitations and differences across analytical approaches continue to contribute to observed effect heterogeneity, the SCM evidence reviewed here indicates that inconsistent findings may also reflect true heterogenous effects of certain firearm policies across different contexts. To advance the field further, future research should aim to explicitly exploit the data-driven algorithms and non-frequentist approach of SCM (and its extensions) to triangulate evidence across evaluation approaches with complementary biases [63]. Only then, can evaluations of firearm policies obtain more reliable answers and move closer towards understanding if, how, and why firearm policies have differential impacts.
References
Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance
Naghavi M, Marczak LB, Kutz M, Shackelford KA, Arora M, Miller-Petrie M, et al. Global mortality from firearms, 1990–2016. JAMA. 2018;320:792–814.
Rivara FP, Studdert DM, Wintemute GJ. Firearm-related mortality: a global public health problem. JAMA. 2018;320:764–5.
Werbick M, Bari I, Paichadze N, Hyder AA. Firearm violence: a neglected “Global Health” issue. Glob Health. 2021;17:120.
Grinshteyn E, Hemenway D. Violent death rates in the US compared to those of the other high-income countries, 2015. Prev Med. 2019;123:20–6.
Goldstick JE, Zeoli A, Mair C, Cunningham RM. US firearm-related mortality: national, state, and population trends, 1999–2017. Health Aff. 2019;38:1646–52.
Santaella-Tenorio J, Cerdá M, Villaveces A, Galea S. What do we know about the association between firearm legislation and firearm-related injuries? Epidemiol Rev. 2016;38:140–57.
Hahn RA, Bilukha O, Crosby A, Fullilove MT, Liberman A, Moscicki E, et al. Firearms laws and the reduction of violence: a systematic review. Am J Prev Med. 2005;28:40–71.
Vernick JS, Hepburn LM. State and Federal Gun Law: Trends for 1970–1999. Evaluating Gun Policy. Brookings Institution Press; 2003. p. 345–411.
Rosengart M, Cummings P, Nathens A, Heagerty P, Maier R, Rivara F. An evaluation of state firearm regulations and homicide and suicide death rates. Inj Prev. 2005;11:77–83.
National Research Council. Firearms and Violence: A Critical Review [Internet]. Wellford CF, Pepper JV, Petrie CV, editors. Washington, DC: The National Academies Press; 2005. Available from: https://www.nap.edu/catalog/10881/firearms-and-violence-a-critical-review. Accessed 17 Dec 2021
McClellan C, Tekin E. Stand Your Ground Laws, homicides, and injuries. J Hum Resour. 2017;52:621–53.
Chapman S, Alpers P, Agho K, Jones M. Australia’s 1996 gun law reforms: faster falls in firearm deaths, firearm suicides, and a decade without mass shootings. Inj Prev. 2006;12:365–72.
Ozanne-Smith J, Ashby K, Newstead S, Stathakis V, Clapperton A. Firearm related deaths: the impact of regulatory reform. Injury Prevention. BMJ Publ Group Ltd. 2004;10:280–6.
Lee LK, Fleegler EW, Farrell C, Avakame E, Srinivasan S, Hemenway D, et al. Firearm Laws and Firearm Homicides: A Systematic Review. JAMA Intern Med. 2017;177:106–19.
Schell TL, Griffin BA, Morral AR. Evaluating methods to estimate the effect of state laws on firearm deaths: a simulation study [Internet]. Santa Monica, CA: RAND Corporation; 2018. Available from: https://www.rand.org/pubs/research_reports/RR2685.html. Accessed 17 Dec 2021
Siegel M, Pahn M, Xuan Z, Ross CS, Galea S, Kalesan B, et al. Firearm-related laws in all 50 US states, 1991–2016. Am J Public Health. 2017;107:1122–9.
Lopez Bernal J, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46:348–55.
Fredriksson A, de Oliveira GM. Impact evaluation using Difference-in-Differences. RAUSP Manag. 2019;54:519–32.
Schell TL, Cefalu M, Griffin BA, Smart R, Morral AR. Changes in firearm mortality following the implementation of state laws regulating firearm access and use. Proc Natl Acad Sci USA. 2020;117:14906–10.
Abadie A, Gardeazabal J. The economic costs of conflict: a case study of the Basque Country. Am Econ Rev. 2003;93:113–32.
Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: estimating the effect of California’s Tobacco Control Program. J Am Stat Assoc. 2010;105:493–505.
Abadie A, Diamond A, Hainmueller J. Comparative politics and the synthetic control method. Am J Pol Sci. 2015;59:495–510.
Bonander C, Humphreys D, Esposti MD. Synthetic control methods for the evaluation of single-unit interventions in epidemiology: a tutorial. Am J Epidemiol. 2021. https://doi.org/10.1093/aje/kwab211.
Athey S, Imbens GW. The state of applied econometrics: causality and policy evaluation. Econ Perspec. 2017;31:3–32.
Bouttell J, Craig P, Lewsey J, Robinson M, Popham F. Synthetic control methodology as a tool for evaluating population-level health interventions. J Epidemiol Commun Health. 2018;72:673–8.
Xu Y. Generalized synthetic control method: causal inference with interactive fixed effects models. Polit Anal. 2017;25:57–76.
Ben-Michael E, Feller A, Rothstein J. Synthetic controls and weighted event studies with staggered adoption. arXiv: http://arxiv.org/abs/1912.03290
Cook PJ, Goss KA. The gun debate: Wwhat everyone needs to know. USA: Oxford University Press; 2020.
Browse State Gun Laws [Internet]. Giffords. [cited 2021 Dec 17]. Available from: https://giffords.org/lawcenter/gun-laws/browse-state-gun-laws/
Cherney S, Morral AR, Schell T, Smucker S. RAND State Firearm Law Database [Internet]. [cited 2019 Oct 11]. Available from: https://www.rand.org/research/gun-policy/law-navigator.html
Parast L, Hunt P, Griffin BA, Powell D. When is a match sufficient? A score-based balance metric for the synthetic control method. J Causal Inference. 2021;8:209–28.
DegliEsposti M, Spreckelsen T, Gasparrini A, Wiebe DJ, Bonander C, Yakubovich AR, et al. Can synthetic controls improve causal inference in interrupted time series evaluations of public health interventions? Int J Epidemiol. 2020;49:1–11.
McClelland R, Gault S. The synthetic control method as a tool to understand state policy [Internet]. Washington, DC: Urban Institute; 2017 p. 1–46. Report No.: 20037. Available from: https://www.urban.org/sites/default/files/publication/89246/the_synthetic_control_method_as_a_tool_1.pdf. Accessed 17 Dec 2021
McCourt AD. Concealed carry of firearms in the United States: a public health law analysis of state policy and state suicide Mortality [Internet] [Ph.D.]. ProQuest Dissertations and Theses. [Ann Arbor]: The Johns Hopkins University; 2019. Available from: https://www.proquest.com/dissertations-theses/concealed-carry-firearms-united-states-public/docview/2352101087/se-2?accountid=13042. Accessed 17 Dec 2021
Buggs SAL. Evaluating efforts to reduce gun violence in Baltimore: drug law enforcement, cure violence, and focused deterrence. Dissertation Abstracts International: Section B: The Sciences and Engineering. 2019;80. https://jscholarship.library.jhu.edu/handle/1774.2/60065
Williams MC Jr. Race and Gun Violence in the United States: A Case Study of Policy Reform in Missouri [Internet] [Ph.D.]. ProQuest Dissertations and Theses. [Ann Arbor]: City University of New York; 2018. Available from: https://www.proquest.com/dissertations-theses/race-gun-violence-united-states-case-study-policy/docview/2046418049/se-2?accountid=13042. Accessed 17 Dec 2021
Rehkopf DH, Basu S. A new tool for case studies in Epidemiology - the Synthetic Control Method. Epidemiology. 2018;29:503–5.
Moody CE, Marvell TB. Do right to carry laws increase violent crime? A comment on Donohue, Aneja, and Weber. Econ Journal Watch. 2019;16:84–96.
Bartos BJ, McCleary R, Mazerolle L, Luengen K. Controlling gun violence: assessing the impact of Australia’s gun buyback program using a synthetic control group experiment. Prev Sci United States. 2020;21:131–6. Only SCM evaluation outside the US, examining the impact of Australia’s 1996 national Gun Buyback Program on homicide and suicide.
Gius M. Examining the impact of child access prevention laws on youth firearm suicides using the synthetic control method. Int Rev Law Econ. 2020;63:105914. US-wide analysis of 22 states controversially showing that child access prevention laws may not be as effective as prior research identified.
Gius M. Using the synthetic control method to determine the effects of firearm seizure laws on state-level murder rates. Appl Econ Lett. 2020;27:1754–8.
Gius M. Using the synthetic control method to determine the effects of concealed carry laws on state-level murder rates. Int Rev Law Econ. 2019;57:1–11.
Castillo-Carniglia A, Kagawa RMC, Cerdá M, Crifasi CK, Vernick JS, Webster DW, et al. California’s comprehensive background check and misdemeanor violence prohibition policies and firearm mortality. Ann Epidemiol. 2019;30:50–6. Identified that California’s comprehensive background checks and misdemeanour violence prohibition laws were not associated with changes in firearm homicide or suicide.
Castillo-Carniglia A, Kagawa RMC, Webster DW, Vernick JS, Cerdá M, Wintemute GJ. Comprehensive background check policy and firearm background checks in three US states. Inj Prev England. 2018;24:431–6.
Crifasi CK, Meyers JS, Vernick JS, Webster DW. Effects of changes in permit-to-purchase handgun laws in Connecticut and Missouri on suicide rates. Prev Med. 2015;79:43–9.
Kagawa RMC, Castillo-Carniglia A, Cerda M, Shev A, Wintemute GJ, Vernick JS, et al. Repeal of comprehensive background check policies and firearm homicide and suicide. Epidemiol. 2018;29:494–502.
Kivisto AJ, Phalen PL. Effects of risk-based firearm seizure laws in Connecticut and Indiana on suicide rates, 1981–2015. Psychiatr Serv. 2018;69:855–62. Two state examination of firearm seizure laws and found a significant reduction in suicide in both Indiana and Connecticut, which became more pronounced following the mass shooting in Virginia Tech.
Castillo-Carniglia A, Webster DW, Wintemute GJ. Effect on background checks of newly-enacted comprehensive background check policies in Oregon and Washington: a synthetic control approach. Inj Epidemiol. 2019;6:45.
Bhatt A, Wang X, Cheng A-L, Morris KL, Beyer L, Chestnut A, et al. Association of changes in Missouri Firearm laws with adolescent and young adult suicides by firearms. JAMA Netw Open. 2020;3:e2024303. Examination of the impacts of three firearm law changes in Missouri (repeal of permit-to-purchase laws and lowering legal age of obtaining a carrying a concealed weapon) on youth suicide.
Rudolph KE, Stuart EA, Vernick JS, Webster DW. Association between Connecticut’s permit-to-purchase handgun law and homicides. Am J Public Health. 2015;105:e49-54.
McCourt AD, Crifasi CK, Stuart EA, Vernick JS, Kagawa RMC, Wintemute GJ, et al. Purchaser licensing, point-of-sale background check laws, and firearm homicide and suicide in 4 US states, 1985–2017. Am J Public Health. 2020;110:1546–52. Comprehensive examination of permit-to-purchase law enactment and repeal and comprehensive background checks law enactment showing permit-to-purchase have a robust preventive effect but comprehensive background checks laws do not.
Kahane LH, Sannicandro P. The impact of 1998 Massachusetts gun laws on suicide: a synthetic control approach. Econ Lett. 2019;174:104–8. A comprehensive evaluation of Massachusetts’ Gun Law Reform in 1998 which changed 23 firearm laws and resulted in a sustained reduction in firearm suicide.
O’Sullivan J. ‘I will not comply’ rally draws gun-rights supporters to Olympia. Seattle Times [Internet]. 2014; Available from: http://www.seattletimes.com/seattle-news/lsquoi-will-notcomplyrsquo-rally-draws-gun-rights-supporters-to-olympia/. Accessed 17 Dec 2021
Allen R. Few violations of Colorado’s ‘unenforceable’ gun laws. USA Today [Internet]. 2014; Available from: http://www.usatoday.com/story/news/nation/2014/01/30/colorado-gun-lawsenforcement/5055523/. Accessed 17 Dec 2021
Miller M, Hepburn L, Azrael D. Firearm acquisition without background checks: results of a national survey. Ann Intern Med. 2017;166:233–9.
Crifasi CK, Merrill-Francis M, Webster DW, Wintemute GJ, Vernick JS. Changes in the legal environment and enforcement of firearm transfer laws in Pennsylvania and Maryland. Inj Prev. 2019;25:i2-4.
Donohue JJ, Aneja A, Weber KD. Right-to-carry laws and violent crime: a comprehensive assessment using panel data and a state-level synthetic control analysis. J Empir Leg Stud. 2019;16:198–247. US-wide analysis of 33 states showing an increase in violent crime following the enactment of shall issue right-to-carry laws.
Guettabi M, Munasib A. Stand Your Ground laws, homicides and gun deaths. Reg Stud. 2018;52:1250–60. US-wide analysis of 14 states identifying heterogenous effects of Stand Your Ground laws on firearm homicides, with a robust associated increase following the enactment of Florida’s Stand Your Ground law.
Degli Esposti M, Wiebe DJ, Gravel J, Humphreys DK. Increasing adolescent firearm homicides and racial disparities following Florida’s ‘Stand Your Ground’ self-defence law. Inj Prev. 2019;26:187–90.
Morral A, Ramchand R, Smart R, Gresenz C, Cherney S, Nicosia N, et al. The science of gun policy: a critical synthesis of research evidence on the effects of gun policies in the United States [Internet]. RAND Corporation; 2018 [cited 2019 Feb 18]. Available from: https://www.rand.org/pubs/research_reports/RR2088.html
Abadie A. Using synthetic controls: feasibility, data requirements, and methodological aspects. J Econ Lit. 2021;59:391–425.
Degli Esposti M, Wiebe DJ, Gasparrini A, Humphreys DK. Analysis of “Stand Your Ground” Self-defense Laws and Statewide Rates of Homicides and Firearm Homicides. JAMA Netw Open. 2022;5(2):e220077. https://doi.org/10.1001/jamanetworkopen.2022.0077.
Lawlor DA, Tilling K, Davey SG. Triangulation in aetiological epidemiology. Int J Epidemiol. 2016;45:1866–86.
Grossman DC, Mueller BA, Riedy C, Dowd MD, Villaveces A, Prodzinski J, et al. Gun storage practices and risk of youth suicide and unintentional firearm injuries. JAMA. 2005;293:707–14.
The effects of child-access prevention laws [Internet]. [cited 2021 Dec 27]. Available from: https://www.rand.org/research/gun-policy/analysis/child-access-prevention.html
Webster DW, Vernick JS, Zeoli AM, Manganello JA. Association between youth-focused firearm laws and youth suicides. JAMA. 2004;292:594–601.
Shioda K, Schuck-Paim C, Taylor RJ, Lustig R, Simonsen L, Warren JL, et al. Challenges in estimating the impact of vaccination with sparse data. Epidemiol. 2019;30:61.
Shi X, Miao W, Hu M, Tchetgen ET. Theory for identification and inference with synthetic controls: a proximal causal inference framework. arXiv: http://arxiv.org/abs/2108.13935
Chernozhukov V, Wuthrich K, Zhu Y. Practical and robust t-test based inference for synthetic control and related methods. arXiv preprint [Internet]. 2020; Available from: https://drive.google.com/file/d/1DS6u6WCLh-Ioea5DOSaszVL1m-MdFHze/view. Accessed 17 Dec 2021
Li KT. Statistical inference for average treatment effects estimated by synthetic control methods. J Am Stat Assoc. 2020;115:2068–83.
Chernozhukov V, Wüthrich K, Zhu Y. An exact and robust conformal inference method for counterfactual and synthetic controls. J Am Stat Assoc. 2021;116:1849–64.
Firpo S, Possebom V. Synthetic control method: inference, sensitivity analysis and confidence sets. J Causal Inference. 2018;6:20160026.
Pang, X., Liu, L., & Xu, Y. (2022). A Bayesian Alternative to Synthetic Control for Comparative Case Studies. Political Analysis, 30(2), 269–288. https://doi.org/10.1017/pan.2021.22
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare no competing interests.
Human and Animal Rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is part of the Topical Collection on Injury Epidemiology.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Esposti, M.D., Wiebe, D., Kaufman, E. et al. Synthetic Control Methodology for Examining Firearm Policy. Curr Epidemiol Rep 9, 109–125 (2022). https://doi.org/10.1007/s40471-022-00294-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40471-022-00294-9