Racial Disparity in U.S. Imprisonment Across States and Over Time

Abstract

Objectives

We analyze the contribution of changes in the black–white racial disparity in imprisonment to changes in the black incarceration rate. We also describe the behavior of the racial disparity across states and across time.

Methods

We use state level incarceration data for non-Hispanic black and white males to perform a decomposition of the black incarceration rate. This allows us to attribute changes in black incarceration to changes in the racial disparity and changes in the overall incarceration rate. We use a Fourier approximation to identify structural change points for the racial disparity at both the state and national level.

Results

The large increase in black imprisonment between 1978 and 1999 was driven by increases in the overall rate of imprisonment, while the smaller decrease which occurred between 1999 and 2014 was driven by reductions in the black–white racial disparity. For many states, the racial disparity increased starting in the mid-1980s, where this increase may have been linked to the crack epidemic. Many states experienced a downturn in the racial disparity starting in the 1990s. Whatever its other effects, this suggests that the 1994 crime bill did not aggravate the preexisting racial disparity in imprisonment. California’s experience has been strongly counter to national trends with a large increase in the racial disparity beginning in the early 1990s and continuing until near the end of our sample.

Conclusion

While the racial disparity in imprisonment has been falling since 1996, it remains quite high as of 2014. Future work is required to better understand the policy determinants of this disparity.

Introduction

The extraordinarily high rate of incarceration of Black Americans is a function of two factors. First, the United States has a very high overall incarceration rate. In 2014, this rate was 693 per 100,000 inhabitants, which was the second highest rate in the world.¹ Second, there are very large racial disparities in imprisonment rates in the United States. In 2014, non-Hispanic black males were incarcerated at 5.5 times the rate of non-Hispanic white males.² The high overall incarceration rate in the United States combined with the large racial disparity led to an incarceration rate of 2784 per 100,000 Black males in 2014.³ The primary focus of this paper is on this racial disparity. We provide a largely descriptive analysis of how this disparity varies across states and across time. It is notable that there is wide variation in the black–white imprisonment ratio across states. For example, in 2014 the black-to-white incarceration ratio ranged from 1.60 in Hawaii to 13.8 in Minnesota.

Between 1978 and 1999 there is an extraordinarily large increase in the national black incarceration rate as it rose from about 1080 to near 3500. We decompose this change at the state level in order to attribute the sources of the change at the national level. We find the increase in black incarceration is overwhelmingly due to increases in the overall incarceration rate in the U.S. during this period. Between 1999 and 2014 there is a moderate fall in the black incarceration rate to about 2800. Over 50% of this fall is due to a reduction in the black–white disparity during this period. There is already a large Black–White disparity in imprisonment at the beginning of our sample which is 1978.⁴ In many states, this disparity worsens beginning in the mid-1980s. This corresponds to the period of the crack epidemic. Evans et al. (2016) document an upsurge in incarceration of young black men during this period and attribute this directly to the crack epidemic. Many states experienced a downturn in the racial disparity beginning sometime in the 1990s. The timing of these downturns suggests that, whatever its other effects, the 1994 crime bill did not exacerbate the racial disparity in imprisonment. Several provisions of this bill, including incentives to build prisons, hire more police and for the adoption of truth-in-sentencing laws, had the potential for producing a racially disparate impact, but we find no prima facie evidence that this is the case. California is a notable exception to the national trend as it experienced a large rise in the racial disparity beginning in the early 1990s and continuing until near the end of our sample period.

Background

Incarceration rates rose sharply from the 1970s through the 1990s. One intuitive explanation for the spike in the incarceration rate is that this occurred in response to a spike in crime. As crime rates rise/fall we expect incarceration rates to rise/fall as well (perhaps with a lag). However, as shown in Fig. 1, the data do not show such a simple relationship between crime and incarceration rates. Incarceration rates followed the violent crime rate up from 1978 to the early 1990s. However, the substantial drop in the violent crime rate since the early 1990s has only been met with a flattening out of the incarceration rate.⁵ Thus, while the violent crime rate plays a role in determining the imprisonment rate, other factors are also clearly important.

Fig. 1

Sources: National Prisoner Statistics (NPS), compiled by the Bureau of Justice Statistics. FBI, Uniform Crime Reports, prepared by the National Archive of Criminal Justice Data

U.S. Black male and white male incarceration rates and violent crime (incarceration rates are on the left scale and violent crime rates on the right scale).

Blumstein and Beck (1999) state that only 12% of the growth in the state prison population between 1980 and 1996 was due to increases in rates of offending. They attributed the remaining 88% to increases in the imposition of sanctions (51%) and longer sentences served (37%). While other articles in the literature echo this view (e.g., Zimring and Hawkins 1993), the literature finds the crime rate coefficient to be positive and significant when explaining prison populations or admissions (Michalowski and Pearson 1990; Arvanites 1992; Arvanites and Asher 1995, 1998; Beckett and Western 2001; Greenberg and West 2001; Jacobs and Helms 2001; Sorensen and Stemen 2002; Smith 2004). For example, Stemen et al. (2005) found that incarceration rates grew more in states with higher property crime rates than in other states. Some studies suggest that these coefficients could have been underestimated as they ignore the simultaneous relationship between prisons and crime (Listokin 2003).⁶

The political climate of the last few decades has led to much harsher sentencing policies. Those policies have been at the heart of the debate aimed at explaining the boom in incarceration rates in the United States. A series of laws were passed in the 80s and 90s which reflected the political imperative to get “tough on crime”. The 1984 Comprehensive Crime Control Act signed by President Ronald Reagan was the first major revision of the U.S. criminal code since the early 1900s. The Sentencing Reform Act, which was part of the 1984 Act, prescribed sentencing guidelines and eliminated judicial discretion in order to increase consistency in federal sentencing.

Soon after, the 1986 Anti-Drug Abuse Act established mandatory minimum sentences for federal drug offenses. This act is widely believed to have had a racially disparate impact that disfavored black Americans. The act sets a minimum sentence of 5 years without parole for possessing 5 g of crack cocaine (mostly used by blacks) while giving the same punishment for holding 500 g of powder cocaine (mostly used by whites).⁷ In the early 90s, a growing number of states started to pass three-strikes and truth-in sentencing laws which overall, resulted in longer prison sentences. This was followed in 1994 by President Clinton’s Crime Bill, the largest crime bill in the history of the United States.⁸ The bill promised $10 billion to states for prison construction between 1995 and 2000, but only for states that passed truth-in-sentencing laws that eliminated most “good time” provisions and required convicted offenders to serve 85% of their prison sentence. The effect on state policy was almost immediate, the number of states with truth-in-sentencing statutes growing from 4 in 1992 to 27 in 1998. The bill also provided funds for the hiring of 100,000 additional police.⁹ These provisions of the bill could affect black incarceration either through the overall incarceration rate or through an effect on the racial disparity. Our analysis will focus on possible effects on the racial disparity. To the best of our knowledge this question has not been addressed in the prior literature.

The key feature of the data is that the imprisonment rate has not followed the crime rate down. Pfaff (2017) argues that this is due to an increased probability of imprisonment, conditional upon arrest. He attributes this to changes in prosecutorial behavior at the county level. Pfaff (2017, 5–6) also argues that the war on drugs is not primarily responsible for mass incarceration as only about 16% of state prisoners are incarcerated for drug offenses.¹⁰

We turn next to analyses of the racial disparity in imprisonment. Blumstein (1982) concluded that racial patterns of arrests explained about 80% of the disparities in imprisonment. His conclusions were confirmed by Langan (1985) who used victim data instead of arrest and prison admission data instead of population data.¹¹ More recent studies (Tonry and Melewski 2008; Baumer 2013), using the same method as Blumstein, found that racial disparities in imprisonment became much worse in the twenty-first century and that between 39 and 66% of racial disparities could not be explained by arrest patterns. However, Beck and Blumstein (2017) argue that a higher percentage of the racial disparity can be attributed to differential rates of offending, if Hispanic arrestees are properly taken into account. On average, they find that 72% of the disparity in imprisonment can be explained by differential rates of offending, but this varies widely by the type of crime. Moreover, this still suggests a substantial portion of the disparity in imprisonment cannot be explained by differential rates of offending.¹²

Papers in the line of Blumstein (1982) use arrest or victimization data in order gain insight into how much of the racial disparity in imprisonment can be explained by differential rates of offending. Typically the focus is on a single year or on a handful of years’ worth of data. By contrast, we make no attempt to analyze how much of the racial disparity can be explained by differential rates of offending. Rather we analyze the behavior of the racial disparity over a long time period across all fifty states. The perspective we gain from such an analysis provides clues as to the role of events such as the crack epidemic or laws, such as the crime bill of 1994 on the racial disparity. We also perform a decomposition which gives us insight into how changes in the racial disparity and the overall level of imprisonment have each affected the black imprisonment rate over time.

An important predecessor to our work is Bridges and Crutchfield (1988). They regressed both black imprisonment rates and white imprisonment rates on state characteristics, and then tested the statistical differences between those two models. They also regressed the black-to-white imprisonment rates ratio on those same state characteristics. Overall, social characteristics of states were found to contribute significantly to racial disparity in imprisonment. Racial disparities are found to be positively associated with the degree of urban concentration of blacks and to a lesser extent the black to white economic inequality ratio.¹³ However, they find state legal characteristics to have little to no effect on imprisonment rates and racial disparity in imprisonment. It is worth noting that we use the same measure of racial disparity as Bridges and Crutchfield, by taking the ratio of the black to white imprisonment rate. Our work contrasts with theirs by analyzing the behavior of the racial disparity over time rather than focusing on the cross sectional determinants of the disparity.

Bridges and Crutchfield provide a regional analysis of the racial disparity. Another important work along these lines is by Muller (2012). He argues that the origin of the racial disparity in northern states was a reaction by recent European immigrants to the influx of blacks from the south. His statistical analysis of census data from the period 1880–1950 shows that greater black migration into a state leads to a higher black incarceration rate within that same state. Muller’s work is notable because, as we shall see, northern and Midwestern states tend to have a very large racial disparity in imprisonment. The dynamics described by Muller for this earlier period are likely important in explaining the initial racial disparity we observe at the beginning of our data set in 1978. As in Muller and Bridges and Crutchfield, our work provides an extensive regional analysis of the racial disparity.

Our purpose in this paper is two-fold. First, we want to describe the sources of the vast increase in black imprisonment since the late 1970s. How much of this is due to the overall increase in imprisonment in the United States and how much is due to changes in the racial disparity in imprisonment? We do this by decomposing changes in the black imprisonment rate at the state level and then aggregate up to the national level. Second, we want to shed light on the question of whether one particular piece of legislation, the 1994 crime bill, aggravated the preexisting racial disparity in imprisonment. We do this by estimating when local maxima and minima occur in the black–white disparity at the state level in the years before and after this crime bill is passed.

Data

We use two different datasets to construct our two main variables: incarceration rates by state and their corresponding black-to-white ratios. We obtained state-level prison populations from 1978 to 2014 using the National Prisoner Statistics (NPS) dataset compiled by the Bureau of Justice Statistics. The NPS breaks down the number of prisoners under the jurisdiction of each state by sex and race. Jurisdictional data on race only goes back to 1978, which is why this is the first year in our dataset. In our analysis, we only focus on non-Hispanic white and black males.¹⁴ We extracted state-population estimates broken down by sex and race from the U.S. Census Bureau. Because those estimates are created using censuses conducted every 10 years and some variables/definitions changed over time, we made a few changes to maintain consistency. First, prior to 1980, the Census did not differentiate Hispanics from non-Hispanics. In order to be consistent with the following years as well as with the prisoners’ data (which separated Hispanics from non-Hispanics for the whole period), we estimated the years 1978–1980.¹⁵ We used the average growth of the population for each state in order to perform this estimation.

Second, in 2000, the Census started to allow individuals to self-identify with more than one race. This led to the creation of a new variable (two or more races) which impacted all the other variables. To deal with this new variable, the census made available two different datasets: one that excluded all those identified as two or more races from the other races, and the other that included them (hence, allowing for individuals to be accounted two times or more). For example, an individual who identifies as both white and Asian would be accounted for only once under the two or more races variable in the first dataset, and would be accounted for once under the white variable, and once under the Asian variable in the second dataset.¹⁶ This change thus led to discrepancies between data prior to 2000 and post-2000, whichever post-2000 dataset we used. We decided to use the growth rates of the raw data to estimate all the values after 1999 in order to get a more continuous variable. Specifically, we used the average percent yearly change during each 5-year period (i.e. 2000–2004, 2005–2009, and 2010–2014) in the raw data combined with the previous value (starting with 1999) to estimate all the values after 1999. For example, any 2000 value would be equal to its corresponding 1999 value times one plus its 2000–2004 corresponding growth rate.¹⁷

Using these two datasets, we calculated the incarceration rate per 100,000 male inhabitants for both blacks and whites, where these rates are denoted by B and W respectively. We also calculated the overall incarceration rate, denoted A, which corresponds to the sum of the black plus white prisoners per 100,000 of the black plus white population. Figure 1 shows the black and white imprisonment rates over time. The black imprisonment rate rises sharply between 1978 and 1999 and then falls slowly between 1999 and 2014. As a result of this pattern, we will focus some of our analysis on the years 1978, 1999 and 2014. Part of our goal is to understand the sharp rise and smaller subsequent decrease in the black imprisonment rate in terms of overall incarceration in the United States and the racial disparity between blacks and whites. Table 1 shows the black imprisonment rate across states for the years 1978, 1999, and 2014. The cross state variation is very large. For example, in 2014, this rate (per 100,000) ranges from about 900 in Hawaii to about 5500 in Wisconsin and Vermont. The rate aggregated over all 50 states (but excluding federal prisoners) rises from 1080 in 1978 to about 3500 in 1999. This rate subsequently falls to 2784 by 2014.

Table 1 Black incarceration rates (BIR)—per 100,000 population

Our measure of the racial disparity, r = B/W, is shown in Table 2.¹⁸ Nationally, there is a large initial racial disparity equal to 6.9 in 1978 and this rises slightly to 7.1 in 1999. This rate then has a fairly substantial decrease to 5.5 in 2014. What the national figures do not show is the high variability in this ratio across states. For example, in 2014, the racial disparity ranged from 1.6 in Hawaii to 13.8 in Minnesota. States with the highest black-to-white ratio are disproportionately located in the Northeast and Midwest. In 2014, the states with the ten highest racial disparities were all in the Northeast or Midwest. On the other hand, states with the lowest black-to-white ratio are overwhelmingly from the South, with some from the West. In 2014, 9 of the 10 states with the lowest racial disparity were in the South or West. These results are largely in line with the prior analysis of Bridges and Crutchfield (1988, Table 1).

Table 2 Black–White ratio

Decomposition of the Changes in Black Imprisonment Over Time

As noted earlier, the national black-male-incarceration rate rose from about 1080 in 1978 to a peak of about 3500 in 1999. It then declined to 2784 in 2014. In this section, we decompose the sharp upward movement in this rate between 1978 and 1999 and the smaller decrease observed between 1999 and 2014. Our goal is to understand how changes in the racial disparity and the overall incarceration rate contributed to the rise in black imprisonment. To perform this decomposition, we cannot simply rely on the national data. For example, if the overall incarceration rate were to rise in a state with a large black population, this would increase the black–white ratio at the national level even if this ratio is unchanged in the state in question. To avoid this type of aggregation issue, we first decompose the incarceration rates at the state level and then aggregate up to the national level. Our decomposition will allow us to state how, for example, the changing racial disparity in Florida contributed to the national rise in black incarceration in the 1978–1999 period. To do this we first need to determine how the changes in each state’s black incarceration rate contributed to the national rate. We will then decompose changes in the state black incarceration rate into changes in the racial disparity, the overall incarceration rate and demographics within that state.¹⁹

Consider first how state incarceration rates aggregate to form the national rate. Let $S_{i}^{t}$ be state i’s share of the national black population in year t, and let $B_{i}^{t}$ be that state’s black incarceration rate in year t. The national black incarceration rate in year t is $B_{N}^{t}$. We then have the following:

$$B_{N}^{t} = \sum\limits_{i = 1}^{50} {S_{i}^{t} B_{i}^{t} } .$$

(1)

We can compute the contribution of changes in state i’s black incarceration rate to the change in the national rate between 1999 and 1978 by using the difference $B_{i}^{1999} - B_{i}^{1978}$. This change can be weighted by either $S_{i}^{1999}$ or $S_{i}^{1978} .$ We do this both ways and then take the average contribution to determine how changes in the state’s black incarceration rate contributed to changes in the national rate. Let $B_{Ni}^{1999 - 1978}$ be the contribution of the change in the black incarceration rate in state i to the change in the national black incarceration rate between 1999 and 1978. We then have

$$B_{Ni}^{1999 - 1978} = (0.5)\left( {B_{i}^{1999} - B_{i}^{1978} } \right)\left( {S_{i}^{1999} + S_{i}^{1978} } \right)$$

(2)

We can express $B_{Ni}^{2014 - 1999}$ by a suitable adjustment of the time superscripts in Eq. (2).

By making an analogous computation, we can compute the contribution of changes in the demographic weights S to the change in the black national imprisonment rate. For example, we have $S_{Ni}^{1999 - 1978} = (0.5)\left( {S_{i}^{1999} - S_{i}^{1978} } \right)\left( {B_{i}^{1999} + B_{i}^{1978} } \right)$. Combined with (2), this yields the full decomposition of the national black incarceration rate as follows:

$$B_{N}^{1999} - B_{N}^{1978} = 0.5\sum\limits_{i = 1}^{50} {\left[ {\left( {B_{i}^{1999} - B_{i}^{1978} } \right)\left( {S_{i}^{1999} + S_{i}^{1978} } \right) + \left( {S_{i}^{1999} - S_{i}^{1978} } \right)\left( {B_{i}^{1999} + B_{i}^{1978} } \right)} \right]} .$$

(3)

Equation (3) provides an exact decomposition of the change in the national rate into the changes in the state rates and the black population shares. In what follows we will focus on the changes in the state black incarceration rates as shown in Eq. (2).

Once we have the contribution of changes in the B_i to the change in the national black incarceration rate, the next step is to decompose the change in each state’s black incarceration rate into three component parts: changes in the state’s racial disparity r = B/W; changes in the overall incarceration rate in the state, A (including whites and blacks only); and changes in the demographic weights x, where x is the number of black men in state i divided by the total number of black and white men in state i. This will allow us to control for demographic effects on the black–white imprisonment ratio.

Note that the overall incarceration rate in state i in year t is simply the weighted average of the black and white rates:

$$A_{i}^{t} = x_{i}^{t} B_{i}^{t} + \left( {1 - x_{i}^{t} } \right)W_{i}^{t} .$$

(4)

Substitute $W_{i}^{t} = B_{i}^{t} /r_{i}^{t}$ into (4) and solve for $B_{i}^{t}$ to get the following:

$$B_{i}^{t} = \frac{{r_{i}^{t} A_{i}^{t} }}{{r_{i}^{t} x_{i}^{t} + 1 - x_{i}^{t} }}$$

(5)

Equation (5) expresses the black incarceration rate as a function of the overall incarceration rate $A_{i}^{t}$, the racial disparity $r_{i}^{t}$ and demographic weights, $x_{i}^{t}$. We will use Eq. (5) to decompose the change in a state’s black incarceration rate into components which reflect changes in A, r, and x. Note, we are not implying a causal relationship via (5), as the black incarceration rate is simply a component of the overall rate. However, it is useful to decompose the black incarceration rate into overall incarceration trends and the racial disparity. If r >1, which is generally true, then a rise in x, which is an increase in the black share of the black plus white male population, will lead to a decrease in B holding A and r constant. While we report the effects of changes in the x’s on the tables which follow, this will not be a focus of our discussion.

There is not a unique way in which to perform this decomposition. If r, A and x are set at their 1978 values in (5) we can estimate the effect of changing r on B, by changing r to its 1999 value, while keeping A and x at their 1978 values. However, we can also compute the effect of changing r with either A or x at its 1999 value or with both A and x at their 1999 value. Since there are four ways to compute the effect of a change in r on B, we simply average across all four. We compute the contributions of changes in A and x in a completely analogous fashion. Let $A_{Bi}^{1999 - 1978}$, $r_{Bi}^{1999 - 1978}$ and $x_{Bi}^{1999 - 1978}$ be the respective contributions of the overall incarceration rate, the racial disparity and the demographic weights to the change in state i’s black incarceration rate between 1999 and 1978. We then have the following:

$$\begin{aligned} r_{Bi}^{1999 - 1978} & = 0.25\left( {\frac{{r^{1999} A^{1978} }}{{r^{1999} x^{1978} + 1 - x^{1978} }} - \frac{{r^{1978} A^{1978} }}{{r^{1978} x^{1978} + 1 - x^{1978} }}} \right. \\ & \quad + \frac{{r^{1999} A^{1999} }}{{r^{1999} x^{1978} + 1 - x^{1978} }} - \frac{{r^{1978} A^{1999} }}{{r^{1978} x^{1978} + 1 - x^{1978} }} \\ & \quad + \frac{{r^{1999} A^{1978} }}{{r^{1999} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1978} A^{1978} }}{{r^{1978} x^{1999} + 1 - x^{1999} }} \\ & \quad \left. { + \frac{{r^{1999} A^{1999} }}{{r^{1999} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1978} A^{1999} }}{{r^{1978} x^{1999} + 1 - x^{1999} }}} \right) \\ \end{aligned}$$

(6a)

$$\begin{aligned} A_{Bi}^{1999 - 1978} & = 0.25\left( {\frac{{r^{1978} A^{1999} }}{{r^{1978} x^{1978} + 1 - x^{1978} }} - \frac{{r^{1978} A^{1978} }}{{r^{1978} x^{1978} + 1 - x^{1978} }}} \right. \\ & \quad + \frac{{r^{1999} A^{1999} }}{{r^{1999} x^{1978} + 1 - x^{1978} }} - \frac{{r^{1999} A^{1978} }}{{r^{1999} x^{1978} + 1 - x^{1978} }} \\ & \quad + \frac{{r^{1978} A^{1999} }}{{r^{1978} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1978} A^{1978} }}{{r^{1978} x^{1999} + 1 - x^{1999} }} \\ & \quad \left. { + \frac{{r^{1999} A^{1999} }}{{r^{1999} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1999} A^{1978} }}{{r^{1999} x^{1999} + 1 - x^{1999} }}} \right) \\ \end{aligned}$$

(6b)

$$\begin{aligned} x_{Bi}^{1999 - 1978} & = 0.25\left( {\frac{{r^{1978} A^{1978} }}{{r^{1978} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1978} A^{1978} }}{{r^{1978} x^{1978} + 1 - x^{1978} }}} \right. \\ & \quad + \frac{{r^{1999} A^{1978} }}{{r^{1999} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1999} A^{1978} }}{{r^{1999} x^{1978} + 1 - x^{1978} }} \\ & \quad + \frac{{r^{1978} A^{1999} }}{{r^{1978} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1978} A^{1999} }}{{r^{1978} x^{1978} + 1 - x^{1978} }} \\ & \quad \left. { + \frac{{r^{1999} A^{1999} }}{{r^{1999} x^{1999} + 1 - x^{1999} }} - \frac{{r^{1999} A^{1999} }}{{r^{1999} x^{1978} + 1 - x^{1978} }}} \right) \\ \end{aligned}$$

(6c)

Note that $A_{Bi}^{2014 - 1999}$, $r_{Bi}^{2014 - 1999}$ and $x_{Bi}^{2014 - 1999}$ are defined and computed in an analogous manner. In (6a), where we are computing the effect of the change in r_i on B_i, each of the four lines in the parentheses has r¹⁹⁹⁹ in the first term and r¹⁹⁷⁸ in the second. The four lines show the four combinations of values of A and x that are obtained by using either the 1978 or 1999 value. Similar statements are true for (6b) and (6c) where we compute the effect of changing A and x on B.

Because of the functional form in (5), the decomposition of the change in the black incarceration rate into (6a)–(6c) is not exact. Using (5) and (6a–6c) it can be shown that

$$r_{Bi}^{1999 - 1978} + A_{Bi}^{1999 - 1978} + x_{Bi}^{1999 - 1978} - (B_{i}^{1999} - B_{i}^{1978} ) = (0.25)(A^{1999} - A^{1978} )(x^{1999} - x^{1978} )Z,$$

(7)

where $Z = \frac{{r^{1999} (r^{1999} - 1)}}{{(r^{1999} x^{1978} + 1 - x^{1978} )(r^{1999} x^{1999} + 1 - x^{1999} )}} - \frac{{r^{1978} (r^{1978} - 1)}}{{(r^{1978} x^{1978} + 1 - x^{1978} )(r^{1978} x^{1999} + 1 - x^{1999} )}} .$

The right-hand side of (7) shows the amount by which the sum of our decomposition terms deviates from the actual change in black incarceration in state i. We will call this deviation the remainder. This remainder can be positive or negative and its magnitude is increasing in the magnitude of the changes in A, x, and r. However, one factor contributing to a small remainder is that changes in the demographic weights, x, tend to be small over time. During the 1978–1999 period the average magnitude of the remainder is 0.55% of each state’s change in the black imprisonment rate. During the 1999–2014 period the average magnitude of the remainder becomes 3.03%.²⁰

We can aggregate the contributions of the racial disparity, overall incarceration rates and the demographic weights to $B_{Ni}^{1999 - 1978}$ as follows:

$$r_{N}^{1999 - 1978} = \sum\limits_{i = 1}^{50} {B_{Ni}^{1999 - 1978} r_{Bi}^{1999 - 1978} }$$

(8a)

$$A_{N}^{1999 - 1978} = \sum\limits_{i = 1}^{50} {B_{Ni}^{1999 - 1978} A_{Bi}^{1999 - 1978} }$$

(8b)

$$x_{N}^{1999 - 1978} = \sum\limits_{i = 1}^{50} {B_{Ni}^{1999 - 1978} x_{Bi}^{1999 - 1978} } .$$

(8c)

Analogous expressions apply to the changes between 2014 and 1999.

All the expressions we have derived thus far have been in absolute terms, but we will convert these to percentages for the purposes of presenting our results. The percentage contribution of the change in the racial disparity in state i to the change in the national black incarceration rate between 1999 and 1978 is the product of the percentage contribution of the state’s black incarceration rate to the national rate, $B_{Ni}^{1999 - 1978}$, times the percentage contribution of the racial disparity to the change in the state’s black incarceration rate, $r_{Bi}^{1999 - 1978}$. Suppose $B_{Ni}^{1999 - 1978}$ provides 5% of the national increase in black incarceration between 1978 and 1999 and $r_{Bi}^{1999 - 1978}$ provides 20% of the increase in the state’s black incarceration rate. Then, 1% (= .2 × .05) of the increase in the nation’s black incarceration rate is due to the increase in the racial disparity in state i.

Table 3 shows the state level contributions to the increase in black incarceration between 1978 and 1999, while Table 4 shows the state level contributions to the decrease from 1999 to 2014. On both tables these contributions are aggregated to both the regional and national levels.²¹ Between 1978 and 1999, the black incarceration rate increases by about 2400 per 100,000. The dominant factor in explaining the national increase in black incarceration between 1978 and 1999 is the overall increase in incarceration rates experienced in the United States. This explains 100% of the increase in black incarceration during this period.²² The rise in incarceration interacted with a large preexisting racial disparity present at the beginning of our sample in 1978, but this racial disparity does not worsen significantly over this period. More than 1/5 of the entire national increase is due to increases in the incarceration rates in Texas and California. Texas is in the South and increases in incarceration in the South led to fully 50% of the national increase in black incarceration. Note, however, that for every region of the country, increases in the overall incarceration rate contributed positively to the national increase in black incarceration. It is also notable that while the racial disparity is not contributing greatly to the increase in black incarceration during the 1978–1999 period, the worsening racial disparity in California does contribute 2.2% towards this increase. In no other state does an increase in the racial disparity contribute as much as 1% towards the national increase. Roughly half of all the states increased their black-to-white ratio while the other half decreased it over the 1978–1999 period.

Table 3 Decomposition of the increase in the national black incarceration 1978–1999^a

Table 4 Decomposition of the decrease in the national black incarceration: 1999–2014^a

During the 1999–2014 period, black incarceration fell nationally by 713 per 100,000. A positive entry during this period means the factor in question is contributing to the decline in black incarceration. Nationally, over 50% of the reduction in black incarceration is due to reductions in the racial disparity. It is notable, however, that reductions in the overall incarceration rate in California and Texas combined to explain over 50% of the reduction in the black incarceration rate during this period. When aggregating to the national level, this is partially offset by many states having an increase in their overall incarceration rate. California again stands out in terms of the racial disparity, as this contributes − 3.4% to the decline in black incarceration during this period, because the racial disparity significantly increased in California between 1999–2014. California is the only state in which an increased racial disparity is a significant factor in both the 1978–1999 and 1999–2014 periods. An increased racial disparity in Texas is significant in the 1999–2014 period, but this is due to a temporary jump down in the racial disparity in Texas in 1999. (This can be seen in Fig. 3, which we discuss below.) Reductions in incarceration in New York contribute 14% to the overall drop in black incarceration, but this was accomplished without increasing the black–white disparity in the state. This stands in contrast to California. There are 5 other states which increased their black-to-white ratio over the 1999–2014 period (Colorado, Idaho, Nebraska, New Mexico, and Vermont), but the total effect from these states is less than 1/3 of the effect of the increased racial disparity in California.

The key takeaway from these tables is that an increase in the overall incarceration rate is the dominant cause of rising black incarceration between 1978 and 1999, while a falling racial disparity is the most significant source of the fall in black incarceration between 1999 and 2014. An increase in overall incarceration in the South and Midwest are the biggest regional drivers of the increase in black imprisonment during 1978–1999, while decreased racial disparities in the South and Midwest are the biggest regional drivers of the decrease in black imprisonment during the 1999–2014 period.

Estimating Structural Change Points in the Racial Disparity

In this section, we estimate the structural change points in the black–white incarceration rate ratio for the U.S. as a whole and for each individual state.²³ Finding these break points can provide clues as to the policies or events (e.g., the crack epidemic or the 1994 Crime Bill) that altered racial disparities. If there are no shifts in the value of this ratio in the years subsequent to year t, it is difficult to claim that a policy implemented in t affected the relative incarceration rate. This is important because (except for California and a few small states) we do not find any sustained positive shifts in the state data in the years following the enactment of the 1994 Crime Bill.

In the traditional time-series literature, it is standard to use a dummy-variable approach to estimate structural breaks. However, this method is not particularly suited for estimating the change points in the statewide black-to-white incarceration ratios. To explain, suppose that one of our racial disparity variables is known to have n breaks that occur at dates $\tau_{1} , \tau_{2} , \ldots ,\tau_{n}$. As such, the usual approach is to construct n dummy variables to form:

$$r_{i}^{t} = a_{0} + b_{0} t + \mathop \sum \limits_{k = 1}^{n} a_{k} D_{k} + e\left( n \right)_{t}$$

(9)

where $r_{i}^{t} = B_{i}^{t} /W_{i}^{t}$ is our measure of the racial disparity in period t, $D_{k}$ is defined as a dummy variable such that $D_{k} = 1$ if $t > \tau_{k}$, and is 0 otherwise, n is the number of breaks, t = 1978, …, 2014, a₀, b₀ and the a_k are the parameters to be estimated, and $e\left( n \right)_{t}$ is the approximation error. The notation $e\left( n \right)_{t}$ is designed to point out that the approximation error is a function of the number of dummy variables included in (9).

Of course, it is possible to incorporate other explanatory variables and/or lagged values of $r_{i}^{t}$ into the model. However, since the break dates (i.e., the values of the various $\tau_{k}$) are unknown to us, this method would necessitate estimating the break dates using a methodology such as the Bai and Perron (1998, 2003) procedure. Specifically, Bai and Perron (1998, 2003) develop an efficient algorithm to estimate every possible combination of break dates, select the combination with the greatest likelihood, and test the statistical significance of the break coefficients. However, most of the changes in our data set are actually smooth shifts so that this dummy-variable approach (i.e., assigning a break to a particular point in time) is inappropriate. To explain, the dashed line in Panel a of Fig. 2 is the time series of the actual black-to-white incarceration rate ratio when aggregating over all states. The solid line shows the fitted values using the dummy-variable approach allowing for a maximum of five potential structural breaks.²⁴ Since the actual series steadily rises from 1987 to 1994, the Bai–Perron methodology best represents this gradual increase using a step function with sharp breaks in 1987 and 1994. Similarly, the steady smooth decline after 1997 is represented by a step function with breaks in 1998, 2002 and 2009. The point is that the dummy variable approach is misspecified in the presence of smooth shifts. In particular, the number of breaks is likely to be overestimated by a step function and the associated break dates are problematic.

Fig. 2

Actual and fitted U.S. black/white incarceration rates. a Fitted using the dummy variable approach, b fitted using a Fourier approximation. Note: Actual in state custody

Since nearly all of the shifts in each state’s black-to-white ratio data are gradual rather than sharp, we used a more recent approach specifically designed to capture smooth structural change. Moreover, we wanted to smooth the data since some of the series contained values that seemed overly erratic (e.g., the values in Panel a of Fig. 2 show a sharp decline in 2001 followed by a sharp increase in 2002). In particular, we utilized the type of Fourier series approximation developed in Enders and Lee (2012). By way of background, note that under very weak conditions, a Fourier approximation (i.e., a sum of simple sine and cosine waves) is able to represent any absolutely integrable function to any degree of accuracy. For each of our time series variables, $r_{i}^{t}$, it is possible to use a Fourier approximation to capture its behavior as:

$$r_{i}^{t} = a_{0} + b_{0} t + \mathop \sum \limits_{k = 1}^{n} a_{k} \sin \left( {\frac{2\pi kt}{T}} \right) + \mathop \sum \limits_{k = 1}^{n} b_{k} \cos \left( {\frac{2\pi kt}{T}} \right) + e\left( n \right)_{t} ,$$

(10)

where a₀ is an intercept, b₀ is the slope of the time trend, n is the number of frequencies used in the approximation, k = 1, …, n are the frequencies of the trigonometric terms, a_k and b_k (i = 1, …, n) are amplitude parameters, and $e\left( n \right)_{t}$ is the approximation error shown as a function of the number of included frequencies n.²⁵

Some of the key features of the approximation are²⁶:

• 1. If n = T/2 the fit is perfect (since the number of coefficients equals the number of observations) and if all a_k = b_k = 0, the series has no breaks. Moreover, as the value of n increases from 1 to T/2, the approximation error necessarily declines as additional regressors are added to the model. Of course, in a regression framework it is not possible to include all n = T/2 frequencies since the resultant estimation would contain no degrees of freedom. As such, the use of the Fourier approximation transforms the usual problem of estimating the break dates into one of selecting the most appropriate frequencies to include in the approximation.

• 2. Both the sine and cosine functions have a maximum of +1 and a minimum of − 1. The coefficients a_k and b_k multiply the amplitude of these trigonometric waves. The parameter k represents the number of cycles the series makes over the sample period. For example, as T = 1, …, 37, sin(2πt/T) and cos(2πt/T) exhibit one complete cycle (i.e., the frequency k = 1) over the sample period while for k = 2, sin(2π2t/T) and cos(2π2t/T) exhibit two complete cycles. Near zero, sin(2πkt/T) acts like a straight line with a slope of +1 while cosine acts as a parabola opening downward. By appropriately aligning these sine and cosine functions–by using the least squares principle–it is possible to capture the behavior of almost any integral function by the appropriate choice of n.

• 3. Since the approximation improves as the value of n increases, the issue is to select the appropriate value of n. Enders and Lee (2012) and Enders and Jones (2014) show that a small value of n (n ≤ 3) usually works quite well in econometric applications since gradual breaks are low-frequency events. Moreover, the values of a_k and b_k to use in the approximation can easily be obtained by estimating Eq. (10) using ordinary least squares (OLS).²⁷

• 4. Note that a Fourier series approximation is an orthogonal basis that fully spans the domain of the series in question. This helpful for testing purposes because each term in the approximation is uncorrelated with every other term. Moreover, in large samples with serially uncorrelated and normal errors, Gallant and Souza. (1991) show that the coefficients of (10) can be tested using a standard t-test and/or F-test.

To explain our methodology, note that the dotted line in Panel b of Fig. 2 again shows the time series of the actual black-to-white incarceration rate ratio aggregated over all states. With T = 37, and for k = 1, 2, 3 we created the six variables: sin 1 = sin(2πt/37), cos 1 = cos(2πt/37), sin 2 = sin(2π2t/37), cos 2 = cos(2π2t/37), sin 3 = sin(2π3t/37), cos 3 = cos(2π3t/37) and estimated regression equations of the form of (10). Our regression results for the U.S. as a whole are:

$$\hat{r}_{us}^{t} = \mathop {7.34}\limits_{(58.6)} - \mathop {0.031t}\limits_{( - 5.03)} - \mathop {0.361}\limits_{( - 3.85)} \sin 1 - \mathop {0.742}\limits_{( - 12.5)} \cos 1$$

(11)

$$\hat{r}_{us}^{t} = \mathop {7.42}\limits_{(71.6)} - \mathop { 0. 0 3 6 {\text{t}}}\limits_{( - 6.75)} - \mathop { 0. 4 1 2}\limits_{( - 5.64)} \sin 1 - \mathop {0.737}\limits_{( - 18.9)} \cos 1 - \mathop {0.020}\limits_{( - 0.414)} \sin 2 + \mathop {0.261}\limits_{(6.70)} \cos 2$$

(12)

$$\hat{r}_{us}^{t} = \mathop {7.55}\limits_{{\left( {64.4} \right) }} - \mathop {0.042}\limits_{{ \left( { - 7.03} \right) }} t - \mathop {0.491}\limits_{{\left( { - 6.14} \right) }} \sin 1 - \mathop {0.731}\limits_{( - 19.4)} \cos 1 - \mathop {0.060}\limits_{{\left( { - 1.16} \right)}} \sin 2 + \mathop {0.268}\limits_{{\left( {7.12} \right)}} \cos 2 - \mathop {0.094}\limits_{{\left( { - 2.14} \right)}} \sin 3 + \mathop {0.001}\limits_{{ \left( {0.038} \right)}} \cos 3$$

(13)

where $\hat{r}_{us}^{t}$ denotes the fitted value of the black-to-white incarceration rate ratio aggregated for all states and t-statistics are in parentheses. The fitted values from (11) are shown as the solid line in Panel b of Fig. 2 and the fitted values from (12) are the shown by the long-dashed line.²⁸

Since our sample size is small and the errors are not normally distributed, we did not want to rely on t- or F-tests to select the most appropriate model or to eliminate any of the regression coefficients. Instead, we used a standard goodness-of-fit measure, the Akaike Information Criterion (AIC), to select the most appropriate number of frequencies.²⁹ The AIC values for Eqs. (11), (12), and (13) are − 0.031, − 0.926, and − 1.07, respectively. As such, we conclude that (13) best fits the data (− 1.07 < − 0.926) taking into account that it includes more regressors than the other equations. We then calculate the maxima and minima using the fitted Fourier values from Eq. (13). Since some of the series have several break points, each can be considered a local maximum or minimum. In addition, we identify the global maximum and minimum. In order to eliminate the possibility of fitting an outlier, a particular year is a local maximum (minimum) only if the previous 3 years and the following 3 years are each of a smaller (larger) value.

Notice that the Fourier approximation fits the data reasonably well except for the years 1995, 1998 and 1999 because the smooth Fourier function cannot readily capture very sharp changes in the data. Nevertheless, in order to properly calculate the local maxima and minima, we deemed it necessary to smooth out such seemingly erratic values in the series. More importantly, the Fourier function does fit the actual series reasonably well in that it mimics the fall in the racial disparity from the beginning of the sample until the early 1980s and also captures the subsequent rise peaking around 1996. Thereafter, the estimates series exhibits a steady decline through 2014. In order to gain some insight into the underlying reasons for the aggregate behavior, we again look at the data on a state-by-state basis.

We followed the same procedure for each of the state series in our data set and obtained the dates of the overall maximum and minimum for values for each series. Figure 3 shows the actual and fitted black-to-white incarceration rate ratios in four large states: California, Florida, Ohio and Texas. Florida, Ohio and Texas at least roughly fit the national pattern. Consistent with our earlier discussion, California runs strongly counter to the national trends. The disparity declines slightly from 1978 to the early 1990s and then doubles between the early 1990s and 2014. It is notable that California adopted its three strikes law in 1994. The data is consistent with this law having a strong racially disparate impact. However, while the data is suggestive, it in no way provides a conclusive test regarding the effects of the law.³⁰

Fig. 3

Black/white incarceration rates in four key states. a California, b Florida, c Ohio, d Texas

The results for all of the states are presented in Table 5. For each state, up to three local maxima and minima are provided. Note that a local minimum implies a subsequent upturn on the black–white racial disparity, while a local maximum implies a subsequent downturn in this disparity. Table 6 summarizes the data on local maxima and minima, while Table 7 summarizes the data on global maxima and minima.

Table 5 Structural change points in the racial disparity

Table 6 Local maxima and minima

Table 7 Global maxima and minima

There are thirty-eight local minima reached in the 1980s, while only 6 such minima are reached in the 1990s. Conversely, 14 local maxima are reached in the 1980s, while 37 local maxima are reached in the 1990s. This implies that in many states there was an upturn in the racial disparity in the 1980s and for many states there was a downturn beginning in the 1990s. In the 2000–2011 period, there are an equal number of local maxima and minima, 24. To get some potential insight on the effects of the 1994 crime bill on the racial disparity, Table 6 also summarizes the data for 1984–1993 and 1995–2004, the 10 years prior and subsequent to the passing of the law. In the 1984–1993 period, 31 local minima and 21 local maxima are reached. Thus, in the years prior to the passage of the bill, upturns in the racial disparity outnumber downturns by about 50%. In the 1995–2004 period, there are 9 local minima and 34 local maxima. Thus, in this period downturns in the racial disparity greatly outweigh upturns in the racial disparity. In 1995, the year after the 1994 crime bill takes effect, there are 13 maxima and 0 minima. It is also notable that the aggregate disparity (Fig. 2) turns down beginning in 1996 and declines until the end of our sample. While for many states a downturn in the racial disparity begins in the 1990s, this does not imply that the 1994 crime bill was causal in this pattern. In particular, the minima become less frequent by the late 1980s and maxima begin to occur by the early 1990s even before this bill is signed. However, there is no evidence based on these results that the 1994 bill aggravated the racial disparity.³¹

Table 7 summarizes the timing of the global maxima and minima. Twenty-seven global maxima are reached in the 1990s. During the 2000–2014 period, 29 global minima are reached, while only 7 global maxima are reached. For 26 states, 2014 marks the global minima for the racial disparity, so that in more than half the states the lowest value for the racial disparity is observed at the end of the sample period. The latest date for a global maximum is 2012, which is for California.³²

The general upturn in the racial disparity in the 1980s is likely due to the emergence of the crack epidemic. There is ample evidence that the epidemic had a racially disparate impact (Fryer et al. 2013; Evans et al. 2016; Bjerk 2017). In addition to the violence surrounding the drug trade, the sentencing disparity between crack and powder cocaine is consistent with an increased racial disparity in imprisonment during this period.³³ As the effects of the crack epidemic wane, we see a general downturn in the racial disparity beginning in the 1990s. However, California is a large state that runs counter to the national trend, with a significant worsening in its racial disparity beginning in the early 1990s and continuing until near the end of our sample. At the end of our sample in 2014, the black to white incarceration ratio is still extraordinarily high at 5.5. Note that over the first 21 years of our sample, from 1978 to 1999 this ratio actually rises slightly from 6.9 to 7.1. All of the net decline occurs in the last 15 years of the sample over which the ratio declines from 7.1 to 5.5. Whether or not this recent downward trend can be maintained is an open question.

Conclusion

This paper has largely been a descriptive analysis of how the racial disparity in imprisonment in the United States varies across states and time. The analysis focuses on non-Hispanic black and white males. Our sample period begins in 1978 at which time the black incarceration rate was 6.9 times the white rate. While this ratio initially falls, it begins a sustained increase in the early 1980s and peaks at a value of 8.1 in 1995. Since reaching this peak, there has been a sustained decrease in the racial disparity down to a level of 5.5 in 2014. While the sustained decrease in the racial disparity is welcome, it is still very large at the end of the sample period.

Between 1978 and 1999 there is a massive increase in the black incarceration rate from 1080 to about 3500. The racial disparity did not change much over this period and the increase in incarceration was overwhelmingly due to an increase in the overall incarceration rate. The very large increase in the black imprisonment rate is only partially offset during the 1999–2014 period, and during this period a reduction in the racial disparity explains over 50% of the fall in black imprisonment.

Many upturns in the racial disparity are concentrated in the 1980s and it is possible that this is connected to the crack epidemic. Many downturns in this rate occur in the 1990s including the period after the 1994 Crime bill became law. Whatever the other effects of this bill, there is no prima facie evidence that it exacerbated the racial disparity in imprisonment at the state level. For more than half the states, the lowest observed value of the racial disparity occurs in the last year of the dataset. While the racial disparity remains large, it is at least somewhat encouraging that it has generally been falling since the 1990s. It is notable that the experience in California runs counter to the national trend with a large increase in the racial disparity beginning in the early 1990s and continuing until near the end of our sample.

We have used a Fourier approximation to gain insight into the time series behavior of the racial disparity. This method can be widely applied within criminology whenever it is believed that the data exhibit gradual shifts over time. For example it can be applied to an analysis of overall incarceration rates rather than just the racial disparity. It can also be applied to data such as prison admissions, crime rates and arrest rates among others. A better understanding of the time series behaviors of these variables can, in turn, help provide insight into how federal and state policy impacts the criminal justice system.

Our approach is largely descriptive and it does not allow for formal testing of how policies such as truth in sentencing or three strikes laws have affected the racial disparity. These policies could be formally analyzed in a panel data analysis. The effect of these policies on incarceration has previously been analyzed in a panel setting, but to date this has not been done for the racial disparity.³⁴ Thus, this is a potentially fruitful avenue for future research. Such work may shed light into how the racial disparity in imprisonment can be further reduced.