Introduction

Racial disproportionality in prisons in the United States has long been a matter of major concern. This concern has become even more salient as other aspects of racial inequality in the criminal justice system have been drawn to the public’s attention. In recent years, the publication of Michelle Alexander’s The New Jim Crow (2010), has redrawn the focus on the racial differences among prisoners, especially those charged with drug offenses. Alexander’s argument targets the operation of all elements of the criminal justice system and includes implicit as well as explicit discrimination, claiming that the racial difference in prison is largely attributable to intentional emphasis on offenses that blacks commit. Alexander likens this intentional emphasis to the Jim Crow efforts in the South to control blacks during the Reconstruction period following the Civil War.

The concern about racial disproportionality in prison is stimulated by the high black incarceration rate (the number of black prisoners divided by the black population) that was reported to be about 5.6 times that of the white adult incarceration rate, excluding Hispanics and persons of other races, at yearend 2015 (Carson and Anderson 2016). This ratio is quite troubling, and would be so even in the absence of any discriminatory intent. One unfortunate consequence of this more intense involvement with the criminal justice system is distrust of the system, which diminishes cooperation and the overall effectiveness of the system.

It is easy to blame the higher rate of incarceration among blacks than whites on discrimination since few aspects of American life are entirely free of discrimination. However, the challenge of separating fact from rhetoric lies in being able to distinguish the degree to which the black disproportionality is simply a reflection of greater involvement in crimes that lead to prison or a consequence of discrimination at various stages of the criminal justice system. For some crime types or some regions of the country, the differences may be largely a reflection of involvement in crime. For other crime types and regions, there may be a large difference between involvement and incarceration, which warrants an appropriate investigation as to the factors contributing to the difference.

To account for the differences, one would like to compare racial differences in incarceration to differences in involvement, or perhaps at arrest, which often is the closest available indicator of involvement. This comparison has been done in the past for the nation as a whole (e.g., Blumstein 1982) and for individual states (e.g., Steffensmeier et al. 2011, for Pennsylvania). Repeating such analyses at the current time is difficult because the administrative data in prisons on which recent estimates have been made, include Hispanic inmates as a separate race, leaving the counts of black and white inmates to be based on non-Hispanic inmates only. In contrast, police records of arrest have no separate Hispanic category, so that counts of black and white arrestees include Hispanics. Establishing a comparison by race requires consistent measurement of white and black inmates and arrestees, regardless of their Hispanic origin.

The consequence of not having consistent measurement is highlighted by the National Academy of Sciences (NAS) report (Travis et al. 2014). Studies cited in the report did not take appropriate account of the race of Hispanic prisoners. This led to an inaccurate conclusion that the problem was getting worse. An important objective of our study is to introduce a methodology to account for the Hispanic inmate population and to provide corrected estimates of the link between racial differences in criminal involvement, arrest, sentencing and time served and racial disproportionality in the prison population.

In many quarters, racial disproportionality in prison is seen as a powerful indicator of discrimination throughout the criminal justice system. That is certainly a major theme of Alexander’s The New Jim Crow (2010). There are simply too many reports, often explicit and more often implicit, that indicate discriminatory actions by individuals or agencies of the criminal justice system. On the other hand, there are also too many indicators of differential involvement in serious crimes to attribute all of the racial disproportionality in prison to discrimination.

To the extent that the racial disproportionality in prison reflects a strong influence of racial discrimination and control, it would require considerable effort to repair. If it were more a matter of differential involvement in the kinds of crime that lead to prison, then efforts should be directed at addressing the social and economic conditions that contribute to differential involvement. Thus, understanding the many aspects of racial disproportionality becomes a matter of primary concern.

Identifying Racial Differences in Participation in Crime

As noted by Crutchfield et al. (1994), there remains within criminology continuing controversy concerning the presence, pervasiveness, and causes of racial differences. In their assessment of the literature, the authors conclude that differing outcomes may be traced to limitations of some studies to single stages (or decision points in the criminal justice system) and to single jurisdictions. Both limitations could well mask differences in treatment across stages or jurisdictions. By implication, the authors call for studies that examine the entire criminal justice system, from arrest to incarceration, and extend beyond single jurisdictions to account for variations in criminal justice responses in different contexts. Moreover, because there may be considerable variation among states in the processing of white and black offenders, arrestees, defendants, and sentenced inmates, studies that simply aggregate across states and jurisdictions may mask significantly different patterns of treatment by the police, prosecutors, and courts.

Consideration of the extent of differential involvement in crime by race, the underlying causes, and trends is not new. Sampson and Lauritsen (1997) provide an overview of the literature, data sources, and theories related to racial disparities in criminal offending. More recently researchers have examined the role of factors such as education and poverty in accounting for trends. Lafree and Arum (2006) assessed the impact of racial inclusiveness in schools on incarceration rates for 5-year cohorts of African Americans and whites born since 1930. They conclude that blacks educated in states where a higher proportion of their classmates were white experienced significantly lower incarceration rates as adults. Lafree et al. (2010) examined race-specific homicide arrest and census data on social, economic, and demographic conditions for 80 large U.S. cities from 1960 to 2000, and found substantial convergence in black-white homicide arrest rates over time, although this convergence stalled from the 1980s to the 1990s. They found that, since the 1960s, the racial gap in homicide arrests declined more substantially in cities that had greater reductions in the ratio of black-to-white single-parent families, as well as in cities that experienced greater population growth and increases in the proportion of the population that is black.

In our study, the goal is to provide an indication of how much of the racial disproportionality in state prisons can be attributed to differential involvement in crime and if this has changed over time. We build on the critiques of Crutchfield et al. (1994) by examining three stages, including arrest, sentencing and time served, for which data are available at the national and state levels. We examine these intermediate stages to assess the degree to which racial differences at each stage impact the racial disproportionality in prisons.

For ease of understanding, we address disproportionality in terms of “accountability.” If one could fully account for those differences in prison, i.e., 100% accountability, then one could reasonably attribute the differences in prison fully to differential involvement in crime. On the other hand, if accountability were only 50%, then differential treatment in the criminal justice system, including decisions to sentence to prison and to time served, accounts for half of the disproportionality, and would raise important concern over discrimination.

Most studies have examined arrest accountability at the national level; however, the degree to which race differences at arrest account for racial disproportionality in prison could well vary across the states. States could well display very different traditions or policies in controlling for discriminatory treatment. We address this issue by examining arrest and prison statistics in 42 individual states in which corresponding data are available and organize the states into regions.

Most studies of racial disproportionality in prison have used information on race at arrest as a proxy for differential involvement in crime. However, the link between criminal involvement and arrest may vary by crime type. For example, differential assignment of police to neighborhoods may contribute to racial bias in the risk of arrest. This bias may be greater for minor crimes than for more serious crimes. Arrests for minor crimes nevertheless contribute to a prior record, which is often an important factor when judges decide between a probation and prison sentence. Racial profiling and other explicit or implicit forms of police bias may also contribute to discriminatory arrest outcomes. But this may be less likely for the more serious crimes, which often involve extensive police or prosecutorial investigation before leading to conviction and sentencing to prison.

Comparisons between reports of perpetrator race in victim surveys and arrests have been made for various cities, and the racial distributions have generally been found to be consistent (Hindelang 1978). However, due to the limited coverage of many kinds of crimes, the victim surveys have not been very helpful for generating estimates of racial participation in crime overall. Moreover, the sample of victims in any geographic location is often extremely small and cannot easily be used to provide any indicator of geographic variation.

We address the issue of differential involvement in crime by drawing on the reports of victims in BJS’s National Crime Victimization Survey (NCVS). Such information is only meaningful for violent crimes, in which the victim sees the offender. For these crimes, if the offenders’ race as reported by the victim is found to be consistent with the race of arrestees, accountability may be more directly linked back to criminal involvement, and differences in arrest may reflect processes at a later stage in the criminal justice system. Other studies have examined this relationship, including Tonry (1995) and Lafree et al. (2006). We extend the analysis using 2010 NCVS data and including incidents in which victims report multiple perpetrators.

Estimating the Degree to which Racial Disproportionality in Prison Is Accounted for by Racial Differences in Arrests

The linkage of the race of arrestees to that of prisoners was addressed initially by Blumstein (1982). He found national differences in arrest accounted for 80% of the racial differences in state prison in 1979. He found that the percentage differed considerably across crime types. Arrest differences for murder and robbery almost totally accounted for the prisoner differences by race; whereas arrest differences for drug offenses accounted for only about 50% of the prisoner differences. The overall estimate of 80% represented the sum of the attribution to each individual crime type, weighted by the fraction each represented in the prison population.

Blumstein (1993) repeated the analyses for the state prison population in 1991 and found that the percentage attributable to racial differences in arrest had dropped from 80 to 76%. The decline was not linked to changes in the contribution of any specific crime types, but was largely linked to an increase in drug offenders from 5% of the state prison population in 1979 to 20% in 1991. Because racial differences in drug arrests provide the poorest explanation for racial differences in prison, the increased weight of drug offenses in the total estimate resulted in the overall decrease from 80 to 76%.

Harris et al. (2009) extended the analysis of Blumstein (1982) by examining state prison admissions and stock populations, and by including available data on Hispanics at each stage. Using data from Pennsylvania for 2003–2007, their results indicated that the representation of blacks, whites, and Hispanics among offenders admitted to state prison and in the prison population corresponds closely to their representation in arrest statistics. Using arrests as a marker of violent offending, they concluded that the overrepresentation of blacks among offenders admitted to state prisons occurs because arrests indicate that they commit a disproportionate number of frequently imprisoned (i.e., violent) crimes.

Ulmer et al. (2016) further extended the analysis of disproportionate imprisonment of blacks and Hispanics. Using 2005–2009 federal court and Pennsylvania state court data, they investigated the extent to which the disproportionate punishment of black and Hispanic men can be attributed to unexplained disparities in local sentencing decisions, compared to the extent to which such differences are mediated by legally prescribed factors set by policy or other case-processing and extralegal factors. Their findings suggest that most disproportionality (particularly in Federal courts) is determined by processes prior to sentencing, especially sentencing policies that differentially impact minority males.

Steffensmeier et al. (2011) argued that prior studies that rely on white and black national crime statistics are confounded by absence of data on Hispanic offenders. Using data from California and New York that identify Hispanic arrestees, they generated estimates of black involvement in violent crime for the nation as a whole that account for the effects of the rapid increase in Hispanic arrests in recent years. Their results suggest that little change has occurred in the black share of violent offending in both UCR and NCVS estimates during the last 30 years. In addition, racial imbalances in arrest as compared with in-stock incarceration levels across the index violent crimes are small and show mixed patterns of both under-incarceration and over-incarceration of blacks. Although limited to index violent crimes, their analysis underscores the need to examine the racial imbalance among drug offenders due to their large impact on incarceration, especially on black offenders.

These previous studies clearly underscore the need for an adequate accounting for Hispanic arrests, information not included in the FBI’s UCR records, to relate to prison data that often include Hispanics as a separate racial category. Also, the studies highlight the need for a full accounting and careful matching of crime types at arrest and in prison, beyond the index violent crimes. They also call attention to the need for multi-state comparisons of the arrest and incarceration patterns by race, in light of differences in justice systems among the states.

Measuring Accountability

The approach we use to estimate the degree to which racial disproportionality in prison is attributable to racial differences in arrest is straightforward. We define the variable X as the fraction of the disproportionality in prison that is accounted for by racial differences in arrest; we express this in terms of “accountability.” To the extent that X equals 100%, it would suggest that racial differences in arrest fully account for the racial disproportionality in prison. If X exceeds 100%, it would suggest that blacks are underrepresented in prison for that crime type. If X is less than 100%, it would suggest that blacks are overrepresented in prison for that crime type.

The value of X is calculated as the ratio of the black-to-white arrest rate divided by the ratio of the black-to-white incarceration rate:

$${\text{X = }}\frac{\text{ratio of black to white arrest rate}}{\text{ratio of black to white incarceration rate}}$$

Here, the numerator is the ratio of the number of black arrestees to the black resident population divided by a similar ratio for white arrestees to the white resident population.

The denominator is defined similarly for prisoners. Since the rates in the denominators of black and white resident populations cancel out, the formula is simplified:

$${\text{X = }}\frac{{\text{black percentage in prison} * (1 - {\text {black percentage of arrestees}})}}{{\text{black percentage of arrestees} * (1 - {\text {black percentage of prisoners}}) }}$$

or

$${\text{X}} = \frac{\text{P*(1 - A)}}{\text{A}*(1 - \text{P})}$$
(1)

where A is the black percentage of the black and white arrests, and P is the black percentage of the black and white prisoners.

This ratio can be calculated separately for each crime type (i), leading to an estimate Xi. Then by weighting each Xi by the fraction of prisoners serving time for crime type (i), or Fi, we can calculate X for the nation as the sum of \({\text{X}}_{\text{i}} *{\text{F}}_{\text{i}}\) so that \({\text{X}} = \sum\nolimits_{\text{i}} {({\text{X}}_{\text{i}} *{\text{F}}_{\text{i}} )}\).

Obtaining a Consistent Set of Crime Types for Arrestees and Prisoners

To generate estimates of the black-white ratio for the crime types in prison (Pi for crime type (i) and relate those to the black-white ratio of arrests (Ai) for comparable crime types, it is necessary to establish consistency between the crime types recorded for prisoners in BJS’s National Corrections Reporting Program (NCRP) and those recorded for arrestees in the FBI’s Uniform Crime Reports (UCR).

For some crime types (e.g., murder and non-negligent manslaughter, robbery, and burglary), the NCRP and UCR classifications are comparable. In addition, assault includes both simple and aggravated assault in NCRP but only aggravated assault in UCR; however, the NCRP assault category is limited to prisoners with sentences of greater than 1 year, which would largely exclude offenders sentenced for simple assault, who typically receive shorter sentences, jail sentences or probation sentences upon conviction. Attempted murder or manslaughter and conspiracy to commit murder or manslaughter are classified as aggravated assault in both the NCRP and UCR.

For other crime types, some accommodation is needed. For example, in NCRP, rape includes a variety of other forms of sexual assault, while rape in UCR is more restricted. The correctional codes, from which NCRP codes are derived, often are inconsistent and lack sufficient detail to uniquely classify rape as separate from other forms of sexual assault. There are a variety of other violent offenses in NCRP, including negligent manslaughter, kidnapping, reckless endangerment, blackmail, extortion, and criminal trespass against a person, which are included as “other violent offenses.” However, in UCR, “other violent offenses” is limited to offenses for which arrest statistics are reported, including other assaults (except aggravated assault).

Among the property offenses in both NCRP and UCR, we include motor vehicle theft with larceny. Under “other property offenses,” both NCRP and UCR include forgery, fraud, counterfeiting, and embezzlement as well as arson; buying, receiving, possessing and distributing stolen property; and destruction of property (vandalism). In NCRP, “other property offenses” also includes hit and run driving with property damage only and unauthorized entry or criminal trespass.

Drug offenses represent the single largest category in state prisons. We include drug trafficking and possession (including fraudulent prescriptions and paraphernalia) in accordance with the listing in the UCR.

“Public-order and other/unspecified offenses” is a broad category that includes regulatory offenses, weapons offenses; offenses against the courts or legal process (e.g., obstruction and contempt); probation and parole violations; failure to appear; vice, morals and decency; drunkenness; vagrancy; invasion of privacy; liquor law violations; driving under the influence; and bribery and conflict of interest.

For purposes of tracking changes over time, these aggregations lead us to partition prisoners into the following 10 crime types:

  • Murder and non-negligent manslaughter.

  • Rape and other sexual assaults.

  • Robbery.

  • Aggravated assault.

  • Other violent offenses.

  • Burglary.

  • Larceny and motor vehicle theft.

  • Other property offenses.

  • Drug offenses.

  • Public-order and other/unspecified offenses.

Based on these crime types, which provide reasonable consistency between the arrest and prison data, we are able to generate a national estimate of X. We use state prison data from 2011 and UCR arrest data from 2010 to take into account the lag between time of arrest and incarceration.Footnote 1

NAS Report on More Recent Estimates of Disproportionality Attributable to Arrest Differences

The National Academy of Sciences report on the growth of incarceration (2014) analyzed the causes and consequences of the growth of incarceration. Table 3 of the report presented Blumstein’s estimates for 1979 (80%) and 1991 (76%) along with more recent estimates that were much lower,Footnote 2 61% in 2004 (Tonry et al. 2008) and 55% in 2008 (Baumer 2010). The panel thereby noted that the disproportionality in prison attributable to differential involvement in arrest has decreased over time. The NAS report suggested that discrimination has increased based on its observations that the more recent estimates are appreciably lower than those reported by Blumstein. The panel observed that “racial disparities in imprisonment became much worse in the twenty-first century compared with that found by Blumstein for 1979 and 1991″ (p. 96). The panel report went on to state that “the reason for increased racial disparities in imprisonment relative to arrests is straightforward: severe sentencing laws enacted in the 1980s and 1990s greatly increased the lengths of prison sentences mandated for violent crimes and drug offenses for which blacks are disproportionately often arrested” (p. 96).

In light of the near stability between 1979 and 1991, it was surprising to see the dramatic drop from 76% in 1991 to 61% in 2004, and then to 55% in 2008. This was particularly surprising in the face of a drop of only 4% when the fraction of drug offenders in prison quadrupled from 5% in 1979 to 20% in 1991 and did not change very much after that. This drop from 1991 warranted a further examination of the methodologies used in the Baumer and Tonry and Melewski studies on which the NAS panel’s conclusions were based.

Accounting for the Race of Hispanic State Prisoners

A complication in establishing comparability between arrest and prisoner counts is in how the race of Hispanic arrestees and prisoners is taken into account. In 2011, an estimated 282,353 state prisoners (with sentences of more than 1 year) were identified as Hispanic. They represented 21% of all sentenced state prisoners (Carson and Golinelli 2013). However, the collection of data on the Hispanic origin and race of inmates has changed over time.

In 1978 the National Prisoner Statistics (NPS) series began collecting data on the race and Hispanic origin of prisoners in separate items. Separate counts were provided in annual publications through 1998. In 1999, with the increasing presence of Hispanics in state and federal prison, the number of departments of corrections that recorded Hispanic origin and race as a single designation, and the growing number of Hispanic inmates reported as race “unknown,” BJS began collecting race and Hispanic origin as a single item. This change was consistent with the Office of Management and Budget’s Statistical Policy Directive Number 15 (1997) for collection of data from administrative data, where self-identification was not feasible.

The FBI’s UCR arrest statistics record race as white, black, American Indian or Alaska Native, and Asian or Pacific Islanders, but ignore Hispanics altogether. Since the large majority of Hispanics are likely to have been classified as white or some other race when arrested, not counting Hispanics among arrestees would significantly understate the degree of disproportionality in prison due to racial differences in arrest. To provide a comparison, researchers must rely on estimates of the racial identification among Hispanic inmates.

Blumstein’s studies (1982, 1993) relied on BJS inmate surveys in which data on inmate racial self-identification were collected separately from Hispanic origin. In these surveys, conducted in 1979 and 1991, Census Bureau interviewers obtained the racial identification of each respondent. In contrast, Tonry and Melewski relied on BJS published estimates by offense, race, and Hispanic origin, which were derived from administrative data in the NPS and NCRP. However, these estimates separated non-Hispanic whites, non-Hispanic blacks and Hispanics. Baumer (2010) attempted to reproduce Tonry and Melewski and extend the analysis to 2008. However, his estimates were not based on BJS published estimates, and were not adjusted for the race of Hispanic inmates.

Tonry and Melewski make no mention of Hispanic prisoners in their presentation of their estimates; however, a comparison of their estimates (provided in Table 2 of their report) with published BJS data suggests that they did account for Hispanic prisoners (Sabol et al. 2007). Of the 242,700 Hispanic inmates at yearend 2004, they allocate 75% as white and 25% as black. This inference is drawn from the following counts for the 6 major offense categories, shown in Table 1. Based on a 75/25% allocation of Hispanics, the percentages by race match those published by Tonry and Melewski in their report.

Table 1 Tonry–Melewski’s allocation of Hispanic prisoners in 2004
Full size table

We have re-estimated the values of X here with alternative allocations of Hispanic prisoners to white and black categories. Tonry and Melewski’s allocation is much more heavily black than is merited, and clearly results in an appreciably lower X value. In addition, we have drawn on published BJS data for more recent estimates of the racial distribution of state prisoners at yearend 2008 to assess Baumer’s calculations of X.

Revision of the 2004 and 2008 Estimates of X

To accurately estimate X, it is necessary to allocate Hispanics to the white and black prisoner counts. This could be done by relying on Census data that provide distinct choices for race, including white, black or African American, American Indian and Alaska Native, Asian, Native Hawaiian and Other Pacific Islander, and two additional categories (i.e., “some other race” and “two or more races”). A separate ethnicity category permits individuals to indicate “yes” or “no” with respect to “Hispanic, Latino, or Spanish origin.” Of those who selected the Hispanic ethnicity in the 2010 Census of Population and Housing, 94.0% selected one race, 53.0% of those selected white, 2.5% black, and 1.9% other races. A full 36.7% of those identifying as Hispanic selected “some other race.” Another 6.0% of Hispanics selected “two or more races” (Humes et al. 2011).

We could apply the racial self-identification of Hispanics in the general population to the prisoners identified as Hispanics in the prisoner counts. This would result in an allocation of 93.8% of the 282,353 Hispanic state prisoners in 2011 to white, 4.4% to black, and 1.8% to one or more of the other races. However, Hispanics in prisons may differ from those in the resident population. A more direct approach would be to base the allocation on the self-reported race of Hispanics held in prison.

BJS has conducted numerous surveys of state prisoners in which Hispanic inmates have been asked to identify their race. Based on our analysis of data from the four most recent BJS surveys (i.e., Survey of State Prison Inmates, 2004, and the National Inmate Survey, 2007, 2008–2009 and 2011–2012), we have concluded that Hispanic prisoners are best allocated as 80% white, 10% black, and 10% other (representing primarily American Indians and those identifying as two or more races). Although the exact percentage who may be classified as white varies across surveys (84% in 2004; 85% in 2007; 82% in 2008–2009; and 84% in 2011–2012), we chose a more conservative allocation (80%), so as to not overestimate the number of white prisoners among Hispanic inmates who reported no racial identity.

With this allocation, we have generated in Tables 2 and 3 our estimates of the degree to which arrests account for the racial disproportionality in the nation’s state prisons in 2004, 2008, and 2011. We use the 10 crime types and compare racial differences in arrest from the previous year with the yearend state prison populations.

Table 2 Estimation of Xi and X for 2004 and 2008, allocating Hispanic prisoners as 80% white/10% black
Full size table
Table 3 Estimation of Xi and X for 2011 with different allocations of Hispanic prisoners
Full size table

For each year, we provide an estimate of Ai, Pi, and Fi (the fraction of prisoners serving time for crime type i). In Table 2 we show these estimates after allocating Hispanic prisoners as 80% white and 10% black and the resulting X values for 2004 (72%) and 2008 (71%), the two years addressed in the NAS report. In Table 3, we first show the calculations and the X values that would result under three approaches to taking Hispanics into accountFootnote 3: (1) not at all (as was done by Baumer), (2) allocating Hispanics as 75% white and 25% black (as was done by Tonry and Melewski), and (3) 80% white and 10% black, our estimate based on past inmate surveys. The resulting estimates of X for each of the allocation methods are 50, 64, and 70% respectively.

We find that allocating Hispanic prisoners with 80% as white and 10% as black generates significantly higher estimates of X than when Hispanics are excluded. Moreover, Tonry and Melewki’s allocation of 75/25% significantly understates the estimate of X for 2004. The estimate for X in 2011 reaches 70%, well above the estimates in the NAS report and does not show the pattern of steady decline suggested in the report. The results suggest that approximately 70 to 75% of the racial disproportionality among state prisoners is accounted for by black-white differences in arrests. Also, the Xi values for the individual crime types, especially the most prevalent ones, are consistent over time, with drug offenses at the lowest value of about 50%, and are similar to Blumstein’s earlier estimates.

Many factors could be contributing to these race differences, all of which may be contributing to differential involvement in crime. These include differential arrest vulnerability between whites and blacks for certain types of crime as well as factors often identified as “root causes” of involvement in crime. These factors include socioeconomic differences, educational opportunities, job opportunities, and cultural differences, all of which have been the subject of past criminological investigation.

However, differential involvement in crime remains a significant factor that should be accounted for when considering racial disproportionality in prison. Factors that account for the unexplained residual of 25 to 30% may be numerous, including differences in prior record that affect sentencing decisions, and differences in socioeconomic status that impact the ability of defendants to hire counsel. Following arrest, overt or more subtle implicit discrimination by prosecutors, judges, parole boards, and others may also affect sentencing decisions and length of stay in prison, which in turn will produce racial disproportionality in prison.

Racial Disproportionality in the 2011 Prison Populations by State

Past studies of disproportionality have largely been focused on national data on arrest and incarceration by race. Yet, the national data are an aggregation of data generated from decisions made at the local and state level. Understanding variations at the state level may provide further insight into the nature of the racial disproportionality in prison. We do this using detailed data by race and crime type that BJS collects from state departments of corrections and comparing these to state-level arrests by race and crime type from the UCR.

State-Level Incarceration Rates by Race

We look first at the incarceration rate of blacks compared to whites at the state level. For each state, we calculate a ratio of the black incarceration rate to the white incarceration rate:

$${\text{Incarceration}} - {\text{rate Ratio}} = \frac{\text{black prisoners/black population}}{\text{white prisoners/white population}}$$
(2)

As with prisoner data on race at the national level, it is necessary to allocate Hispanics to the white and black prisoner counts. At the state level we use data from the annual National Prisoner Statistics series, which since 1999 has collected race and Hispanic origin as a single item (BJS 2013). This necessitates the allocation of Hispanic inmates to racial categories as was done in our previous analyses of national level data. However, while appropriate as a national estimate, the allocation of Hispanic inmates as 80% white and 10% black across all states would mask state variations among Hispanic inmates.

To provide more accurate state-specific estimates by race, we have utilized data from the National Inmate Survey 2007, 2008–2009 and 2011–2012. These data provide estimates of self-reported racial identification among Hispanic inmates in each state that reports Hispanics as a separate racial category. Estimation procedures differed somewhat by year, due to the shift from two items in 1990 to a single item in 2011 and differences in the number of inmates missing race and Hispanic origin altogether.Footnote 4

The state-specific incarceration-rate ratios for 2011 are shown in Table 4, along with the rates in 1990. The table lists the states with the 10 highest and 10 lowest black-to-white incarceration-rate ratios. We have calculated these ratios including in each state’s count of black and white prisoners that state’s Hispanic prisoners based on their racial identification. The ordering of the states among those with the 10 highest and 10 lowest ratios is quite similar for the 2 years and cover a rather broad range.

Table 4 Black-to-white prison incarceration-rate ratios, adjusted for race of Hispanic prisoners, by state, 1990 and 2011
Full size table

The black-white incarceration-rate ratio for the 50 states combined was 4.7 in 2011, down considerably from 6.8 in 1990. In 2011, states range from a high of 11 or more in Iowa, Minnesota, and Wisconsin. The 10 states with the lowest ratio cover the range from Mississippi at 3.2 to New Mexico at 4.1 and are predominantly states in the South and Southwest. States with the highest ratios are predominantly from the North and the Midwest, including Iowa, Minnesota, Wisconsin, Vermont, New Jersey, and Pennsylvania, all states with one or more large cities, where the majority of the state’s black population lives.

The black-to-white incarceration-rate ratios for 1990 are appreciably higher—ranging from 20.0 in Minnesota and 16.0 in Iowa to 4.5 or lower in Arkansas, Mississippi, South Carolina and Tennessee. However, the distributions of the states in both years are quite similar, with the larger North and Midwest states at the high end and the South and Southwest dominating the low end.

These rankings by state might be surprising if we simply presumed that the traditional racial prejudice of the South would contribute to a higher ratio. However, these ratios are strongly related to the overall incarceration rates in each state and the geographic distribution of black residents. States with low overall incarceration rates and a concentration of black state residents in high-crime urban areas have the highest black-to-white incarceration rate ratios; while states with high incarceration rates and broader distribution of black residents across the state have the lowest ratios. The relationship is reflected in the strong negative correlation of −.55 between a state’s black-to-white incarceration-rate ratio and its overall incarceration rate.

State and Regional Differences in X by Crime Type

To shed light on these state-level variations, we can examine state-level differences in the degree to which racial disproportionality in prison is accounted for by arrest. We have generated estimates of Xij by crime type (i) in each state (j) using formula (1), developed earlier.

We have assembled data on the prison population by race and crime type in the 42 states that participated in the 2011 NCRP. The 2011 NCRP data allowed us to examine 14 crime types, as opposed to the 10 for which the national estimates were previously developed. Forgery, fraud, counterfeiting, and embezzlement (FFE) were separated from “other property offenses” and placed into a distinct category. Drug offenses were divided into drug trafficking and drug possession. In drug possession, we included unspecified drug offenses, miscellaneous drug offenses (such as paraphernalia, tools, and illegal prescriptions) and all other drug offenses other than those characterized as drug trafficking. Weapons offenses and DWI/DUI and liquor law violations were separated from “public-order and other/unspecified offenses” and placed into distinct categories.

We have excluded four states (Montana, North Dakota, Vermont, and Wyoming) from the calculations because of the small number of blacks in their state resident and yearend prison populations. All had fewer than 150 blacks in the NCRP prison data, and allocating those to the 14 crime types would result in small numbers and unreliable estimates. Florida was excluded due to the absence of UCR arrest data. Hawaii, Illinois, and Maine were excluded because they did not submit yearend prisoner data to the NCRP. The remaining 42 statesFootnote 5 accounted for 88% of the total state prison population in 2011.

As with prisoner data on race at the national level and NPS aggregated data at the state level, it was necessary to allocate Hispanics to the white and black prisoner counts by offense in NCRP. Unlike the annual NPS data collections, the NCRP collects race and Hispanic origin of inmates in separate items. Within each state, data were estimated based on the nature of the missing data: (1) among Hispanic inmates with missing race, the NIS self-reported racial identification of Hispanics by state was utilized; (2) among non-Hispanic inmates with missing race, the racial distribution of inmates with reported data was used; and (3) among inmates with missing data on both race and Hispanic origin, the combined distributions after allocations from Hispanic and non-Hispanics were used.Footnote 6

In the initial estimates of Xij, we found that some states had anomalous values of Xij, as a result of unreliable counts of arrests or prisoners by crime type. In order to avoid distortion of our aggregate estimates as a result of those states, we first identified as “anomalous” those states that were more than two standard deviations (2σ) from the 42 state mean, recalculated the mean and standard deviation (σ) based on the remaining states, and then assigned the Xij of the anomalous states to the closer 2σ value.

For each crime type considered, we found at most three anomalous states. Table 5 lists the anomalous states identified for each crime type, the mean value of Xi before the anomalous state was adjusted, the mean value after the anomalous state was adjusted, and the percentage change in the value of Xi by modifying the anomaly or anomalies. For 13 of the 14 crime types, the anomalous state had a high value of Xij and so removing its contribution lowered the value of Xi. In two crime types, the negative change was 5% or greater: drug trafficking (down 7.5%) and weapons (down 5.3%). Increases resulting from adjusting the anomalous states occurred in only one crime type, murder (up 0.3%). This suggests that our estimates of the values of Xi and X will be conservatively low if the values of Xij in the anomalous states, were accurate, even if larger than the values in other states.

Table 5 States with anamolous Xij values by crime type in 2011
Full size table
Table 6 Estimation of Xi values and X for arrests-to-prison populations in 42 states for 2011 by crime type
Full size table
Table 7 Estimation of Xci values and Xc for arrests-to-new court commitments in 40 states for 2011 by crime type
Full size table
Table 8 Estimation of Xti values and Xt for new court commitments-to-prison populations (or time served) in 40 states for 2011 by crime type
Full size table
Table 9 National and regional values of Xi for arrests-to-prison populations in 42 states for 2011 by crime type
Full size table
Table 10 Percentage black among violent crime offenders and arrestees, NCVS and UCR, 2010
Full size table

With this mapping, we were able to calculate the Xij values by crime type (i) in each of the 42 states (j). These values are provided by region in Appendix Tables 11, 12, 13, and 14.

Estimating Xi for Individual Crime Types

Based on the Xij values, we calculate Xj for state j as the sum over the 14 crime types, weighting each crime type by its proportionate representation (Fij) in state j’s prison population.Footnote 7 We then calculate the national values Xi for each crime type as the weighted sum of the state-specific values of Xij, with the correction for the anomalous states and the weighting by each state’s prison population. A national estimate of X is then generated by summing the Xi values weighted by the aggregate representation of crime type i (Fi) in the 42 states’ prisons combined.Footnote 8

Table 6 presents the estimated values of Xi for each of the 14 crime types for the 42 states. These values represent the link between the racial composition of arrests and the corresponding prison population by crime type. The table also provides the estimates of Fi, the proportionate representation of each crime type i in the 42 states for which we have data. The sum of \({\text{X}}_{\text{i}} *{\text{F}}_{\text{i}}\), or 69%, is then the estimate of the aggregate national value of X. The Xi and X values encompass all of the stages of the criminal justice system from post-arrest to the prison population.

In Table 6, we see that the most prevalent violent offenses, as shown by their Fi values, are rape and other sexual assault; murder and non-negligent manslaughter; robbery, and aggravated assault, collectively accounting for 48.4% of the states’ prison population. Adding prisoners held for other violent offenses and weapons offenses results in the majority (56.8%) of all inmates. Drug offenses (counting both drug possession and drug trafficking) account for 17.4% of the prisoners, larger than any other single crime type. The offenses of burglary and larceny (including motor vehicle theft) contribute an additional 14.4%. The remaining 11.3% includes public-order and other (4.5%); forgery, fraud and embezzlement (2.3%); other property (2.1%); and DUI/DWI (2.4%).

Among the more prevalent offenses, the highest Xi values are for murder (88%), rape and sexual assault (81%), robbery (70%), and burglary (68%). Also high are a number of the less prevalent mixture of offenses, including other property offenses (90%), forgery, fraud, and embezzlement (86%), and DUI/DWI (79%). The lowest Xi values are for drug trafficking (52%), drug possession (52%), weapons (53%), aggravated assault (56%), and other violent offenses (62%).

We also observe in Table 6 the sum of the \({\text{X}}_{\text{i}} *{\text{F}}_{\text{i}}\) values, which is the 42-state aggregate value of X, or 69%. It represents the degree to which racial differences in arrests in 2010 account for the racial disproportionality of the 42 states’ prisons in 2011. This value is very close to the 70% estimate for 2011 in Table 3 and for the national estimates of 72% for 2004 and 71% for 2008, all based on the 80/10 national allocation of Hispanic prisoners. These values are not appreciably different from each other and all are somewhat lower than the 76% value in 1991 estimated by Blumstein (1993).

Estimating Xi at Two Stages: Arrest-to-Commitment and Time Served

To further understand the nature and extent of racial disproportionality in state prisons, we look to the different stages in the criminal justice process from arrest to incarceration. Ideally, we could separate the entire process from arrest to prison population into various intermediate stages and estimate the Xi values for each stage. It would provide an opportunity to identify which stage is contributing most to the lower Xi values. However, data on many of the intermediate processes from arrest to incarceration are not available. We are able to obtain data for new court commitments (NCC) by race and crime type. These data are collected annually in BJS’s NCRP program. The data allow us to examine the degree to which racial differences in arrest account for racial differences in new court commitments.

We can modify formula 1, and substitute NCC for the yearend prison populations. The result is designated as Xci, which represents the fraction of the disproportionality in new court commitments to prison that is accounted for by racial differences in arrests. Variations by state in Xci would reflect actions at the prosecution and judicial stages that account for the racial disproportionality in prison admissions. In addition, we can similarly analyze the transition Xti between NCC and the yearend prison population, reflecting primarily time served, including time served for parole violations. To do so, we can substitute NCC for the arrestee populations by crime type and state.

In 2011, NCRP data on new court commitments were available for 40 of the initial 42 states. (Connecticut and Alaska were not available.) In generating estimates of Xci and Xti, we also adjusted for states with anomalous Xi values (using procedures similar to those in Table 5).

In principle, these values serve as a two-step partition of the Xi values developed for Table 6. The results are presented in Table 7 for the arrest-to-prison commitment stage and Table 8 for the commitments-to-yearend prison stage (or time served).Footnote 9 The crime types in Tables 7 and 8 are ranked by values of Xci and Xti, respectively.

Not surprisingly, murder/manslaughter and rape/sexual assault show up with high Xi values in both stages, and the two drug offenses, drug trafficking and drug possession, show up with quite low values in both stages. Also high in both stages are forgery, fraud, and embezzlement and other property offenses. Aggravated assault, which can appear in many forms, is strikingly low in both stages.

Other crime types have high Xci values in the arrest-to-NCC stage but low Xti values in the time-served stage, and vice versa. These patterns may reflect factors related to race that contribute differently to commitment and to time served. Crime types that have high Xic values on commitment and low Xti values on time served are larceny and other property offenses. Weapons offenses and DUI/DWI, on the other hand, have low values of Xci on commitment but are among the highest on time served. The pattern for weapons and DUI/DWI offenses may reflect racial differences in sentencing due to mandatory sentences, which are often common for these crime types, but result in little variation in time served once imposed.

It should be noted that the values of Xci and Xti as well as their aggregates over the crime types, Xc and Xt, are significantly larger than the corresponding Xi values in Table 6. The component Xi values are expected to be larger since there is less variation to explain, but also the products of Xci and Xti should approximate the aggregate value Xi. Indeed, all of the products by crime type are within 8 percentage points of their corresponding Xi values in Table 6. While the aggregate product, \({\text{X}}_{\text{c}} *{\text{X}}_{\text{t}} = 0.77*0.92\) = 0.71, is slightly higher than the value of X (.69) in Table 6, perhaps because the partition by stage was based on 40 of the original 42 states.

Regional Variations in X by Crime Type

Our subsequent analyses will focus on regions rather than on individual states, although state-level values of Xij are available by region in Appendix Tables 11, 12, 13, and 14. To provide an analysis of regional differences, we have aggregated our 42 states into 4 regions:

  • Northeast (and border states): Connecticut, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, Delaware, Maryland.Footnote 10

  • South: Alabama, Arkansas, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, Oklahoma, South Carolina, Tennessee, Texas, Virginian, West Virginia.

  • Midwest: Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, Ohio, South Dakota, Wisconsin.

  • West: Alaska, Arizona, California, Colorado, Idaho, Nevada, New Mexico, Oregon, Utah, Washington.

Table 9 presents the values of Xi previously developed for the nation as a whole and for each of the four regions. Although we see some regional variation in the Xi values, they tend to be consistent with the national estimate. The standard deviations among the regions are generally quite small, with the largest variations being in “public-order and other offenses” and drug possession.

Figure 1 displays these values and shows by crime type how close each of the Xi values varies across the four regions. The horizontal axis is a listing of the 14 crime types in decreasing order of their national Xi value, and the vertical axis is the Xi scale. While there is some deviation from the national average, all four regions generally follow the national line. The South, where one might have expected deviation from the national pattern, perhaps most closely follows the national Xi values. The striking exception is its deviation in the rather heterogeneous group “public-order and other offenses.” There are a few other crime-type deviations, most notably the low Xi value in the West for “other property” offenses, and the low values for “public-order and other and drug possession offenses in the North.

Fig. 1
figure 1

National and regional values of Xi for 42 states in 2011

Full size image

The relatively high Xi values for murder and non-negligent manslaughter and for rape and other sexual assault, suggest somewhat of an underrepresentation of blacks for those offenses in prison relative to their representation among arrestees. This could be the consequence of excess arrests of blacks for these offenses and more careful investigation that results in declination by the prosecutors or dismissal/acquittal by the courts. It could also be associated with the phenomenon of “victim discounting” (Baldus et al. 1983), where even in the absence of a racial bias against the perpetrator, there could be a bias against those whose victims are white. Such offenses are predominantly intra-racial, and so that bias could serve to benefit black offenders, and thereby contribute to their underrepresentation in prison.

Connection Between Arrest and Criminal Involvement

The previous analyses used arrest data as a proxy for criminal involvement. This has been feasible because arrest data are available, link naturally with common definitions of crime types, and can be compared to measures of involvement at other stages in the criminal justice system. More direct measures of individual criminal involvement would be preferable; however, such measures are not widely available. One source for identifying the race of the perpetrator is BJS’s National Crime Victimization Survey (NCVS), a survey conducted in over 70,000 households in which all persons age 12 or older are interviewed twice a year. As a measure of criminal involvement, the NCVS data are limited to those incidents in which there was contact between the victim and the assailant. Consequently, the measure is limited to the violent crimes of rape and sexual assault, robbery, aggravated assault, and simple assault.

Analysis of the NCVS data has been carried out by Tonry (1995). He examined the victims’ identification of blacks as their assailant in robberies and aggravated assaults for the years 1980–1991. He found that victim data on perceived race of assailants closely paralleled the black arrests among arrestees. For example, the percentages among robbery assailants in 1980, 1985 and 1990 were 54.8% (nonwhite), 55.5% (black), and 51.5% (black). The percentages of blacks among persons arrested for robbery were 57.7, 61.7, and 61.2%. He did not include victim reports in offenses involving multiple offenders.

We examined the issue using NCVS data on the violent crimes more broadly, including rape/sexual assault, robbery, aggravated assault, and simple assault. We also included data on race of the assailants for incidents involving multiple offenders. Table 10 presents the percentage of black offenders as reported by victims in the 2010 NCVS and the percentage black among adult arrestees as reported in the UCR. For all four crime types, the ratio of the black percentage of arrestees and of reported offenders are very close to 1.0, ranging from .91 to 1.10. The aggregate ratio for violent crimes was 0.97. These results provide a strong indication of support for the use of arrest as a proxy indicator of criminal involvement in non-fatal violent crime.

Discussion and Implications

We have presented the fullest results yet available to reflect accountability, or the degree to which racial differences in arrest or criminal involvement account for racial disproportionality in state prison populations. The estimates vary from year-to-year, but all appear to range between 70 and 75%. The estimates account for all of the crime types for which persons are held in prison. Importantly, they take into account prisoners whose race in the administrative data is characterized as Hispanic. Furthermore, we have provided the first such estimates for individual states and regions.

Counting Hispanic prisoners as a separate racial group and the absence of such counts in arrest records poses an important problem in any analysis of racial differences within the criminal justice system. This problem would be eased if the FBI required law enforcement agencies to report arrests by race and Hispanic origin. Until such time, we must rely on state-level inmate survey data to determine the percentage white and black among Hispanic state prisoners and to include them into the estimates of prisoners’ race. Drawing on data from large-scale national inmate surveys of more than 80,000 state prisoners over 3 years provided the opportunity for more accurate estimates of accountability at the national and state-level.

Our analyses correct the results presented in the NAS report (2014) that raised the concern about the decline in accountability in 2004 and the further decline in 2008. This decline was an artifact of the failure to account appropriately for the race of Hispanic prisoners. Our results contradicting the argument of a worsening racial disparity in prison are further supported by the observed decline of the black-to-white incarceration-rate ratio from 6.8 in 1990 to 4.7 in 2011.

We find that accountability (i.e., the degree to which racial and ethnic differences in criminal involvement and arrest account for racial disproportionality in prison) varies by crime type. Accountability (represented by the Xi values) is highest for murder and non-negligent manslaughter; rape and other sexual assault; forgery, fraud and embezzlement; and other property crimes. These are the crimes for which investigation is most intense.

Accountability is the lowest for drug possession, drug trafficking, and weapons offenses. These offenses are more responsive to police presence and patrol patterns and are the most sensitive to implicit or explicit racial profiling. Drug markets, when operated as street markets, are especially vulnerable to police interventions leading to arrest. This has been the case with crack markets, which were typically operated by black offenders, in contrast to powder-cocaine markets, which were typically operated behind closed doors by white and Hispanic offenders. In addition, stop-and-frisk activities by law enforcement are often disproportionately conducted in minority neighborhoods. These police activities often lead to arrests for weapons offenses and drug possession when these objects are in the possession of the subject of a stop.

Accountability is somewhat higher for aggravated assault (56%) and other violent offenses (62%); however, incarceration for these offenses is often affected by a prior record of involvement in those offenses. Property offenses, including burglary and larceny/motor vehicle theft, and robbery are in the middle range of accountability, with Xi values ranging from 68 to 75%. The accountability for DUI/DWI is strikingly high (79%), perhaps a result of mandatory-minimum statutes for these offenses.

Our estimates of the residual 25–30% may be an indicator of racial discrimination in the criminal justice system. However, there may be other factors that account for the residual, including variations by race in the seriousness of the crimes committed, socioeconomic differences by race that impact the strength of the defense counsel, and variations by race in prior records that involve accumulated arrests for minor offenses among blacks due to differing police practices and patrol patterns.

We believe that our approach of first estimating accountability by aggregating all the post-arrest stages of the criminal justice system, including prosecution, sentencing, prison commitment, time served and post-custody recommitment, addresses the aggregate effects that contribute to racial differences in prison.

With the aggregate effects in hand, we are able to partition the arrest-to-incarceration process into two stages: (1) from arrest to prison commitment and (2) from commitment to time served. The partition enables us to determine where the disproportionality occurs and for which crime types. We find a lower value of accountability (X = 77%) at the early stage (arrest to prison commitment) than at the later stage (commitment to time served, X = 92%). This implies that the amount of racial disproportionality that remains unexplained is higher at the commitment stage (23%) than at the time-served stage (8%). In other words, there is greater racial disparity in determining the initial court decision to send an offender to prison (relative to arrest) than in the length of the prison sentence imposed upon conviction and time served.

Different crime types appear to reflect different biases at the different stages. Racial differences in arrests for weapons, aggravated assault, and DUI/DWI offenses provide a weak indication of the racial disproportionality for these offenses at the commitment stage. In contrast, arrests for other property offenses and rape/sexual assault provide a strong indication of accountability in commitment. Once in prison, racial differences in commitments account well for racial differences in time served, except for drug trafficking and drug possession.

Overall, there appears to be a strong racial bias in commitments and in time served for drug trafficking and drug possession. At the same time, there appears to be a strong racial bias in commitments for weapons offenses, but much less of an effect on time served. There is little evidence of bias in commitment or time served for murder/manslaughter and rape/sexual assault.

Our analysis reveals no clear pattern of regional differences in accountability. We find that the black-to-white incarceration-rate ratio in the South is well below that in other regions. While there are differences in accountability (the X values) across the regions, none of the regions are consistently above or below the others across the crime types.

We have used racial differences in arrest as a proxy for racial differences in offending; however, we recognize that this could be confounded with racial differences in the vulnerability to arrest given offending. Some support for using arrest as an indicator for criminal involvement is provided by the consistency between victim reports of the perpetrators’ race and police reports of arrestees’ race for violent crimes. There could be racial differences between criminal involvement and arrests for public-order crimes, DUI/DWI crimes, drug crimes, weapons crimes, or the other violent crimes. These differences could be a consequence of differences in patrol patterns, which are more intense in the low-SES neighborhoods, or the result of racial profiling in traffic stops or stop-and-frisk street stops. Any police bias would lower our estimates of accountability based on arrest. Certainly more careful research is needed to assess the relationship between involvement in crime and the likelihood of arrest.

Although these concerns should be pursued in further detail, our conclusion is that racial differences in prison are to a large degree reflective of the differences across the races in their involvement in crimes that lead to imprisonment. Factors contributing to that differential involvement include the “root causes” of crime associated with socioeconomic status, job opportunities, family structure and discipline, and local culture and peer influences.

Future research should be directed at identifying more fully the factors that account for the 25–30% unexplained difference between arrest and incarceration. More detailed examination of the various stages in the process between arrest and commitment to prison and time served is needed to understand the role that prosecutors, judges, and probation and parole officers contribute to this disproportionality. Importantly, factors contributing to racial differences in arrest that go beyond differences in offending need to be examined. After taking into account the race of Hispanic prisoners, the black incarceration rate remains high and the black-to-white incarceration-rate ratio, though dropping, was 4.7 in 2011. Clearly, a better understanding of racial disparities in the criminal justice system is important. To address these disparities, we must first understand them.