1 Introduction

By nearly all accounts, the US Congress is as polarized as it has ever been in our nation’s history (McCarty et al. 2016). Although several competing explanations for congressional polarization have been advanced, one of the most common narratives perpetuated by journalists, the media, and even some academics is that polarization is largely a function of our own electoral institutions. Indeed, two specific institutions—congressional primary elections and redistricting—are often singled out as the principal culprits for polarization in Congress.Footnote 1 In short, voters who participate in these primaries tend to be more extreme, which results in the selection of more ideologically extreme candidates. At the same time, state legislatures often gerrymander House districts to be electorally safe, which results in more ideologically extreme behavior on the part of legislators representing those districts. Nevertheless, others believe these institutions have little or no effect on the extent of polarization in Congress.Footnote 2

Although political scientists have attempted to discredit these explanations as factors driving polarization, the misleading narrative that reforming primaries and gerrymandering would serve as a panacea still persists among journalists and political pundits. In this article, we attempt to reconcile this disconnect by examining candidate ideology and its effect on election outcomes. Much of the existing research to date has considered the ideologies only of winning candidates without considering losing candidates as a counterfactual. By drawing upon evidence from Adam Bonica’s research on ideology and campaign contributions for all candidates seeking office, we effectively show that losers tend to be noticeably more ideologically extreme than do winners. As such, we demonstrate that candidate ideology is an important factor in affecting whether candidates can win—especially if they are not good ideological “fits” for their districts. Additionally, our results show that existing electoral institutions actually play an important role in mitigating already high levels of polarization in Congress.

The organization of this paper is as follows. In the next section, we briefly review some of the relevant literature on candidate success in congressional elections in order to motivate our research as well as offer some context for our broader theoretical framework. Next, we outline the hypotheses to be evaluated empirically and provide some descriptive analyses regarding the relationship between challenger ideology and success. We then estimate logistic regressions testing each hypothesis in the context of the US House of Representatives. Finally, we conclude with a discussion of the results as well as the larger implications of our findings for future research.

2 Electoral success in congressional races

It is well documented in modern congressional elections that the incumbent reelection rate regularly exceeds 90% (Jacobson and Carson 2016). Two explanations for this phenomenon have been proposed. First, incumbents typically are quite cognizant of their standing with their constituencies and are relatively good at sensing when they need to be more responsive to the voters who elected them in the first place. Furthermore, incumbents are adept enough to avoid losing by strategically retiring or seeking another office when national or local tides turn against them. Put simply, the most vulnerable incumbents are also the ones most likely not to seek reelection, which artificially inflates the already high incumbent reelection rate (Cox and Katz 2002).

The second reason is that a number of electoral advantages are available to sitting members of Congress. For example, Mayhew (1974) argues that members’ most proximate goal is winning reelection, and that they engage in advertising, credit claiming, and position taking in order to achieve this goal. Each of these activities is much more difficult, if not impossible, to engage in for non-legislators. This allows legislators to enhance their name recognition, which often leads to greater electoral success (Grimmer 2013; Jacobson and Carson 2016). Beyond higher levels of name recognition, incumbents also are advantaged by better fundraising efforts and larger campaign warchests (Abramowitz 1991; Box-Steffensmeier 1996; Fouirnaies and Hall 2014). Furthermore, the constitutionally mandated redrawing of congressional district boundaries every 10 years has been shown to disproportionately benefit incumbents (Abramowitz et al. 2006; Carson et al. 2014). These factors, along with favorable national and partisan tides (Jacobson and Carson 2016; Jacobson and Kernell 1983), have all been shown to contribute to high rates of incumbent reelection.

Although prior work typically focuses on the resources accruing to incumbent legislators when seeking to explain the incumbency advantage, it is important to recognize that incumbents are only part of the equation (Carson 2005). As Hinckley (1980) correctly asserts, it is not just a matter of what incumbents are doing well, but also what challengers are doing poorly or not at all. Over time, voters may grow dissatisfied with their representative, but they cannot be blamed for reelecting him or her when their only alternative is a relatively unknown candidate who is likely a poorly funded political novice. As Kirkland (2014, p. 543) argues, “[e]lections force legislative parties to respond to their constituents through either adaptation or replacement.” Therefore, if challengers are not presenting themselves as viable replacements ideologically, it becomes easier to understand how incumbents can maintain such high levels of success.Footnote 3 It is the relationship between ideological congruence and electoral success that is the focus of this article.

3 Theoretical evidence

As discussed above, a variety of factors assist incumbents in their bids for reelection—broader name recognition, fundraising, pork, redistricting, favorable national tides, and experience. In a similar vein, a variety of common characteristics of challengers reduce their chances of success—poor name recognition, insufficient resources, and lack of experience. However, one additional factor has received far less attention with respect to incumbents’ propensity to win—challenger ideology. To date, much of the literature examining the ideology of challengers has framed the discussion in terms of an important debate on representation—whether challengers are likely to position themselves as close to the median voter as possible as depicted by the spatial theory of elections (Downs 1957) versus a greater likelihood of candidate divergence (see, e.g., Bishin 2009; Calvert 1985; Groseclose 2001; Poole and Romer 1993). Based on analyses of candidate positioning, considerable evidence exists of candidate divergence at the local and national levels.Footnote 4

Ansolabehere et al. (2001) offer the most comprehensive account of candidate positioning to date. In their study of ideological positioning of House candidates running for office from 1874 to 1996, they find that candidates typically embrace the national party ideology as opposed to being responsive to local ideological constituencies. In order to compare the ideologies of challengers and incumbents during this period, they analyze the roll call voting records of incumbents who are paired with challengers “who either will have a roll-call record in the future or has one from the distant past” (Ansolabehere et al. 2001, p. 140). This allows for ideal point comparisons among sitting incumbents against candidates who have served previously or will serve at some future time in Congress. They then compare these results with those of the National Political Awareness Test (NPAT), surveying all congressional candidates in 1996, and find similar results across the two measures.Footnote 5

Ansolabehere et al. (2001) findings on candidate positioning in House elections represent an important contribution to the debate on representation in Congress. However, their work has one notable shortcoming. With the exception of the 1996 NPAT survey (which exists only for that year), their comparisons with incumbents across time include only candidates who have served previously, or eventually will serve, in the House. As such, we cannot say with certainty whether this represents an accurate sample of the behavior of all House candidates.

In an effort to build upon Ansolabehere et al. (2001) important work on candidate positioning, we turn to a dataset examining the ideological positions of all candidates running in a given congressional election. Adam Bonica has developed an empirical technique enabling researchers to place donors to local, state, and federal candidates in elections between 1979 and 2012 on a common liberal-conservative scale. Prior to the creation of Bonica’s measure, the ideological leanings could be measured consistently only for winning candidates who subsequently compiled analyzable voting records (see Poole and Rosenthal 2006). Bonica’s data are not limited to roll-call based measures and include the ideology of the losers as well as the winners. In light of this innovation, we can now pose and answer the following question—do challengers frequently lose because they are poor ideological fits for the constituencies they are seeking to represent?Footnote 6

It is generally accepted that ideological posturing can noticeably influence a candidate’s electoral fortunes. Challengers must differentiate themselves from the incumbents they are seeking to unseat. Candidates cannot afford to be seen as too ideologically incongruent with the constituency they hope to represent. However, they also cannot afford to be seen as a less experienced version of their opponent, lest they lose a substantial number of votes to that candidate. As such, many of the candidates that challenge incumbents tend to be more extreme than the sitting member of Congress. This is also consistent with the extant literature suggesting that primary voters often tend to be more extreme themselves (Boatright 2013; for an alternative perspective, see Norrander 1989).

Furthermore, past research has shown that representatives who are perceived as being “out of touch” with the electorate are more likely to face a strong challenger in an upcoming election (Carson 2005) and are less likely to get reelected (Canes-Wrone et al. 2002; Carson et al. 2010). Although much of the early elections literature suggested that voters largely were unaware of how legislators behave in Congress (see, e.g., Mann and Wolfinger 1980), more recent work has shown that voters are surprisingly in tune with how legislators vote and can often correctly predict how representatives vote on important roll call votes in Congress (Ansolabehere and Jones 2010). Based on this evidence, we believe that voters can indeed hold members of Congress accountable for their behavior.

One of the main reasons incumbents were successful in first getting elected is that they were able to appeal to a broad subset or, at the very least, a plurality of voters. Forcing challengers to occupy a more extreme space on an ideological spectrum prevents them from being able to appeal to a sufficient number of voters to win without “stealing” some votes away from the incumbent.

During general elections, it is comparatively easy for a challenger to differentiate him or herself from an incumbent since normally only one member from either party appears on the ballot. However, given how partisan many constituencies tend to be, it can be difficult to convince an experienced candidate of the out-party to wage what will very likely be a losing campaign. As a result, the only candidates willing to run are often issue-based candidates or “sacrificial lambs,” both of whom are not likely to occupy ideological positions that would make them electorally competitive in a majority of races. Again, ideological placement clearly is crucial to understanding which candidate ultimately will win an election. As such, we seek to test exactly what the relationship is between candidate ideology and the probability of winning. The hypothesis to be tested is as follows:

H 1 :

As ideological distance between a candidate and his or her potential constituency widens, his or her prospects of winning fall.

It is important to note that this relationship works in two ways: a candidate can be too extreme or not extreme enough. Irrespective of the direction, we expect the consequences to be the same. This is important to note because our theoretical expectations do not conform exactly to previous work. Canes-Wrone et al. (2002, p. 138) argue that “holding district ideology constant, in every election between 1956 and 1996 an incumbent’s vote share decreased the more he voted with the extreme of his party.” We differ slightly with this view and argue that no reason exists for a candidate to be punished for being “too extreme” by a congressional constituency that is similar ideologically. Therefore, we offer multiple means of testing our hypothesis. First, we rely on elections data from the US House of Representatives during all general elections between 1992 and 2012. The first is a logistic regression predicting whether or a not candidates won their respective bids for office. The second is a series of logistic regressions subset by the ideological composition of districts. Our analysis begins with some descriptive evidence, which is then followed by more systematic investigation.

4 Descriptive evidence

We begin by offering some preliminary assessments of incumbents and challengers across a range of different variables. First, we compare the ideologies of winning candidates to their unsuccessful counterparts in House races over time. Figure 1 shows that losing candidates consistently had significantly more extreme ideal point estimates than the individuals who ultimately won. A difference of means test shows that winners are less ideologically extreme than the candidates they defeat. Indeed, the average winning House candidate has an extremism score of 0.69 while losing candidates average 0.95 (p < 0.001).Footnote 7

Fig. 1
figure 1

Average absolute CF score by house candidate success. Note The solid line depicts the average ideological extremism of all candidates who won House seats. The dashed line depicts the average ideological extremism of all candidates who lost their bids for House seats. Between 1980 and 2012, losing candidates were significantly more ideologically extreme than their winning counterparts

Full size image

For context, this is the same distance between Adam Smith (Democrat representing Washington’s 9th district and a leader within the “New Democrats”—a self-described centrist group)—and Mike Quigley (Democrat representing Illinois’ 5th district who supports access to safe abortions without any restrictions and was inducted into the Chicago Gay and Lesbian Hall of Fame in 2007).

Also noteworthy is that this pattern of ideological extremity is increasing during the past three decades, which is consistent with the existing literature on polarization. Despite the fact that winners are indeed becoming more ideologically extreme, they are still significantly more moderate than their losing counterparts in each election cycle since 1980.

Figure 2 demonstrates that this relationship persists not only across time but differing levels of electoral experience as well. Though the results are not as striking as those in Fig. 1, a simple difference-of-means test reveals a statistically significant difference in the extremism of winners and losers across incumbents, quality challengers, and amateur House challengers.Footnote 8 The disparity is least pronounced within the incumbent category, which features a few election cycles without significant differences. This is unsurprising, though, as incumbent defeats are so rare. Nonetheless, incumbents who are voted out of office do tend to occupy more extreme positions on the ideological scale. This effect is even greater across quality challengers and amateur candidates.

Fig. 2
figure 2

Average absolute CF score by house candidate success and experience. Note In each of the three panels, the solid line depicts the average ideological extremism of all candidates who won House seats, and the dashed line depicts the average ideological extremism of all candidates who lost their bids for House seats. The top panel depicts this dynamic only for incumbents, the middle panel only includes challengers who previously held elective office, and the bottom panel only includes challengers without previous elective experience. Though there are a few exceptions, losing candidates are more ideologically extreme than winning candidates on average, regardless of experience

Full size image

This idea is supported further by the results depicted in Fig. 3. We plot the average ideal point estimate for each group of candidates and find a clear distinction in the extremism of each. Amateurs represent considerably more extreme ideologies than quality challengers, who in turn represent considerably more extreme ideologies than incumbents. Though it is important to differentiate one’s candidacy from the opposition, it may be the case that most challengers are overcorrecting in this regard, which further contributes to incumbents’ success rates. Furthermore, all incumbents and most quality challengers have spent time in a legislative body and (at least ostensibly) have learned the value of compromise, which could also lead to their more moderate ideologies.

Fig. 3
figure 3

Average absolute CF scores by house candidate experience. Note The solid line depicts the average ideological extremism for House incumbents. The dashed line depicts the average ideological extremism for challengers with had previously held elective office. The dotted and dashed line depicts the average ideological extremism for challengers without previous elective experience. Though extremism among all candidates is increasing between 1980 and 2012, incumbents are consistently the least extreme group, amateur candidates are consistently the most extreme group, and quality challengers exist in between the two. This demonstrates a negative relationship between extremism and experience

Full size image

Moreover, this dynamic is not limited to one party or the other, as depicted in Fig. 4. For Democrats and Republicans, winning candidates are significantly more moderate than losing candidates, all else equal. In short, irrespective of year, previous elective experience, or party affiliation, those candidates who successfully win a House seat are not as ideologically extreme as those who run losing campaigns.

Fig. 4
figure 4

Average absolute CF score by house candidate success and partisanship. Note In both panels, the solid line depicts the average ideological extremism of all candidates who won House seats, and the dashed line depicts the average ideological extremism of all candidates who lost their bids for House seats. The top panel is limited to all Democratic candidates, and the bottom panel is limited to all Republicans. Between 1980 and 2012, losing candidates were significantly more ideologically extreme than their winning counterparts for both parties. This demonstrates that this phenomenon is not limited to or driven by one party

Full size image

In sum, sufficient preliminary evidence presented here exists to warrant additional and more sophisticated testing of the relationship between challenger ideology and electoral success in House races over time.Footnote 9 In the next section, we outline the data employed in the remainder of our analysis and discuss model specification before turning to our main results.

5 Data and methods

As mentioned previously, prior studies of candidate ideology largely have relied on roll call votes. However, such analyses require a candidate to have a roll call record, which cannot be obtained until after winning an election. We overcome this limitation by using Bonica’s (2014) ideal point estimates derived from campaign contributions for both winning and losing House candidates as well as each congressional district to test the effect of a candidate’s ideological position on his or her probability of winning. This measure correlates highly with DW-NOMINATE scores (r = 0.92), and it allows us to put candidates and districts in the same ideological space.Footnote 10 We calculate the absolute value of this measure to construct our extremism scale—zero represents the most moderate a candidate can be, with larger values indicating greater extremism.Footnote 11 We use a similar transformation for our measure of district ideology to control for district extremism.Footnote 12

We also include several control variables in our analysis: candidates’ experience levels (amateurs, quality challengers, and incumbents), differences in spending, candidate and district partisanship, candidate gender, and year fixed effects.Footnote 13 Amateurs are individuals who lack electoral experience, whereas quality challengers are coded 1 if they have served previously in elective office. Candidate spending totals were collected from the Federal Election Commission (FEC) and we utilize the logged difference in spending between the two major party candidates.Footnote 14 Candidate partisanship is coded 1 for Democrats and 0 for Republicans. To measure district partisanship, we use Kernell’s (2009) measure in which larger values denote more support for Democrats.Footnote 15 Candidate gender is coded 1 for females, and we include year fixed effects to control for any unobserved heterogeneity across time, which is likely to be a function of national tides. We include all House general election candidates between 1992 and 2012. Table 1 reports summary statistics for each variable included in our analysis.

Table 1 Descriptive statistics of data employed
Full size table

6 Results

We use the Bonica (2014) data to evaluate the effect of candidate extremism on the likelihood of electoral success. We first test the effect of ideological congruity on outcomes by estimating a logistic regression predicting whether or a not a candidate won while controlling for their extremism.Footnote 16 The results are presented in Table 2. As expected, incumbents and quality challengers are more likely to run successful campaigns than candidates who have never held elective office. Additionally, greater ideological distance has a strong, negative effect on a candidate’s probability of winning. For example, among incumbents, candidates who are maximally congruent (0 on our scale) with their constituencies have a predicted probability of winning of 0.99. However, under the same conditions, an incumbent whose ideology is 1.5 units farther from that of the constituency of interest has a markedly smaller probability of success (0.56).Footnote 17 Finally, incumbents who are maximally incongruent (3 on our scale) with the district they seek to represent have a less than 2% probability of victory.Footnote 18

Table 2 Logistic regression estimates of candidate success, 1992–2012
Full size table

This relationship is further depicted in Fig. 5. Again, consistent with prior literature, we see that greater ideological distance reduces the probability of candidate success controlling for other factors known to influence congressional elections. We also see that although incumbents and quality challengers are indeed advantaged by their experience and political acumen, they are not immune to the perils of being perceived as a poor fit for the district and subsequently experiencing electoral defeat. Regardless of prior experience, extreme candidates are simply unlikely to win their bids for a House seat.

Fig. 5
figure 5

Predicted candidate success by ideological distance. Note The circles depict the predicted probability of winning a House seat for incumbents holding all else equal. The triangles depict the same outcome for challengers with previous elective experience, and the squares depict the same outcome for challengers without previous elective experience. The line through each of these shapes represents the 95% confidence interval. Increased experience increases the probability of winning, but within each group, the probability of winning decreases as the ideological distance between the candidate the constituency they seek to represent increases

Full size image

It is important to note that ideological distance not only hurts extreme candidates, but it could also hurt the chances of a moderate candidate running in an extreme district. Therefore, we subdivide our data by splitting the measure of district extremism at each quartile, resulting in four categories: most moderate, more moderate, more extreme, and most extreme. We then estimate the effect of candidate extremism on the probability of success within each category. The results of these estimations are presented in Table 3. Again, incumbents consistently outperform other candidates across all four types of districts. Moreover, quality challengers consistently outperform amateurs as well. These models also demonstrate that Democratic candidates are more likely to prevail in more moderate districts, while Republicans win more often in the more extreme districts. This result provides additional evidence that Republicans are asymmetrically contributing to polarization in Congress.

Table 3 Logistic regression estimates of candidate success by district type, 1992–2012
Full size table

Most relevant for the hypothesis being tested here, though, is the effect of ideological distance. The magnitude of the coefficient of that variable declines as the extremism of the constituencies increases. This makes intuitive sense because we would expect the most moderate districts to punish extreme candidates electorally the most. In the fourth model specification, we see that this variable fails to achieve statistical significance. That result also conforms to our expectations as extreme districts are more likely to focus on partisanship and experience since voters would not need to not support an extreme candidate. This finding is detailed further in Fig. 6.Footnote 19

Fig. 6
figure 6

Predicted candidate success by experience and extremism. Note The circles depict the predicted probability of winning a House seat for incumbents holding all else equal. The triangles depict the same outcome for challengers with previous elective experience, and the squares depict the same outcome for challengers without previous elective experience. The line through each of these shapes represents the 95% confidence interval. Increased experience increases the probability of winning, but within each group, the probability of winning decreases as the ideological distance between the candidate the constituency they seek to represent increases, except for in the most ideologically extreme House districts. Furthermore, the effect of increased candidate extremism diminishes in magnitude as the ideological extremism of the district increases

Full size image

7 Conclusion

In summary, our research on candidate ideology reveals three distinctive features of congressional elections. First, candidate ideology appears to be an important determinant of election outcomes. Just as is the case with experience, spending, and partisanship, we have provided evidence demonstrating that where a candidate lies on an ideological continuum is a strong and significant predictor of how that candidate will fare in a given election. The second major finding is that the ideology of the constituency matters as well. In general, and consistent with much of the previous research, moving toward the extreme flanks of one’s party rarely will prove to be electorally advantageous, at least in the general election. Nevertheless, that conclusion is attributable not to a candidate’s posturing, but to his or her posturing relative to the constituency he or she seeks to represent. Finally, we see that although polarization in Congress is increasing over time and the ideological gap between the two parties is at an all-time high, the reality of the situation is that things actually could be much worse. If the average challenger was more skilled at winning his or her bid for office, we might actually witness an even more polarized political environment than what we are currently seeing.

Based on the findings reported above, this research has notable implications for the study of congressional elections. When evaluating House elections, it is important that scholars consider the ideological placement of the candidates and the relative extremity of the underlying constituency. However, it is also useful to keep in mind that not all “moderate” or “extreme” candidates are equal when it comes to congressional elections. Candidates that have served previously in state legislatures, for instance, tend to be more moderate than political amateurs. The value of that experience is most likely a function of having to work with others, resulting in a series of compromises on the diverse array of legislative issues they have dealt with at the state level. Though they can easily be lumped together, meaningful variation exists across each group. Moreover, that variation can and should be carefully considered in order to increase our understanding of who gets elected to Congress as well as what policies are pursued once there.

Our study also generates a variety of normative implications. Namely, what do our findings mean for representative democracy? A truly democratic institution calls for both responsiveness and representativeness. If incumbents routinely are effective at winning reelection—thanks in no small part to their opponents’ shortcomings—are voters simply to assume that incumbents are truly acting in their best interests? If no real threat to a member’s electoral security exists, what incentive is there for the incumbent to constrain his or her behavior to meet the underlying interests of the constituency? At the same time, if a significant proportion of challengers truly are more extreme than the incumbents who currently hold office, what are the broader implications for trends such as partisan polarization? These and other important questions need to be carefully considered in future research.

We believe that our findings only begin to scratch the surface of the relationship between candidate ideology and electoral success in congressional races. Would the same patterns we found also hold in the context of congressional primary races? Likewise, is this relationship the same in Senate elections where fewer amateurs often seek office? Addressing these and related questions will help us better understand the various factors contributing to electoral competitiveness in mainstream congressional races.