Abstract
Large partisan gaps in reports of factual beliefs have fueled concerns about citizens’ competence and ability to hold representatives accountable. In three separate studies, we reconsider the evidence for one prominent explanation of these gaps—motivated learning. We extend a recent study on motivated learning that asks respondents to deduce the conclusion supported by numerical data. We offer a random set of respondents a small financial incentive to accurately report what they have learned. We find that a portion of what is taken as motivated learning is instead motivated responding. That is, without incentives, some respondents give incorrect but congenial answers when they have correct but uncongenial information. Relatedly, respondents exhibit little bias in recalling the data. However, incentivizing people to faithfully report uncongenial facts increases bias in their judgments of the credibility of what they have learned. In all, our findings suggest that motivated learning is less common than what the literature suggests, but also that there is a whack-a-mole nature to bias, with reduction in bias in one place being offset by increase in another place.
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Surveys of political knowledge frequently find large partisan differences in reported beliefs about policy-relevant facts. When Democrats control the White House, Democrats are likelier than Republicans to report that economic conditions are improving; the reverse holds true under a Republican administration (Bartels 2002). Similar patterns exist on answers to factual questions about healthcare (e.g., Nyhan 2010), foreign policy (e.g., Jacobson 2010), and social services (e.g., Jerit and Barabas 2012), among other issues. In general, when people are surveyed about facts, they are more likely to report having beliefs that are congenial to their existing beliefs and attachments than beliefs that are uncongenial. While we expect values and preferences to differ in a pluralistic society, we do not expect disagreement over established facts. Factual disagreement fuels concerns about the public’s ability to form preferences that are in line with their values or interests (e.g., Hochschild 2001), hold representatives accountable (e.g., Bartels 2002), and productively deliberate with each other (Shapiro and Bloch-Elkon 2008; Muirhead 2013).
We consider two distinct explanations for differences in survey reports of factual beliefs: motivated learning and motivated responding.Footnote 1 According to the former, when people are presented with information, they are more likely to deduce that the information supports a congenial conclusion than an uncongenial conclusion (Jerit and Barabas 2012; Kahan 2013; Kahan et al. 2017). This bias in learning is thought to stem from the psychological drive to hold internally consistent beliefs that reflect positively on one’s core attachments, such as political party, ideology, or cultural identity (Kunda 1990). Motivated learning is especially concerning from a normative standpoint. It suggests that even when people come across the same information, they can form very different beliefs about the conclusions supported by the information. (Note that learning about conclusions supported by information is different from believing that the conclusions are correct.) Such bias makes reducing gaps in factual beliefs yet harder—mere provision of accurate information is even less likely to work, and may even backfire (see, e.g., Nyhan and Reifler 2010).
However, differences in survey reports of factual beliefs do not always reflect differences in what people believe. Instead, they may be artifacts of the survey response process. Respondents sometimes give congenial but inaccurate answers in response to factual questions even when they have accurate but uncongenial facts at hand (Bullock et al. 2015; Prior et al. 2015). Other times, respondents are ignorant, having no relevant cognitions, and they offer a congenial answer as their best guess (Luskin et al. 2013). In both cases, the survey response process inflates estimates of bias in factual beliefs. We extend this line of thinking to estimates of learning, the process that produces such beliefs. Like studies of stored cognitions, studies of learning may overstate bias if they do not account for motivated responding.
We reassess the extent to which people learn in a motivated manner by simultaneously measuring motivated learning and motivated responding in three separate studies. Building on the design in Kahan et al. (2017)—people are presented data from a social scientific study and asked to report the conclusion supported by the data—we incentivize a random set of respondents to honestly report the conclusion they actually think the data supports. Incentives reduce congeniality bias in reporting the study’s result, suggesting that a portion of what appears to be motivated learning is in fact motivated responding. Correspondingly, we find no evidence that respondents selectively recall the data they were presented. However, there is considerable heterogeneity in responsiveness to the incentive treatment, depending on respondents’ initial position on the issue under study. Moreover, incentivizing respondents to report uncomfortable conclusions supported by a study comes at a price: respondents become more skeptical about the study’s credibility. In all, our findings suggest that motivated learning is less common than what previous work suggests, but that motivated evaluation of information remains common.
Motivated Learning, Reporting, and Interpretation
Psychological motivations are powerful in shaping how people process information. Broadly, people are motivated to both reach conclusions that are accurate and that are congenial to their beliefs (see Kunda 1990). Most research in political psychology focuses on the latter, which Kunda calls “directional” motivation. A great deal of research shows that directional motivations, like partisanship and ideology, bias evaluations of policy arguments (e.g., Lord et al. 1979; Taber and Lodge 2006; Bolsen et al. 2014) and of leaders (e.g., Bartels 2002; Lebo and Cassino 2007; Kim et al. 2010).
Motivated Learning
While most research on motivated reasoning focuses on subjective attitudes, motivated reasoning may also influence factual learning. Suppose, for instance, that there exist some data that support an unequivocal conclusion relevant to public policy. And we ask an individual with an existing opinion on said policy to learn the conclusion supported by the data. The extent to which directional goals outweigh accuracy goals will be correlated with the probability that the individual learns that the data support a congenial conclusion, even if this conclusion is incorrect. We refer to this phenomenon as motivated learning.
Motivated learning has received recent attention. For instance, Nyhan and Reifler (2010) find that partisans ignore corrective information that contradicts their ideological worldview.Footnote 2 Similarly, Jerit and Barabas (2012) find that partisans selectively learn party-relevant factual information. The authors present partisans with facts that reflect either positively or negatively on Democrats or Republicans. They find that partisans are more likely to learn congenial than uncongenial facts, e.g., Democrats are more likely to learn about the success of the Troubled Asset Relief Program than the size of the trade deficit. A major challenge in such experiments is cleanly manipulating information congeniality, while holding constant other attributes, such as topic and difficulty. We now consider a recent study that does just that.Footnote 3
Kahan et al. (2017) cleverly repurpose a “covariance detection task” (see Gilovich 1991) to test whether people engage in motivated reasoning when processing policy-relevant data. Respondents see a \(2 \times 2\) table with data on the relationship between banning concealed carry and rates of crime. The table’s column headings are manipulated so that the data either support the conclusion that banning concealed reduces crime or the conclusion that banning concealed carry increases crime. When asked what result the data support, respondents are more likely to answer correctly when the data support a congenial claim than when the data support an uncongenial claim. That is, liberal Democrats are more likely to report the correct result when banning concealed carry reduces crime than when banning concealed carry increases crime, while the reverse is true among conservative Republicans. Kahan et al. use the terms motivated numeracy and motivated cognition to describe the phenomenon.
The finding is consistent with several other studies that show that people evaluate congenial and uncongenial claims differently, using different evidentiary standards and investing different amounts of effort in processing the information. When evaluating congenial claims, people tend not to be as thorough in searching for evidence (e.g., Kruglanski and Webster 1996; Nickerson 1998), and evaluate available evidence more superficially and less skeptically (e.g., Chaiken and Maheswaran 1994). On the other hand, when evaluating uncongenial claims, people are more skeptical and invest greater processing effort (e.g., Ditto and Lopez 1992; Ditto et al. 1998; Dawson et al. 2002a). For instance, in the covariance detection task, respondents are more likely to only partially consider the data if doing so leads them to think that the data support a congenial conclusion (Dawson et al. 2002b).
Building on this psychological literature, Kahan et al. (2017) argue that our natural tendency to learn from data heuristically (i.e., using mental shortcuts) results in bias. When presented with tabular data, people tend to only consider the most salient datum in the table, for instance, the largest number, to deduce the conclusion supported by the data (Gilovich 1991, p. 31). If heuristic processing yields a congenial answer, people tend to stop processing, concluding that the data supports their beliefs. If heuristic processing instead yields an uncongenial result, people tend to look at the data more carefully to make sure they are correct. For instance, activating directional goals can motivate people to overcome the common error of neglecting ‘cell D’ in a \(2 \times 2\) table (Mata et al. 2015a, b). This imbalance in scrutiny produces a congeniality effect: people learn congenial facts more readily than uncongenial facts. Kahan et al. argue that this effect increases with numeracy, because only respondents with sufficient numerical ability are capable of learning the correct result by considering all four cells.
Aside from asymmetric scrutiny, selective perception may also contribute to motivated learning. A long line of research suggests that expectation structures cognition (Hastorf and Cantril 1954; Bechlivanidis and Lagnado 2013; Kahneman 2013). In our hurry to learn from data, for example, we misperceive data in ways that are consistent with our prior beliefs. When trying to distill information from a contingency table, people may misread the column or row labels in a way that suggests a congenial result. Alternatively, one might misread the numbers in the table. In all, either due to selective perception or an imbalance in scrutiny, people are thought to be more likely to learn correctly when the data are congenial than when they are uncongenial.
Motivated Responding
Motivated responding concerns what survey respondents report on surveys, rather than what they have learned or know.Footnote 4 In particular, it occurs when people with the same underlying beliefs give congenial answers more often than uncongenial answers when asked about their beliefs. In all, it contends that survey responses to factual questions reflect a mix of what people believe and what they wish to be true (Luskin et al. 2013; Prior et al. 2015).
People engage in motivated responding for a variety of reasons. Some deliberately misreport as a way to express their attitudes. For example, a survey respondent who vehemently opposes President Barack Obama may not admit knowing that the unemployment rate declined between 2008 and 2016. The respondent may instead report a rise in unemployment to express their opposition. This kind of expressive self-presentation has been described as “cheerleading” (e.g., Gerber and Huber 2009). Others may misreport just to be consistent within a survey, ensuring that later answers do not contradict their earlier ones (e.g., Sears and Lau 1983; Lau et al. 1990; Wilcox and Wlezien 1993; Palmer and Duch 2001). Yet others may engage in motivated responding to indicate their disbelief in information. For instance, in Kahan et al. (2017), a respondent may pick the congenial answer even after figuring out that the data support an uncongenial conclusion as a way to express their disbelief in the putative data.
In addition to actively misreporting what they believe, motivated responding may take more passive forms. For example, a respondent may withhold their beliefs by selecting “Don’t Know” or skipping a question. Alternately, respondents who don’t know the correct answer may report a congenial answer as their best guess. People may also engage in motivated responding without being consciously aware of it. For example, when asked a factual question in a survey, respondents may scan their memory for a longer time to come up with instances of congenial beliefs than uncongenial beliefs. In all, a variety of reasons explain why people may engage in motivated responding. Our study does not attempt to disentangle these reasons. Instead, our aim is to measure the bias due to motivated responding in estimates of motivated learning derived from ordinary survey instruments.
While both motivated learning and responding fall under motivated reasoning, there are two major differences. First, motivating responding is about the survey response process, as opposed to learning. Motivated responding affects what people say they have learned, not what they actually learn. Second, and relatedly, we suspect that part of motivated responding is just cheap talk that people engage in to publicly protect their core attachments and beliefs. And to the extent that these public pronouncements are shallow, based not in what people deeply believe but what people are prepared to say publicly, these reported “beliefs” are unlikely to shape respondents’ attitudes and behavior. On the other hand, uncongenial beliefs, which respondents are reluctant to express, may nonetheless influence attitudes and behavior. It is therefore important to distinguish between beliefs and instrumental or shallow responses.
A pair of recent studies find evidence that partisans engage in motivated responding on factual questions with political implications. Bullock et al. (2015) and Prior et al. (2015) find that partisans incorrectly report congenial beliefs, even when they know or could have inferred a more accurate answer. Both studies uncover motivated responding by boosting respondents’ accuracy motivation. Bullock et al. (2015) do so by offering respondents bonus payments for correct answers and for admitting their ignorance (by marking “don’t know”), while Prior et al. (2015) use a combination of bonus payments for correct answers, and textual appeals for responding accurately. Importantly, neither of the treatments provide any additional information. Therefore, any change in responses can be attributed to a change in respondents’ motivation, rather than a change in their knowledge. Both studies find that, consistent with motivated responding, accuracy incentives substantially reduce partisan bias—by as much as half.
Because studies of knowledge and learning rely on similar survey instruments, discovery of motivated responding on questions about stored knowledge suggests that estimates of motivated learning may also be inflated. To uncover motivated responding, we borrow a design feature of Prior and Lupia (2008), Bullock et al. (2015), and Prior et al. (2015), offering a random set of respondents small monetary incentives to accurately report their beliefs. To minimize the possibility that incentives affect processing of information, we present information about incentives after respondents have seen the information and can no longer revisit it. These incentives increase respondents’ motivation to honestly report which conclusion they think the data support. Respondents who would ordinarily offer a congenial answer as their best guess or as an act of political expression may be nudged to think more carefully or be more truthful. We therefore hypothesize that this treatment will attenuate the congeniality effect that Kahan et al. (2017) and others observe.
More specifically, we hypothesize that incentives will increase the probability of answering correctly when the correct answer is uncongenial. If respondents learn the uncongenial result correctly, and then knowingly report an incorrect answer, incentives should encourage them to reveal their true belief. If respondents offer an incorrect, congenial response as their best guess, incentives should increase the probability that they will guess more evenhandedly. In both scenarios, incentives should increase correctness. However, we do not expect incentives to significantly increase correctness in the congenial condition, because here the congenial answer is the correct answer. We present our hypothesized pattern of results in a graphical format in Fig. 1. The vertical axis represents the probability of answering correctly, and the dashed lines indicate the size of the congeniality effect. Incentives reduce the congeniality effect by increasing the probability of answering correctly in the uncongenial condition.
Motivated Interpretation
Directional motivations affect not just what people learn and report but also how credible people think a study is. Previous work suggests that people are more likely to question a study’s credibility when its results are uncongenial than when they are congenial (e.g., Lord et al. 1979; Kunda 1990; Ditto and Lopez 1992). This phenomenon likely stems from the more general tendency to spend greater time and effort scrutinizing and refuting uncongenial claims than congenial claims, known as disconfirmation bias (e.g., Edwards and Smith 1996; Taber and Lodge 2006). We therefore expect a congeniality effect on subjective study ratings: respondents will rate the same study as more convincing and well executed when its result is congenial than when it is not. And we expect this congeniality effect to be more pronounced when we incentivize respondents to report the correct study result, because respondents who admit an uncongenial result should be even more likely to report that the study behind the result is unconvincing.Footnote 5
Research Design
To distinguish between motivated learning and motivated responding, we conduct three experiments that build upon the original design of Kahan et al. (2017). In particular, we add an orthogonal manipulation, offering participants a small monetary incentive to accurately report what they have learned. The \(2 \times 2\) design enables us to test whether incentives reduce the congeniality effect, which has been interpreted as evidence for motivated learning. In addition to measuring the outcome used by Kahan et al., we also measure subjective ratings of the study to examine whether the study’s congeniality influences its perceived credibility. Below we elaborate on each of these extensions.
In Study 1, we asked respondents to read a summary of a hypothetical study on gun control. A preface to the study described its purpose: a city government is trying to decide whether or not to ban private citizens from carrying concealed weapons and wants to know if doing so would increase or decrease crime. After the preface, the study was briefly summarized: researchers compared changes in annual crime rates in cities that had banned concealed carry with changes in annual crime rates in cities that had not banned concealed carry. A \(2 \times 2\) contingency table with the study’s putative results came next.
Following Kahan et al. (2017), we manipulated the conclusion supported by the study by switching the column labels. In Table 1, cities that banned concealed carry were more likely to experience a crime decrease relative to cities that did not ban concealed carry. This result can be deduced by comparing the ratios of cities in the first row (75:223 or about 1:3) and the second row (21:107 or about 1:5). Flipping the column labels produces the opposite result—the data now indicate that cities that banned concealed carry saw increases in crime (see Table 2 below).
After presenting the summary of the study, we asked respondents whether cities with a ban were more likely to experience an increase or decrease in crime than cities without a ban. This question serves as our primary dependent variable. Note that it is strictly factual in nature. It simply asks which of two descriptions is consistent with the data. The question does not ask the respondents to assess a causal claim, evaluate gun control, or indicate their faith in the study. Respondents could not access the study description and table when picking which of the conclusions were supported by the data. At the end of the survey, respondents were debriefed and informed that the data were not real.
To measure motivated responding, we independently manipulated respondents’ motivation to give the answer they thought was correct. We offered a random set of respondents a small nudge, an additional $0.10 for the correct answer. Keeping the amount small has the virtue of not raising respondents’ suspicions—some respondents may take a larger amount as a cue that the uncongenial answer is the right one. As to whether the small nudge is sufficient, we cannot strictly say, but note that Prior et al. (2015) uncover about the same amount of motivated responding by emphasizing the importance of accuracy without any extra money as they do by offering another $1 for the correct answer. To ensure that incentives did not affect how respondents processed the contingency table in the treatment condition, we withheld any information about the incentive until after they had seen the table, and could no longer return to it. The control group was not offered incentives.
After measuring the primary dependent variable, we asked respondents to rate how convincing and how well done they found the concealed carry study. Each rating was measured on a 0–10 scale. We also asked respondents to recall the numbers in the contingency table at the end of the survey, in order to test whether respondents are more likely to remember congenial data than uncongenial data. To minimize respondent disengagement, we offered an additional $0.05 for each number recalled correctly.
In Study 2, we readministered the concealed carry task and added another task following it. In the second task, respondents were presented with a study on the impact of raising the minimum wage. Again, respondents were asked to indicate its result based on tabular data. The minimum wage task was very similar to the concealed carry task in design, with two important differences (aside from the change in topic). First, with the intention of making it easier to learn the correct result, we replaced cell frequencies with percentages in the table. The data suggest that the change had the intended effect, as there was a large increase in the percentage of correct responses. Second, we manipulated the study’s congeniality by switching the row labels instead of the column labels in the table. We believe this is a cleaner manipulation as it holds constant the increase-to-decrease ratio in each row, and simply changes the policy associated with each ratio. While lowering the task’s difficulty might change the congeniality effect observed, we do not expect our changes to affect the degree of motivated responding. Lastly, randomization in the second task was conducted independently of the first, but the sequence of the two tasks was fixed.
In Study 3, we readministered both the concealed carry and minimum wage tasks on a more representative sample. Following our hypothesis that incentives influence responses in the uncongenial condition, we presented all respondents with an uncongenial version of the concealed carry task (based on their pre-treatment attitudes) and randomized incentives as in Studies 1 and 2. This simpler design allows us to conserve resources while testing our central theoretical claim that incentives increase correctness in the uncongenial condition. Additionally, we replicated the full \(2 \times 2\) minimum wage task. The purpose of Study 3 was to gather confirmatory evidence and probe the generalizability of the estimates in Studies 1 and 2.
In each study, in order to identify respondents that would find each study’s result congenial or uncongenial, we measured attitudes toward banning concealed carry and raising the federal minimum wage before the tasks (experiments). We expect respondents who oppose concealed carry to find a decrease in crime associated with a concealed carry ban to be congenial, and an increase in crime uncongenial; we expect the opposite among respondents who support support concealed carry. The same logic applies to the minimum wage task. We measured party identification, political ideology, and demographics prior to the experiments in each study. “Question Wording” in the Supplementary Information (SI) contains a complete description of the three studies and the complete wording of each task and question. In Studies 2 and 3, we omitted recall questions and ratings of the minimum wage study due to concerns about the length of the survey.
Data
We recruited respondents from Amazon’s Mechanical Turk (MTurk) in Studies 1 and 2.Footnote 6 We recruited workers for both studies by advertising a task of completing a short survey on “how people learn.” To assess whether our findings generalize beyond samples recruited on MTurk, we recruited respondents via Qualtrics in Study 3. While it is not a “gold standard,” the Qualtrics sample is more representative of the general population, and respondents appear to be less attentive and detail-oriented than MTurk workers. Study 1 was fielded in December 2013–January 2014, Study 2 in March 2015–April 2015, and Study 3 in August 2016. For details of the recruited samples and how the samples compare to established benchmarks, see SI Table 1 in “Sample Characteristics”.
While none of the samples are nationally representative, we can still learn a great deal from them. Multiple studies find that MTurk samples yield high-quality data and are more heterogeneous and representative than other common convenience samples, such as student samples (Buhrmester et al. 2011; Berinsky et al. 2012; Paolacci and Chandler 2014). And recently, Mullinix et al. (2015) replicate a broad array of experiments across different samples and find that treatment effects are broadly similar across samples. More generally, treatment effects vary across samples when they are strongly conditioned by covariates that vary heavily across samples. We do not expect partisans on MTurk to differ enormously from partisans more generally with respect to motivated reasoning. Multiple studies using MTurk samples find partisan bias in both stored knowledge (e.g., Chambers et al. 2014; Ahler and Sood 2016; Chambers et al. 2015; Bullock et al. 2015) and political judgments (e.g., Arceneaux and Vander Wielen 2013; Lyons and Jaeger 2014; Crawford and Xhambazi 2015; Crawford et al. 2015; Thibodeau et al. 2015). In all, we think it likely that our treatment effects would be similar among a nationally representative sample.
Given the theoretical expectation that we should only observe motivated learning among respondents with sufficient numerical ability to complete the covariance detection task, we screened for high-numeracy respondents using a numeracy quiz in Study 1. The numeracy quiz was composed of the five easiest questions in Weller et al. (2012) (see “Numeracy Screener” in SI for exact items). We invited respondents answering four or more items correctly to participate in the full study. We use a threshold of four because Kahan et al. (2017) find that the median respondent answers four items correctly on the full nine-item scale. In Studies 2 and 3, we invited all respondents to complete the main task, irrespective of numeracy, to ensure that our findings in Study 1 were not contingent on the relatively numerate MTurk sample. In “Respondents Enrolled, Screened, Allocated, and Analyzed” in the SI, we track the number of respondents in each phase of the concealed carry experiment. We find that low- and high-numeracy respondents are similar in terms of party identification, ideology, and demographics (see SI Table 1 in “Sample Characteristics”), but we also present their results separately in “Does Numeracy Condition Treatment Effects?” in the SI.
In Study 1, we recruited 1207 respondents and invited 785 (65% of sample) who passed the numeracy quiz to participate in the full survey. Our main analyses include 686 respondents (87% of screened sample) reporting a position on a concealed carry ban, which is necessary to code congeniality. Of them, 34% opposed concealed carry (i.e., favored ban) and 66% supported concealed carry (i.e., opposed ban). In Studies 2 and 3, we recruited another 947 and 1062 respondents, respectively. Of those indicating a position, similar percentages to those in Study 1 opposed concealed carry: 36% in Study 2 and 40% in Study 3. The vast majority of respondents were in favor of raising the federal minimum wage: 85% in Study 2 and 65% in Study 3.Footnote 7
Results
We begin by presenting results from the concealed carry task, first pooling data from Studies 1 and 2, followed by results from Study 3. We separate out Study 3 because the concealed carry task only included the uncongenial condition. We follow it with results from the minimum wage task, and end with describing the impact of the treatments on respondents’ subjective study ratings.
If people learn in a motivated manner, the percentage of respondents answering correctly when the study’s result is congenial should be greater than when the study’s result is uncongenial. Respondents who oppose concealed carry should be more likely to answer correctly if the data support the conclusion that crime is more likely to decrease in cities with concealed carry bans than in cities without such bans. Among respondents who support concealed carry, the reverse should be true. We thus examine whether the congeniality manipulation increases the probability of answering correctly.Footnote 8
Before analyzing data from the covariance detection tasks, we check to see if partisanship, ideology, and demographics are balanced across the experimental conditions. The average p-value of cross-condition comparisons is .42 in Study 1, .47 in Study 2, .56 in Study 3, and .48 overall (see SI Table 2 in “Covariate Balance”). We also confirm that the first experimental task did not affect behavior in the second (see SI Table 3 in “Assessing Spillover”). In all, the data suggest that randomization was successful, so we move on to analyzing data from the concealed carry experiment.
Figure 2 plots the percentage of respondents answering correctly in the concealed carry task by experimental condition across Studies 1 and 2.Footnote 9 We first consider the percentage correct among concealed carry supporters in the absence of incentives (plotted in a, left). When the result is uncongenial (i.e., pro-ban), only 42.6% of respondents mark the right answer. When the result is congenial (i.e., anti-ban), the percentage increases to 54.6%. Thus, simply changing the study’s result from uncongenial to congenial (by swapping column headers) increases the probability of answering correctly by 12.0 percentage points (b). The pattern is similar among concealed carry opponents in the No Incentives condition (c, left). When the result is uncongenial (i.e., anti-ban), 41.0% answer correctly. When the result is congenial (i.e., pro-ban), correctness increases to 55.5%. The congeniality effect is 14.4 points (d). Thus, in the absence of incentives, the congeniality effects among both concealed carry supporters and opponents is statistically significant.
We next examine the extent to which incentives reduce these congeniality effects, which at first blush appear to be evidence of motivated learning. As we note earlier, the incentive treatment was administered in such a way that it did not affect how respondents initially processed the data–incentives were revealed after the respondents had seen the data and could not go back to it. If we were successful in administering the incentive treatment in the way we intended to, respondents should be as good at recalling the data in the No Incentives condition as in the Incentives condition. We tested this hypothesis with the recall questions at the end of the survey. Data suggest that incentives had no impact on the accuracy of recall (see SI Tables 5 and 6 in “No Evidence of Selective Recall”). Thus, it is unlikely that any treatment effects we see are explained by greater attention to the data when incentives are offered.
Examining Fig. 2b, we see that offering incentives to concealed carry supporters does not reduce bias. The congeniality effect is 13.9 points with incentives, which is almost indistinguishable from the congeniality effect in the absence of incentives (difference-in-differences is an insignificant 1.9 points). Since the congeniality effect remains substantial regardless of incentive condition, it appears that concealed carry supporters indeed learn in a motivated manner.
Data from opponents of concealed carry, however, tell quite a different story (Fig. 2d). Here, it appears that motivated responding masquerades as motivated learning. Incentives lower the congeniality effect from 14.4 points to an insignificant −3.9 points. The difference-in-differences is −18.3 points and statistically significant (s.e. = 9.3, \(p = .05\)). Incentives completely wipe out the bias in answering the question about the study’s result. Moreover, consistent with our hypothesis, the reduction in bias is entirely due to an increase in correctness in the uncongenial condition (16.0 points, s.e. = 6.4, \(p < .05\)), rather than any change in the congenial condition.Footnote 10
Results from Study 3 are similar to results from Studies 1 and 2 for both concealed carry supporters and opponents. Recall that Study 3 only included the uncongenial version of the concealed carry task. Fig. 3 displays the percent correct among concealed carry supporters (a) and opponents (b) by incentive condition. Overall, Study 3 respondents had more trouble correctly identifying the study’s result than respondents in Studies 1 and 2. For instance, only 33.6% of concealed carry supporters correctly identified the uncongenial study’s results without incentives, while 42.6% did so in Studies 1 and 2. More relevant for our purposes, we see that concealed carry supporters are again essentially immune to the incentive treatment. The percentage correct among this group is almost identical when we offer accuracy incentives (33.3%). There is no evidence of motivated responding here.
Concealed carry opponents, on the other hand, once again exhibit a pattern of motivated responding. While only 25.6% answer correctly without incentives, 32.5% answer correctly when offered incentives, resulting in a treatment effect of 6.9 points (s.e. = 4.4, \(p < .06\)). While the magnitude of the effect is smaller than in Studies 1 and 2, it is still non-trivial. In all, the evidence from the concealed carry experiment suggests that motivated responding introduces substantial amounts of bias in estimates of motivated learning.
We now turn our attention to the minimum wage task, which we included in Studies 2 and 3, to probe the degree of motivated learning and responding on a different issue, using a slightly different experimental design. The overall percent correct was high in this task (87% in Study 2 and 77% in Study 3), which is unsurprising given our decision to make this task easier on respondents (we replaced frequencies with percentages). Figure 4 summarizes the results from this experiment, pooling respondents in Studies 2 and 3.Footnote 11
Our results from the minimum wage task are more consistent with motivated responding than motivated learning, on balance. Among opponents of raising the minimum wage, we see the familiar pattern that opponents of concealed carry display in the first task. The percentage of these respondents correctly identifying the result of the minimum wage study is 62.0% in the uncongenial condition and 89.3% in the congenial condition (a). This dramatic congeniality effect is significantly attenuated by incentives. Specifically, it decreases from 27.3 to 16.3 points, which is a 40% reduction (see b). Again, this reduction is due primarily to an increase in correctness in the uncongenial condition. Incentives increase the percent correct by 8.4 points in the uncongenial condition, but their effect is null in the congenial condition.
Supporters of raising the minimum wage do not behave in a manner consistent with motivated learning (c). In fact, the congeniality effect is a significant −10.7 points without incentives, indicating that respondents are actually less likely to correctly report a congenial result than an uncongenial result. In this case, the theoretical expectation for incentives is unclear, because the bias in the control condition is neither consistent with motivated learning nor responding. We do not expect incentives to significantly alter behavior if there is no bias to reduce. Indeed, incentives do little to affect responses in either the uncongenial or congenial condition, so the congeniality effect remains negative with incentives (d).Footnote 12 Nevertheless, this finding suggests that a congeniality bias in factual learning is yet less common than the literature suggests. In sum, across the two experimental tasks in our three studies, we see some evidence of motivated learning but also quite a bit of evidence for motivated responding.
Biased Study Ratings
Does the congeniality of the hypothetical study’s result affect how positively respondents rate the study? Recall that after the concealed carry task in both Studies 1 and 2, we asked respondents to rate how “well done” and “convincing” they found the study on a 0–10 scale (see “Question Wording” in SI for exact wording). We find that the two ratings are strongly correlated, so we average them into a single rating (Cronbach’s \(\alpha\) = .82).
In Fig. 5, we plot the effect of the congeniality manipulation on this average rating among concealed carry supporters (a) and opponents (b). We estimate the overall congeniality effect and then disaggregate it by incentive condition. Consistent with our expectations, congeniality affects ratings, but more so when we incentivize respondents. Among concealed carry supporters, the overall congeniality effect is a marginally significant .25 (\(t = 1.53\), \(p = .06\)). Drilling down, we find a null effect without incentives and a significant increase of .41 with incentives (\(t = 1.78\), \(p = .04\)). Similarly, among concealed carry opponents, the overall effect is .20, which is in the hypothesized direction but not significant. And again, the effect only appears in the Incentive condition (\(t = 1.41\), \(p = .08\)).Footnote 13
One possible explanation for this pattern is that when we incentivize respondents to admit having learned an uncongenial fact, they express displeasure via the study ratings. If so, we should see a greater congeniality effect among respondents who correctly report the study’s result than among incorrect respondents. Indeed, we find that the congeniality effect in the Incentive condition only occurs among respondents answering correctly. Among concealed carry supporters answering correctly, the congeniality effect on study ratings in the Incentive condition is 1.05 (\(t=3.31\), \(p < .001\)). Among concealed carry opponents answering correctly, effect in the Incentive condition is 1.39 (\(t=3.43\), \(p < .001\)). These findings are merely suggestive, because answering correctly is of course an endogenous variable.
Discussion
Increasing affective polarization (see, e.g., Iyengar et al. 2012; Iyengar and Westwood 2015) has brought long-standing concerns over motivated reasoning to the fore. These concerns were amply highlighted during the recent presidential election, with numerous stories of people quickly latching on to congenial information, whatever the source, making the rounds. Motivated learning is related to such incidents, but it refers to something even more alarming: people coming across the same information supporting an unambiguous conclusion and yet walking away with different beliefs about what this information supports.
Motivated learning is particularly troubling because it has the potential to upend the benefits bestowed by the information age—easy access to reliable, trustworthy, objective information about a variety of politically relevant topics. Motivated learning means that people may possess different facts even after being exposed to the same information, and may reach preferences that are very different from what they would have if they learned in an unbiased manner (Gilens 2001; Hochschild 2001). Given that motivated learning facilitates disagreement over what the facts are, it also implies that the possibility of democratic deliberation and compromise is slimmer still (Shapiro and Bloch-Elkon 2008; Muirhead 2013).
Our findings confirm that motivated learning occurs in some cases, but also suggest that estimates of motivated learning are upwardly biased. The concealed carry experiment results suggest that supporters of concealed carry likely learn in a motivated manner. Manipulating the congeniality of the data through a minor change has a significant impact on the probability of reporting the correct answer among these respondents. And the accuracy incentives fail to change this tendency. On the other hand, a portion of what is thought to be motivated learning is instead motivated responding. When other respondents are offered a mere $0.10 to report their beliefs accurately, estimates of motivated learning decline sharply. And given that incentives could not have affected how respondents initially processed the data, incentives very likely identify the artifactual component of the evidence for motivated learning.
However, there are other potential explanations for why incentives reduce estimates of motivated learning. The lure of making additional money may cause respondents to choose the answer that they believe the experimenter favors, rather than the one they think is right. Or, respondents may take monetary incentives as a cue that the congenial answer is incorrect. In both cases, the decline would be artifactual. We explore both possibilities, finding little empirical support for either (see “Assessing Experimenter Demand Effects” section in SI, and in particular, SI Tables 9 and 10). On balance, the data suggest that incentives reduced bias in estimates of motivated learning, rather than increase it.
It is possible that the data, even accounting for incentives, still overstate the extent to which people learn (or mislearn) in a biased manner. It is likely that a non-trivial proportion of respondents in the experimental tasks simply tune out because they find the wording too complex, or because they are disinterested in the question. Such respondents may pick an answer by taking a blind guess or by relying on their priors. They are aware that they haven not really learned the result supported by the study. Other respondents may use cheap heuristics to deduce the correct result. And some of these respondents also likely know the fallibility of doing so, and prorate their certainty in their answer accordingly. A simple correct/incorrect scoring does not capture either of these concerns, instead treating each answer as evidence of learning a particular result. To assess these concerns, we asked people how confident they were about the answers they gave after they had selected them in Study 3. Only 13% of respondents are certain of their answer in the concealed carry task without incentives. Even fewer, 10%, are certain and incorrect. This suggests that the proportion of people who become confidently misinformed due to motivated learning, which is the gravest concern, is not very large.Footnote 14
Two other pieces of evidence suggest that motivated learning is less common than what conventional estimates reveal. Firstly, we find that respondents recall the data in a largely unbiased manner. Secondly, among those who support increasing the minimum wage, the congeniality effect is negative—respondents were more likely to report the correct result in the uncongenial condition than in the congenial condition. This is exactly the opposite of what the theory of motivated learning would predict, which suggests that motivated learning may not be a very common feature of learning more generally.
Lastly, data from the minimum wage task suggest that when the task is made easier, motivated learning all but disappears. This may happen because when the truth is transparent and easy to grasp, even people who are otherwise prone to motivated reasoning have trouble denying it (see also Bisgaard 2015, who shows that when economic conditions are unambiguous, as in the 2008 recession, partisans’ factual beliefs about the economy converge). The finding is consistent with bounded rationality, a mechanism proposed for motivated learning. When little effort is required, motivated learning is perhaps not as much an issue. This also suggests that treatments designed to teach people how to infer data correctly from the contingency table ought to prove efficacious. So should treatments that give people more time to learn, and incentivize attention.
More work is needed to understand why incentives reduced the congeniality effect among those who oppose concealed carry, and not among those who support it. One possibility is that supporters of concealed carry have a stronger affective reaction to the issue and experience a greater directional pull toward their preferred conclusion. Another possibility is that this group of respondents differs on certain traits, such as need for cognition, that likely affect the extent to which people learn in a biased learning (e.g., Arceneaux and Vander Wielen 2013).
The fact that incentives reduce estimates of motivated learning has implications for survey measurement. It suggests that in order to precisely measure certain variables, such as factual beliefs with partisan implications, accuracy incentives may be the best way to obtain unbiased estimates. In extending this method to experiments on learning, our study contributes to the burgeoning literature on motivated responding, which combines the insights of survey satisficing with literature on group-based reasoning and affect (Bullock et al. 2015; Prior et al. 2015).
To put the results in perspective, however, note that estimates of motivated learning and motivated responding come from a unique task. The task of interpreting a contingency table is unusual for many people. It is also the case that the data in the table is arranged so that common heuristics (e.g., focusing on the upper-left cell) always lead to the wrong answer. While news media occasionally present studies using tabular or graphical formats, it is not the norm. And in such cases, it is likely that training people on how to interpret tables or incentivizing how attentively they process the data will significantly reduce bias.
Finally, the results suggest that incentivizing accurate reporting among respondents comes at a price. Respondents rate the study as less well done and less convincing when incentivized to report the uncongenial fact. Thus, while it is possible to increase accuracy in reported factual beliefs, doing so may increase bias in interpretations of these facts. Our finding is consistent with other research showing that even when people agree on factual information, they nevertheless tend to interpret the information in a motivated manner (e.g., Gaines et al. 2007; Bisgaard 2015). While the observed reduction in motivated learning is welcome news for the prospect of democratic accountability, reducing differences between groups in one place may increase differences between them in another place.
Notes
Nyhan and Reifler test motivated learning using two-sided information flows. They instill false beliefs in respondents via misleading news stories and then attempt to reduce misperceptions with corrective stories. While this design lends external validity to their study, a one-sided context without contradictory information would afford a stricter test of motivated learning.
It is possible, even likely, that motivated responding also operates outside the survey context, explaining which beliefs people reveal in discussions with social networks, for example. However, in this paper, we focus on motivated responding in surveys.
We do not incentivize study ratings, because unlike the question about the study’s result, ratings are inherently subjective. The logic of providing incentives to be accurate on subjective questions is unclear. Correspondingly, even if we were to provide incentives, we would not be able to interpret the results unambiguously.
Attitudes in our study are similar to Americans’ attitudes in two nationally representative surveys. A CBS/New York Times Survey from January 2013 finds that 34% of Americans favor “a federal law requiring a nationwide ban on people other than law enforcement carrying concealed weapons” (including 19% of Republicans a 52% of Democrats). And an Associated Press/GfK Poll from January 2015 finds that 77% favor raising the federal minimum wage (from $7.25/h).
Kahan et al. (2017) define congeniality on the basis of party identification and ideology. However, overlap between a partisanship - ideology composite and attitudes toward concealed carry is considerably short of 100%. Across the three studies, 47% of self-described liberal Democrats oppose a ban on concealed carry, and 15% of self-described conservative Republicans favor such a ban. We therefore opt for coding congeniality in terms of the attitude most directly related to the data being. Recoding congeniality on the basis of party identification and ideology, following Kahan et al., results in a substantively similar congeniality effect (see “Subsetting Respondents by ‘Ideological Worldview’” in SI).
We subset high-numeracy respondents in Study 2 to ensure commensurability with Study 1. For concealed carry task results separated out by study, see SI Figs. 5 and 6 in “Covariance Detection Task Results by Study”.
For equivalent logistic regressions, see SI Tables 7 and 8 in “Concealed Carry Task Results as Logistic Regressions”.
In Study 2, only 129 respondents (15%) oppose raising the minimum wage. Pooling them with opponents in Study 3 yields a large enough sample to analyze. We obtain substantively similar results we see when analyze each study separately (see SI Figs. 7 and 8 in “Covariance Detection Task Results by Study”).
One possible explanation for this finding is that behavior in this task was affected by the previous task on concealed carry, since we did not randomize task order. We explore this possibility in “Assessing Spillover” in the SI but find little support for it. Another, related possibility is that respondents mistook the row labels in the second task to be parallel to the row labels in the first task (i.e., cities enacting the given policy are always in the first row). Properly testing this hypothesis (e.g., by fully randomizing row and column labels, and task order) is beyond the scope of this study.
Pooling concealed carry supporters and opponents, we find that ratings increase from 4.79 if the study is uncongenial to 5.02 if the result supported by the study is congenial (\(t = 1.79\), \(p = .04\)). With incentives, the effect is .42 (\(t=2.30\), \(p=.01\)), and in the absence of incentives, the effect is only .08 (\(t=.46\), \(p=.32\)). All t-tests reported here are one-tailed.
In the minimum wage task, 23% of respondents are certain of their answer, while only 4% are certain and incorrect. We present these results in more detail in SI Table 11 in “How Confident are Respondents in Their Answers?”.
References
Ahler, D., & Sood, G. (2016). The parties in our heads: Misperceptions about party composition and their consequences. Working paper.
Arceneaux, K., & Vander Wielen, R. J. (2013). The effects of need for cognition and need for affect on partisan evaluations. Political Psychology, 34(1), 23–42.
Bartels, L. M. (2002). Beyond the running tally: Partisan bias in political perceptions. Political Behavior, 24(2), 117–150.
Bechlivanidis, C., & Lagnado, D. A. (2013). Does the “why” tell us the “when”? Psychological Science, 24(8), 1563–1572.
Berinsky, A. J., Huber, G. J., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368.
Bisgaard, M. (2015). Bias will find a way: Economic perceptions, attributions of blame, and partisan-motivated reasoning during crisis. The Journal of Politics, 77(3), 849–860.
Blais, A., Gidengil, E., Fournier, P., Nevitte, N., Everitt, J., & Kim, J. (2010). Political judgments, perceptions of facts, and partisan effects. Electoral Studies, 29(1), 1–12.
Bolsen, T., Druckman, J. N., & Cook, F. L. (2014). The influence on partisan motivated reasoning on public opinion. Political Behavior, 36(2), 235–262.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.
Bullock, J. G., Gerber, A. S., Hill, S. J., & Huber, G. A. (2015). Partisan bias in factual beliefs about politics. Quarterly Journal of Political Science, 10(4), 519–578.
Chaiken, S., & Maheswaran, D. (1994). Heuristic processing can bias systematic processing: Effects of source credibility, argument ambiguity, and task importance on attitude judgment. Journal of Personality and Social Psychology, 3(66), 460–473.
Chambers, J. R., Swan, L. K., & Heesacker, M. (2014). Better off than we know: Distorted perceptions of incomes and income inequality in america. Psychological Science, 25(2), 613–618.
Chambers, J. R., Swan, L. K., & Heesacker, M. (2015). Perceptions of U.S. social mobility are divided (and distorted) along ideological lines. Psychological Science, 26(4), 413–423.
Crawford, J. T., Kay, S. A., & Duke, K. E. (2015). Speaking out of both sides of their mouths: Biased political judgments within (and between) individuals. Social Psychological and Personality Science, 6(4), 422–430.
Crawford, J. T., & Xhambazi, E. (2015). Predicting political biases against the Occupy Wall Street and Tea Party movements. Political Psychology, 36(1), 111–121.
Dawson, E., Gilovich, T., & Regan, D. T. (2002a). Motivated reasoning and performance on the Wason selection task. Personality and Social Psychology Bulletin, 28(10), 1379–1387.
Dawson, E., Gilovich, T., & Regan, D. T. (2002b). Motivated reasoning and susceptibility to the “Cell A” bias. Manuscript submitted for publication.
Ditto, P. H., & Lopez, D. F. (1992). Motivated skepticism: Use of differential decision criteria for preferred and nonpreferred conclusions. Journal of Personality and Social Psychology, 4(63), 568–584.
Ditto, P. H., Scepansky, J. A., Munro, G. D., Apanovitch, A. M., & Lockhart, L. K. (1998). Motivated sensitivity to preference-inconsistent information. Journal of Personality and Social Psychology, 1(75), 53–69.
Edwards, K., & Smith, E. E. (1996). A disconfirmation bias in the evaluation of arguments. Journal of Personality and Social Psychology, 71(1), 5.
Gaines, B. J., Kuklinski, J. H., Quirk, P. J., Peyton, B., & Verkuilen, J. (2007). Same facts, different interpretations: Partisan motivation and opinion on Iraq. Journal of Politics, 69(4), 957–974.
Gerber, A. S., & Huber, G. A. (2009). Partisanship and economic behavior: Do partisan differences in economic forecasts predict real economic behavior? American Political Science Review, 103(3), 407–426.
Gilens, M. (2001). Political ignorance and collective policy preferences. American Political Science Review, 95(2), 379–396.
Gilovich, T. D. (1991). How we know what isn’t so: The fallibility of human reason in everyday life. New York: The Free Press.
Hastorf, A. H., & Cantril, H. (1954). They saw a game: A case study. The Journal of Abnormal and Social Psychology, 49(1), 129.
Hochschild, J. L. (2001). Where you stand depends on what you see: Connections among values, perceptions of fact, and political prescriptions. In J. H. Kuklinski (Ed.), Citizens and politics: Perspectives from political psychology. New York: Cambridge University Press.
Iyengar, S., Sood, G., & Lelkes, Y. (2012). Affect, not ideology: A social identity perspective on polarization. Public Opinion Quarterly, 76(3), 405–431.
Iyengar, S., & Westwood, S. J. (2015). Fear and loathing across party lines: New evidence on group polarization. American Journal of Political Science, 59(3), 690–707.
Jacobson, G. (2010). Perception, memory, and partisan polarization on the Iraq War. Political Science Quarterly, 1(125), 31–56.
Jerit, J., & Barabas, J. (2012). Partisan perceptual bias and the information environment. Journal of Politics, 74(3), 672–684.
Kahan, D. M. (2013). Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision Making, 8(4), 407–424.
Kahan, D. M., Peters, E., Dawson, E. C., & Slovic, P. (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy, Forthcoming. Yale Law School, Public Law Working Paper No. 307. Available at SSRN: https://ssrn.com/abstract=2319992.
Kahneman, D. (2013). The marvels and flaws of intuitive thinking. In J. Brockman (Ed.), Thinking: The new science of decision-making, problem-solving, and prediction. New York: Harper Collins.
Kim, S., Taber, C. S., & Lodge, M. (2010). A computational model of the citizen as motivated reasoner: Modeling the dynamics of the 2000 presidential election. Political Behavior, 32(1), 1–28.
Kruglanski, A. W., & Webster, D. M. (1996). Motivated closing of the mind: “Seizing” and “freezing”. Psychological Review, 103(2), 263–283.
Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480–498.
Lau, R. R., Sears, D. O., & Jessor, T. (1990). Fact or artifact revisited: Survey instrument effects and pocketbook politics. Political Behavior, 12(3), 217–242.
Lebo, M. J., & Cassino, D. (2007). The aggregated consequences of motivated reasoning and the dynamics of partisan presidential approval. Political Psychology, 28(6), 719–746.
Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098–2109.
Luskin, R. C., & Bullock, J. G. (2011). “Don’t know” means “don’t know”: DK responses and the public’s level of political knowledge. The Journal of Politics, 73(2), 547–557.
Luskin, R. C., Sood, G., & Blank, J. (2013). The waters of Casablanca: Political misinformation (and knowledge and ignorance). Unpublished manuscript.
Lyons, J., & Jaeger, W. P. (2014). Who do voters blame for policy failure? Information and the partisan assignment of blame. State Politics & Policy Quarterly, 114(3), 321–341.
Mata, A., Garcia-Marques, L., Ferreira, M. B., & Mendonça, C. (2015a). Goal-driven reasoning overcomes cell D neglect in contingency judgements. Journal of Cognitive Psychology, 27(2), 238–249.
Mata, A., Sherman, S. J., Ferreira, M. B., & Mendonça, C. (2015b). Strategic numeracy: Self-serving reasoning about health statistics. Basic and Applied Social Psychology, 37(3), 165–173.
Muirhead, R. (2013). The case for party loyalty. In S. Levinson, J. Parker, & P. Woodruff (Eds.), Loyalty. New York: New York University Press.
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.
Nyhan, B. (2010). Why the “death panel” myth wouldn’t die: Misinformation in the health care reform debate. The Forum, 8(1), 5.
Nyhan, B., & Reifler, J. (2010). When corrections fail: The persistence of political misperceptions. Political Behavior, 32(2), 303–330.
Palmer, H. D., & Duch, R. M. (2001). Do surveys provide representative or whimsical assessments of the economy? Political Analysis, 9(1), 58–77.
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188.
Prior, M. (2013). Media and political polarization. Annual Review of Political Science, 16, 101–127.
Prior, M., & Lupia, A. (2008). Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science, 52(1), 169–183.
Prior, M., Sood, G., & Khanna, K. (2015). You cannot be serious: The impact of accuracy incentives on partisan bias in reports of economic perceptions. Quarterly Journal of Political Science, 10(4), 489–518.
Sears, D. O., & Lau, R. R. (1983). Apparently self-interested political preferences. American Journal of Political Science, 27(2), 223–252.
Shani, D. (2006). Knowing your true colors: Can knowledge correct for partisan bias in Political perceptions? Presented at the Annual Meeting of the Midwest Political Science Association, Chicago, IL.
Shapiro, R. Y., & Bloch-Elkon, Y. (2008). Do the facts speak for themselves? Partisan disagreement as a challenge to democratic competence. Critical Review, 20(1–2), 115–139.
Stroud, N. J. (2010). Polarization and partisan selective exposure. Journal of Communication, 60(3), 556–576.
Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50(3), 755–769.
Thibodeau, P., Peebles, M. M., Grodner, D. J., & Durgin, F. H. (2015). The wished-for always wins until the winner was inevitable all along: Motivated reasoning and belief bias regulate emotion during elections. Political Psychology, 36(4), 431–448.
Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E. (2012). Development and testing of an abbreviated numeracy scale: A Rasch analysis approach. Journal of Behavioral Decision Making, 26(2), 198–212.
Wilcox, N., & Wlezien, C. (1993). The contamination of responses to survey items: Economic perceptions and political judgments. Political Analysis, 5(1), 181–213.
Acknowledgements
We would like to thank Princeton Research in Experimental Social Science (PRESS) for financial support. We are also very grateful to Dan Kahan for generously sharing experimental design details; Doug Ahler, Martin Bisgaard, Katie McCabe, Peter Mohanty, and Markus Prior for insightful comments on a previous draft; participants of the PRESS workshop for suggestions about the experimental design; and finally, the editor of this journal and three reviewers for their critical feedback and guidance. The data and code necessary to replicate the results in this paper are available at https://dataverse.harvard.edu/dataverse/polbehavior.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Khanna, K., Sood, G. Motivated Responding in Studies of Factual Learning. Polit Behav 40, 79–101 (2018). https://doi.org/10.1007/s11109-017-9395-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11109-017-9395-7