2013/2014 sees the release of the Fifth Assessment Report of the Intergovernmental Panel on Climate Change (IPCC). The IPCC assessment reports have played a central role in bringing climate change science to the attention of policy makers, global media and the general public. Through the publication of probabilistic statements about climate risks and impacts, the IPCC has had a powerful influence on the concept of climate change in public and media discourses (Hulme 2010). It is essential, therefore, to understand how the information that the IPCC conveys is understood by members of the public.

The IPCC has previously prescribed that risk and uncertainty information should be communicated with words (e.g., ‘it is unlikely that…’) rather than numbers (‘there is a 20–30 % chance that…’), with the intended numerical ranges of these words outlined at the start of the reports (see Table 1; IPCC 2007). Research has shown that there is a high degree of variability in how these verbal probability expressions (VPEs) are interpreted (Beyth-Marom 1982; Brun and Teigen 1988; Budescu et al. 2009, 2012; Budescu and Wallsten 1985; Wallsten et al. 1993). Furthermore, the interpretations of these expressions have been shown to be affected by characteristics of the events described, such as the event’s base rate (how likely it is to occur in general–Fischer and Jungermann 1996; Wallsten et al. 1993; Weber and Hilton 1990) and its severity. Expressions referring to very negative outcomes, for example, are interpreted as denoting a higher probability than expressions referring to more neutral outcomes (Harris and Corner 2011; Weber and Hilton 1990). It has, furthermore, been shown that English speaking participants’ interpretations of these verbal probability expressions are often not in line with the IPCC’s intended probabilities (Budescu et al. 2009, 2012).

Table 1 English and Chinese likelihood scale of the Intergovernmental Panel on Climate Change
Full size table

Tackling climate change is, however, a global problem. Consequently, the IPCC reports have been published in six languages.Footnote 1 This results in an additional difficulty for the communication of risk and uncertainty information with VPEs, suggesting yet another source of variance and thus inconsistency in the understanding of risks described with VPEs. The VPEs outlined in Table 1 must be translated such that the same meaning is conveyed across languages and cultures. Little research has been undertaken on the cross-cultural interpretation and, indeed, translation of VPEs. Evidence from accounting suggests that differences will arise for both translational and cultural reasons (Davidson and Chrisman 1993, 1994; Doupnik and Richter 2003, 2004). As the world’s key emerging economic and political power, and given the potential for human behavior to mitigate climate change risks, what Chinese citizens think about climate change is of considerable importance (Geall 2011). Moreover, previous research has shown Chinese participants to have a less consistent, and less precise lexicon for describing numeric probabilities (Lau and Ranyard 1999). We therefore focussed our cross-cultural investigation of the interpretations of the VPEs used by the IPCC on China.

The IPCC provides guidance as to the numerical ranges implied by its VPEs at the outset of its reports. Budescu et al. (2009, 2012) have shown that the typical guidance–provided in isolation from the use of the VPEs–does not result in high levels of consistency between participants’ interpretations and the IPCC’s prescriptions. IPCC reports are many pages long and it is a strain on participants’ cognitive resources to have to keep track of the numerical equivalents of the VPEs used if they do not fit with participants’ natural interpretations. The closer the IPCC usage can be to the natural interpretations of participants, the more natural and effective the communication will be.Footnote 2

1 Study 1

Study 1 investigated the interpretations of the VPEs used by the IPCC across the UK and China in a homogenous sample of first-year undergraduate psychology students.

1.1 Method

Only participants who answered all questions were retained for analysis. 110 UK-based and 84 Chinese-based students from UCL and Shandong Normal University respectively were presented with the VPEs prescribed for the communication of risk and uncertainty information by the IPCC (Table 1) individually, with order of presentation randomised across participants. Chinese participants received the materials in Chinese, with the VPEs being taken from the IPCC’s (2007) Chinese assessment report. Chinese participants were presented with one additional VPE because the literal translation of the IPCC’s term for ‘unlikely’ is ‘impossible’ (http://mandarintools.com/worddict.html), which is simply an error in the Chinese assessment report. Thus, both this term and a more appropriate translation (不大可能–‘unlikely’) were included in the eight VPEs presented to Chinese participants. Before receiving the VPEs, participants were simply told that they would be presented with seven (eight for the Chinese participants) probability words and their task was to provide a number that they thought best represented the probability implied by the word. Subsequently, for each VPE participants were asked to indicate their best numerical estimate (0 [absolutely impossible]-100 [absolutely certain]) of the probability implied if an expert described a future event with the presented VPE. By asking for a single ‘best’ probability estimate (as in Budescu et al. 2012), we are ascertaining what participants conceive of as the most natural interpretation of these expressions. We assume that this is one that they will be best able to make use of in informing subsequent decisions (and if this falls outside the range prescribed by the IPCC, people will likely find reasoning with these VPEs difficult). Participants subsequently provided their gender and age.

The UK-based participants completed the study as part of a lab demonstration, whilst the Chinese-based participants were recruited outside class hours.

1.2 Results and discussion

Sample homogeneity

17.3 % of the UK sample were male, against 17.9 % of the Chinese sample, χ 2(1) = 0.01, p = .92. The age range in the UK sample was 17 to 22 years (N = 110), whilst the age range in the Chinese sample was 17 to 21 years (N = 84, after excluding a 70 year old to maintain homogeneity across the samples, and 3 participants who did not report their age). A t-test suggested that the Chinese sample was slightly older on average (Chinese mean = 19.36, SD = 0.93; UK mean = 18.73, SD = 0.90), t(192) = 4.78, p < .001. However, given the similar range and the same median (19 years), we do not see this difference as problematic in the interpretation of the results. Based on these analyses, and the fact that all participants were first year psychology undergraduate students, our samples can be considered highly comparable.

Main analyses

Our first finding confirmed that the IPCC Chinese translation of ‘unlikely’ was incorrect. The modal interpretation (51 % of participants) of this phrase as indicating a probability of zero indicates that this phrase has been mis-translated by the IPCC. Using our own translation of ‘unlikely’ in subsequent analyses, Fig. 1 compares the median estimates (with interquartile ranges [IQRs]) of the different VPEs in the UK and China.Footnote 3 Inspection of Fig. 1 suggests that there are systematic differences in numerical interpretations of the IPCC’s VPEs between the UK and China. The interpretations of the VPEs are regressive (i.e., closer to the midpoints of the scale than intended by the IPCC) in both the UK and China, but they are notably more so in China than in the UK (interpretations of the VPEs excluding ‘as likely as not’ were closer to 50 % in China than in the UK (mean percentage points from 50 % of 19.59 and 31.96 respectively), F(1, 192) = 34.74, p < .001). Although this main effect was qualified by an interaction with VPE, F(4.30,825.62) = 7.79, p < .001 (where homogeneity of variance assumptions are violated, Greenhouse-Geisser corrections are applied), the direction of the effect was consistent across all VPEs. A 7 × 2 (VPE x location) ANOVAFootnote 4 on participants’ numerical interpretations yielded main effects of VPE, F(3.08, 590.96) = 316.49, p < .001, MSE = 428.23, eta p 2 = .62, location, F(1, 192) = 11.65, p = .001, MSE = 451.89, eta p 2 = .06, and a significant VPE x location interaction, F(3.08, 590.96) = 20.10, p < .001, MSE = 428.23, eta p 2 = .10. This interaction confirms the suggestion in Fig. 2, that Chinese participants (by nature of their interpretations being more regressive than the UK participants) did not differentiate between the VPEs as much as the UK participants did.Footnote 5 There was also greater variance in the Chinese participants’ understanding of these expressions. A t-test comparing the IQRs of interpretations by Chinese and UK participants across the seven probability expressions confirmed (with marginal significance) that IQRs were greater amongst Chinese participants (Chinese mean = 25.89; UK mean = 13.46), t(6) = 2.41, p = .053.

Fig. 1
figure 1

Medians and IQRs of numerical interpretations of the VPEs in the UK and China in Study 1. Dashed horizontal lines represent the boundaries for the expressions as shown in Table 1

Full size image
Fig. 2
figure 2

Percentage of participants whose interpretations of the VPE agree with the IPCC prescriptions (with inequalities interpreted as non-strict inequalities) in Study 1

Full size image

As Table 1 shows, the IPCC has strict prescriptions for how the VPEs should be used and interpreted. However, these are not the interpretations that come naturally to most native speakers of Chinese or English, with the Chinese interpretations of the higher VPEs–‘virtually certain’ (median = 86.5 %), ‘very likely’ (median = 70 %) and ‘likely’ (median = 50 %)–far below their intended probabilities. Only two UK participants (zero Chinese participants) provided interpretations that were in line with the prescriptions of the IPCC for all seven probability expressions (if the strict inequalities in Table 1 are replaced by non-strict inequalities [e.g., ≤90 %], four UK participants [zero Chinese] provided interpretations that matched the IPCC prescriptions). Figure 2 plots the percentage of participants whose interpretations agreed with the prescriptions of the IPCC for each VPE individually (employing non-strict interpretations of the inequalities). As can be seen, there was greater correspondence between the IPCC’s prescriptions and the interpretations of the UK participants for all VPEs, with this difference being significant (p < .05, by a chi-square test) for all but ‘very likely’ (p = .08), ‘very unlikely’ (p = .78), and ‘exceptionally unlikely’ (p = .23). It is also clear, however, that correspondence between the IPCC’s prescriptions and people’s intuitive interpretations of these VPEs was poor even amongst the UK participants (c.f. Budescu et al. 2009), with three of the VPEs being interpreted in line with the IPCC’s interpretations by less than 50 % of participants in both cultures.

Finally, participants’ interpretations of these expressions have been shown to display violations of the order implied by the IPCC (Table 1) (Budescu et al. 2009). To compare the correspondence of the Chinese interpretations and the UK interpretations, we calculated the proportion of participants who provided a strict ordering that matched that implied in Table 1. Only 37 % of UK participants provided such interpretations, but this was a much greater percentage than for Chinese participants, of whom a mere 6 % provided interpretations whose order matched that implied in Table 1, χ 2(1) = 25.83, p < .001.

These results demonstrated considerable differences between how the VPEs used by the IPCC are interpreted in China as opposed to the UK. Furthermore, Chinese interpretations of these VPEs were especially variable and unaligned with the prescriptions of the IPCC.

2 Study 2

Study 1 recruited participants from two universities with the aim of maximising the homogeneity of the two samples. It is conceivable, however, that the differences reported in Section 1 were not the result of cross-cultural differences, but rather represented other systematic differences between students from these two university samples. Study 2 was therefore a replication of Study 1, but used non-university samples.

2.1 Method

80 UK participants and 105 Chinese participants were retained for analysis (although the Chinese sample included one individual who did not provide a response to the ‘very likely’ term). The study design was identical to Study 1, with two exceptions and one addition. First, the IPCC’s erroneous translation of ‘unlikely’ was not included–rather we included our own translation of this term, 不大可能 (as included in Section 1). Second, because the experiment was not computer-based, 7 different orders of the presentation booklet (containing the VPEs on separate pages) were constructed. The addition was that participants were asked for their highest level of educational attainment on a 5-point scale, ranging from “No GCSE or O levels or CSE” to “Postgraduate qualification” and their Chinese equivalents, “初中以下” (lower than junior middle school qualification) to “研究生及以上” (postgraduate degree or above), as agreed upon by JX and XD, who both have experience with the Chinese educational system (see supplementary materials for the full scales used).

UK participants were approached in Hyde Park, London. Chinese participants were approached in Quancheng Park, Jinan, Shandong Province. All participants provided informed consent and completed the experiment without remuneration.

2.2 Results and discussion

Sample

34 % of the UK sample were male, against 57 % of the Chinese sample, χ 2(1) = 9.97, p = .002. The age range in the UK sample was 16 to 70 years (N = 80), whilst the age range in the Chinese sample was 18 to 64 years (N = 105). A t-test confirmed that the Chinese sample was older on average (Chinese mean = 34.67, SD = 9.96; UK mean = 27.06, SD = 9.67), t(183) = 5.21, p < .001. Degree of educational attainment differed significantly between the two samples, χ 2(4) = 24.82, p < .001. The Chinese sample had a slightly higher level of educational attainment, with 83 % of the sample holding an undergraduate degree (or equivalent) or higher, compared with 54 % of the UK sample.

Main analyses

Figure 3 compares the interpretations of the different VPEs in the UK and China. The interpretations of the VPEs were again regressive in both the UK and China, but as in Section 1, they were notably more so in China than in the UK, F(1, 182) = 4.94, p = .028 (mean percentage points from 50 % of 18.77 and 24.07 respectively). Once again an interaction was observed between VPE and location, F(3.59, 653.90) = 21.70, p < .001. In contrast with Study 1, however, the pattern of means was not in the same direction for all VPEs. Figure 3 shows that UK participants’ estimates were closer to the midpoint than were Chinese participants’ estimates for ‘exceptionally unlikely’ and ‘very unlikely’. A 7 × 2 (VPE x location) ANOVA on participants’ numerical interpretations yielded main effects of VPE, F(3.16, 575.38) = 161.87, p < .001, MSE = 1014.46, eta p 2 = .47, location, F(1, 182) = 25.61, p < .001, MSE = 954.29, eta p 2 = .12, and a significant VPE x location interaction, F(3.16, 575.38) = 11.01, p < .001, MSE = 428.23, eta p 2 = .06. This interaction confirms the suggestion in Fig. 3, that Chinese participants, as in Section 1, did not differentiate between the VPEs as much as did the UK participants. The pattern and significance of the latter results did not change after controlling for age, gender and educational attainment. In the analysis concerning the degree of deviation from the midpoint, controlling for these factors led to the main effect of location becoming marginal, F(1, 164) = 2.99, p = .085.Footnote 6 This is probably a consequence of ‘very unlikely’ receiving more regressive interpretations in the UK than in China, as confirmed by the fact that the main effect of location is significant if ‘very unlikely’ is excluded from this analysis, F(1, 164) = 11.09, p = .001, eta p 2 = .06.

Fig. 3
figure 3

Medians and IQRs of numerical interpretations of the VPEs in the UK and China from the non-university sample in Study 2. Dashed horizontal lines represent the boundaries for the expressions as shown in Table 1

Full size image

Figure 3 shows that the variance in participants’ interpretations for the low probability terms was greater in the UK than in China. Very high interpretations of ‘exceptionally unlikely’ in English may have resulted from an inappropriate focussing on ‘exceptionally’, after one participant was overheard thinking aloud that she would use a high number for ‘exceptionally unlikely’ because ‘exceptionally’ sounded like a very positive word(!)

As in Section 1, participants’ interpretations did not match the prescriptions of the IPCC. Once more, Chinese interpretations of the higher VPEs–‘virtually certain’ (median = 80 %), ‘very likely’ (median = 70 %) and ‘likely’ (median = 50 %)–were far below their intended probabilities. No participants provided interpretations that were strictly in line with the IPCC prescriptions, although one UK participant (zero Chinese) matched these criteria if the strict inequalities are replaced by non-strict inequalities. Figure 4 plots the percentage of participants whose interpretations agreed with the prescriptions of the IPCC for each VPE individually. There was greater correspondence between the IPCC’s prescriptions and the interpretations of the UK participants for all VPEs aside from ‘very unlikely’. This difference was significant (p < .05, by a chi-square test) for all but ‘as likely as not’ (p = .49), ‘unlikely’ (p = .069), and ‘exceptionally unlikely’ (p = .42). ‘Very unlikely’ was interpreted in line with the IPCC’s prescriptions significantly more in China than in the UK.

Fig. 4
figure 4

Percentage of participants whose interpretations of the VPE agree with the IPCC prescriptions (with inequalities interpreted as non-strict inequalities) in Study 2

Full size image

Finally, only 20 % of UK participants provided interpretations whose ordering matched that implied by the IPCC (Table 1), but this was a much greater percentage than for Chinese participants, of whom a mere 9 % provided such interpretations, χ 2(1) = 4.96, p = .026.

2.3 Combining Studies 1 and 2

Study 1 was conducted with two university samples. Study 2 was conducted with two samples from outside the university. We carried out a sequential (type 1 sum of squares) ANOVA, to determine the effect of cultural location (UK vs. China) after controlling for gender and within-culture location (university or non-university sample), in that order, with age included as a covariate. The main effect of cultural location remained significant, F(1, 369) =17.91, p < .001, eta p 2 = .05, as did the interaction between VPE and cultural location, F(3.31, 852.20) = 13.96, p < .001, eta p 2 = .04. We also tested the degree to which interpretations deviated from the midpoint in the same way. The main effect of cultural location remained significant, F(1, 369) = 16.87, p < .001, eta p 2 = .04, with interpretations in China being closer to the midpoint (mean deviation = 16.30), and thus more regressive, than interpretations in the UK (mean deviation = 26.45). The interaction between VPE and cultural location also remained significant, F(3.81, 473.59) = 14.29, p < .001, eta p 2 = .04. Thus, across both studies, the pattern of findings is clear in that differences exist between London, UK and Jinan, China in how the VPEs used by the IPCC are interpreted. Given that London and Jinan are both sizeable cities (populations of over 8 million and 6 million respectively–Wikipedia, August 12th, 2013), we believe our results represent a general cross-cultural effect between the UK and China, although further research employing random sampling methods (e.g., random digit telephone dialling) would increase confidence in this conclusion.

3 General discussion

Across two studies, we observed systematic differences between Chinese and UK participants’ numerical interpretations of the VPEs used by the IPCC. Study 1 recruited psychology university students, so as to maintain a homogenous sample, whilst for Study 2 participants were approached in local parks. The results of Study 2 corroborated those of Study 1: Considerable intra-cultural variability (c.f., Budescu et al. 2009, 2012), and also systematic effects of culture on the interpretation of these expressions. Specifically, interpretations of the VPEs used by the IPCC were more variable and further from the IPCC’s prescriptions in China than they were in the UK. In contrast to Budescu et al. (2009, 2012), the present studies did not embed the VPEs within a climate change context, nor did we provide participants with a numerical translation table for these VPEs (as is provided in the IPCC reports). It is possible that had we conducted our study with these inclusions we might not have observed the same cross-cultural differences. Given, however, that Budescu et al. also observed considerable inter-individual variance in participants’ interpretations, we suspect that such a change would not have affected our fundamental conclusions. Research investigating accountants’ use of VPEs has recognised that interpretational differences can result from both cultural differences and translation errors (Doupnik and Richter 2003, 2004). We do not distinguish between these two sources of difference. It may be that a ‘better’ translation of the IPCC’s VPEs does exist. Indeed, it is clear from Figs. 1 and 3 that the Chinese interpretations are particularly problematic at the high end of the probability scale (median translations of ‘likely’ being equal to those of ‘as likely as not’- a result that is in line with Lau and Ranyard’s 1999 finding that Keneng [‘likely’ in Table 1] was used by their Chinese participants to refer to probabilities between 20 % and 95 %). Given, however, the lower number of Chinese VPEs, relative to English VPEs, generated in Lau and Ranyard’s study, these issues might not be straightforwardly overcome. The difficulties and pitfalls associated with translating VPEs into multiple languages are many. Indeed, we found that the translation of ‘unlikely’ was simply wrong. Such pitfalls are straightforwardly avoided: the use of numerical ranges obviates any requirement for translation.

That interpretations of VPEs at the high end of the probability scale showed the most cross-cultural variability is of practical relevance for a number of reasons. Most obviously, these are the events that are most likely to occur, and underestimating the likelihood of negative events has been argued to be a more costly error than an overestimate (e.g., Harris et al. 2009; Weber 1994). Moreover, past research (not investigating cultural differences) has raised particular concerns over the negative VPEs (e.g., unlikely) used at the low end of the probability scale by the IPCC. Smithson et al. (2012) have shown that interpretational inconsistencies within U.S.-based participants are greatest for these expressions, whilst Teigen et al. (2013) have demonstrated a conflict between different types of interpretation according to either a causal propensity (these events will not happen) or frequentist (these events will occur rarely) interpretation (see also, Sirota & Juanchich, 2012). With the present work demonstrating the largest cross-cultural differences at the other end of the probability scale, it shows that difficulties associated with communicating risks with words are not limited to one part of the probability spectrum.

In conclusion, interpretations of VPEs are considerably variable both across and within individuals. These interpretations do not seem well matched to the intentions of the IPCC. For intergovernmental communication, numbers are an unambiguous common currency, with no requirement for potentially ambigous translation. More recently, the IPCC has recommended that, “where there is sufficient information” (p. 3) a numeric distribution or range of probabilities be specified, rather than the VPEs in Table 1 (Mastrandrea et al. 2010). What the current work demonstrates, however, is that the VPEs in Table 1 are not an efficient form of risk communication. If the terms are supposed to correspond to numerical ranges, why not specify these ranges in all instances–to as narrow a range as is appropriate (c.f. Budescu et al. 2009)–to guard against such discrepancies in interpretation as those reported here. Following other researchers, from studies that have included numerical information (e.g., Budescu et al. 2009, 2012) or, like us, demonstrated extant issues associated with the interpretation of VPEs (e.g., Harris and Corner 2011; Patt and Dessai 2005), we therefore recommend including numbers in addition to the VPEs where they are used (see Witteman and Renooij 2003, for potential benefits associated with including VPEs alongside numeric probabilities) for one simple reason: it will improve the communication of climate-related risk and uncertainty information.