1 Introduction

The extent and consequences of changes to the Earth’s climate in coming decades are highly uncertain (Parry et al. 2004; Pereira et al. 2010). Not only is the climate system impossible to predict, but so are the highly complex human and natural systems that climate impacts upon (Stainforth et al. 2005). These systems also interact strongly, so that small uncertainties within one can be amplified through other systems (e.g. Lavorel et al. 2006; Dawson et al. 2011). Nevertheless, because the impacts of climate change depend to a large extent on the actions (and reactions) of people and institutions, accurate assessment of the magnitude of these uncertainties is needed. Attempts to lessen, mitigate and respond to climate change are global political priorities (nominally at least) and social, economic and environmental necessities (e.g. Anderson and Bows 2011). Decision-makers therefore require accessible and accurate information about the effects of climate change, and about the limitations of our knowledge of these effects (Polasky et al. 2011).

Many models have been developed to explore the likely course, magnitude and consequences of climate change (IPCC 2013). However, individual models tend to focus on single physical or socio-economic processes that are implicitly assumed to operate in isolation, neglecting the many feedbacks that occur between these processes in reality (Warren 2011). To tackle this problem, models are combined increasingly in large ensembles or in integrated assessment platforms (IAPs) (e.g. Rowlands et al. 2012; Ibáñez et al. 2013). The CLIMSAVE IAP is one example of this, combining 10 different meta-models focused on distinct sectors or processes (Harrison et al. 2013; this issue; Holman et al. 2013). IAPs of this kind can be used to explore impacts, mitigation and, potentially, adaptation options under climate change (Füssel 2010). The CLIMSAVE IAP is specifically designed to allow exploration of cross-sectoral impacts and the potential for adaptation to offset vulnerabilities across Europe.

However, where disparate models are combined, particular attention must be paid to the reliability of the generated results. A major source of potential errors and uncertainties in these results is the design of models themselves, including the underlying data used, the parameters that are included, and the treatment, simplifications or assumptions concerning the interactions of these parameters (Murphy et al. 2004). It is crucial to understand these model-based uncertainties in any application, but especially where they may combine in previously unconsidered and potentially unpredictable ways (van Vuuren et al. 2009). Where simplified meta-models are employed, as in the CLIMSAVE IAP, investigation of model-based uncertainty becomes an even greater imperative. Hence, a range of methods have been established to quantify model uncertainties, some of which have been applied to the CLIMSAVE IAP (e.g. Stainforth et al. 2005; Dunford et al. 2014).

Another class of uncertainties relates to assumptions about future climatic, economic or social change, made to allow consideration of potential options for mitigation (Moss et al. 2010). Such assumptions are widely made in the development of scenarios that describe plausible but unpredictable conditions at some future point in time (Mote et al. 2011). Consequently, scenarios themselves incorporate substantial uncertainties about their plausibility and robustness (although it must be remembered that scenarios are intended to be exploratory rather than predictive). These uncertainties are more difficult to identify and quantify but, once again, a number of methods have been established and used with the CLIMSAVE IAP (Mote et al. 2011; Dunford et al. 2014).

The CLIMSAVE IAP, in common with other similar model combinations, has therefore a number of important sources of uncertainty. It is crucial to understand these uncertainties because the IAP is intended to provide policy-makers and others with information about the impacts of climate change and potential political, economic and social responses to minimise vulnerabilities (Harrison et al. 2013). Thus, a rigorous investigation has been performed of the qualitative uncertainties in the input parameters of the CLIMSAVE meta-models (Dunford et al., in review). A sensitivity analysis has also quantified the effect of variations in single parameters on the IAP outputs (Kebede et al. (2014)).

Here we investigate the form and extent of aggregate uncertainties in IAP outputs arising from uncertainties in input parameters (and therefore the underlying model and scenario uncertainties). We assign values to input parameters to explore joint parameter uncertainty space, and summarise IAP outputs using a small number of metrics operating at different spatial scales, intended to capture the range of impacts of climate change on human and natural systems. This allows quantification of the uncertainty of climate change impacts under different socio-economic scenarios. The analysis is also intended to test the robustness of the CLIMSAVE scenarios to convergence or divergence across the range of model and scenario uncertainties, allowing ranges of output variables to be identified that are plausible across different scenarios with an ascribed level of uncertainty.

2 Methods

2.1 IAP details

The CLIMSAVE IAP provides a method for assessing the cross-sectoral impacts of, and potential adaptation to, climate change in Europe (Harrison et al. 2013; Jäger et al. 2014). It incorporates 10 meta-models with the over-arching aim of exploring the possible future development of interacting climate, human and natural systems (Harrison et al. (2014a)); Holman et al. 2013). The IAP has been used to simulate four scenarios of future socio-economic change developed through a stakeholder participatory process (Kok et al. 2014) in combination with climate change scenarios (Dubrovsky et al. 2014). These scenarios span a range of uncertainties relating to political and economic change, emission scenarios, climate sensitivities and climate models, and reflect a large number of different plausible futures.

The climate scenarios are based on the IPCC emissions scenarios A1b, A2, B1 and B2 (Dubrovsky et al. 2014). Climate variables also derive from one of five global climate models and three levels of climate sensitivity (see Table 1, Online Resource 2). The socio-economic scenarios are named Icarus, Should I Stay or Should I Go, Riders on the Storm and We are the World (Harrison et al. (2014b)) and are distinguished in terms of economic development and the provision of ‘solutions by innovation’. We are the World is the most prosperous scenario, and also the most politically and socially stable. In Icarus, economic growth stalls and political instability occurs, as it does in Should I Stay or Should I Go, which also assumes a series of economic crises. Riders on the Storm is also economically unstable, but political stability counteracts this. A total of 89 parameters are used to quantify each scenario based on stakeholder views and expert judgment. Further details about the IAP and its constituent meta-models are available in Online Resource 1.

Table 1 Descriptions of the selected impact metrics
Full size table

2.2 Parameter sampling procedure

Information about the ranges of values taken by input parameters, and the importance of these ranges, were available to this analysis from Dunford et al. (in review), who investigated model and scenario uncertainties using a qualitative method (their results are incorporated in the IAP interface and the parameter ranges given in Online Resource 2; see also Holman et al. 2013), and Kebede et al. (2014) who investigated the sensitivity of the IAP outputs to variations in individual input parameters. Parameter value ranges incorporate information about data, model and scenario uncertainties (derived from stakeholder input and model developers’ judgements), and are scenario-specific and conditionally independent given the scenario. Each parameter has a ‘possible’ range that defines the most extreme values that a parameter could take under each scenario, a ‘credible’ range that defines the most probable values, and a ‘central’ value judged to be the most probable single value. These ranges and values were interpreted probabilistically to produce probability density functions (PDFs). In doing this, the assumptions were that: a) the central (‘most likely’) value of a parameter’s range represented the mode of its PDF; b) the credible range of a parameter spanned most of the PDF, quantified here as approximately 90 %; c) the possible range of a parameter (now set to approximately 10 % of the PDF) gave the limits of the PDF, so that values outside this range could not be taken.

Beta distributions were chosen to represent parameter PDFs because of their flexibility, compatibility with the above assumptions and limited range, meaning that truncation was not required (all of which make them more suitable than, for example, normal distributions). They have also been used in previous uncertainty studies (e.g. O’Hagan 1998; Heath and Smith 2000) and provide a good approximation of the underlying (and not fully known) uncertainties in the physical processes being modelled. Separate distributions were fitted to each parameter in each scenario, using an online tool that fits beta distributions to observational data (AAHS 2013). Fits were calculated on the basis of the mode (the central parameter value) and the closest of the 5 % or 95 % limits (the lower or upper credible range limits), both of which were scaled so that the possible range spanned the interval [0, 1]. This ensured that all assumptions were satisfied, although it was not possible to precisely define the range of probabilities within the credible ranges, especially where distributions were strongly asymmetrical and skewed (which was rare). The resulting distributions were visually checked and judged to adhere to the probabilistic interpretation of the parameter ranges better than other distributional functions. Only 21 of the 89 parameters used by the IAP were relevant and appropriate to this analysis, and were selected according to their influence on the climate change impacts measured here (Kebede et al. 2014). A more detailed description of the parameter sampling procedure is given in Online Resource 2.

Samples were taken from every PDF for each of the four scenarios using the Sobol quasi-random sequences method, a Monte Carlo resampling procedure implemented in the soboljansen function in R package sensitivity (Jansen 1999; Sobol et al. 2007; Saltelli et al. 2010; Pujol et al. 2013). This variance-based method was chosen for its lack of assumptions, its ability to generate model parameterisations that systematically explore joint parameter uncertainty space, and its (relatively) low computational cost of N(k + 2), where N is the number of initial samples and k the number of parameters included in the analysis. We tested different initial sample sizes (of 250 and 50) in order to establish the minimum number of IAP runs, and ran a total of 9,200 parameterisations (1,150 from each scenario with 50 samples of 21 parameters and 5,750 from the scenario with 250 samples) through the IAP in batch files to the year 2050, with parameters that were not included in the analysis being held at their default (central) values. More details about method selection and implementation are given in Online Resource 3.

2.3 Analysis of results

Each IAP run produced results expressed in 174 output metrics, at the scale of 23,871 individual grid cells (10′ × 10′) across Europe (the 27 EU member states as of June 2013 together with Norway and Switzerland). In order to make results manageable and easily interpretable, we focused on six main metrics that illuminate the predicted sectoral impacts of climate change, and which formed the basis of the previous sensitivity analysis (Kebede et al. 2014): the food available per capita, land use diversity, land use intensity index, people flooded by a 1 in 100 years flooding event, water exploitation index (the ratio of total water supply and demand) and biodiversity vulnerability index (Table 1). We expressed these metrics at the national and European levels by averaging up from the cell level, and assessed uncertainty in the results visually (by plotting distributions of the metrics) and via the standard deviations of the values taken by each metric.

3 Results

3.1 Aggregate results

Expressed at their most aggregate level, the results of our analysis show the entire range of uncertainties in the chosen output metrics, across scenarios and EU member states. An initial sample size of 50 per parameter (corresponding to 1,150 IAP runs) was found to adequately represent the IAP uncertainty ranges compared with a sample size of 250 (on the basis that the overall shape and range of uncertainties were established by the smaller number of samples; see Online Resource 3). The ranges of values for each metric (averaged across all modelled grid cells) are shown in Fig. 1. The clearest single implication of this Figure is that most of the chosen metrics were well-constrained at the European level, with obvious peaked distributions and limited variances. There are also signs of multi-modality in some cases, due either to differences between scenarios or to influential input variables with a discrete range of settings. Some of the distributions also have long tails of more extreme values that may have substantially increased probability in some scenarios.

Fig. 1
figure 1

Overall uncertainties in selected output parameters (results for all scenarios at European level)

Full size image

Biodiversity vulnerability results suggest that it is highly unlikely that there will be a net gain in biodiversity (indicated by negative values of the index) across Europe whatever climatic or socio-economic conditions develop. Conversely, uncertainty or geographic variability in land use intensity makes it impossible to predict the overall direction of any future change. Both the food per capita and water exploitation indices were peaked around relatively low values, but with long tails probably associated with intensification of agriculture. Land use diversity, however, was the most tightly constrained of the indices, suggesting that whatever the level of land use intensity, a number of different land uses are likely to persist in proximity to one another, on average.

3.2 Scenario results

Ranges of values taken under each scenario are plotted in Fig. 2. As suggested by the aggregate results above, a great deal of overlap exists between the results of different scenarios, with some overlap in every metric. Nevertheless, results remained strongly peaked in most cases, meaning that a particular range of output values can be predicted with high confidence for each scenario. Figure 2 also shows that the bi- or tri-modality visible in some cases in Fig. 1 was not simply a consequence of the combination of scenario results, but instead occurred within individual scenarios, therefore being a consequence of input parameter settings. In the case of the biodiversity vulnerability index, the second mode of the distributions often corresponded to negative values, suggesting that the direction of change is dependent upon one or a small number of input settings.

Fig. 2
figure 2

Scenario-based uncertainties in selected output parameters at European level

Full size image

Figure 2 also exposes the influence of the socio-economic conditions described by the scenarios. The biodiversity vulnerability, land use diversity and, to a lesser extent, people flooded indices were largely insensitive to scenario at the European scale. The other indices were more sensitive, and showed some strong differences between the scenarios. The Should I Stay or Should I Go scenario consistently produced the highest or some of the highest values for food per capita, intensity index, water exploitation and number of people flooded. The Icarus scenario produced some of the highest water exploitation values, but lowest food per capita, while We are the World and Riders on the Storm produced similar results for most metrics. A striking similarity was in the number of people flooded, which took a tri-modal form under every scenario, arising from the assumed level of flood protection taking one of three input values. In this and other metrics, low values tended to have slightly higher confidence, with lower variances. It is also likely that higher modes, with greater variance, produced tails of extreme values that extend across the range of values visible in the plots.

3.3 National results

National-level results were generated in order to identify countries and regions where uncertainties were largest, and therefore where the impacts of climate change may be least predictable or where model results were least reliable. The values taken by each of the six metrics at the national level, and the uncertainties associated with them, are presented in Figs. 3a–f (the number of people flooded is presented at the grid cell level due to very large differences between individual cells, which mean that national results obscure the sites of greatest uncertainty). These results show that the locations and sizes of uncertainties varied from metric to metric, but were broadly consistent across all four scenarios. There was no clear geographical trend to the uncertainty values, but the highest values of the metrics generally corresponded to the largest uncertainties.

Fig. 3
figure 3

Scenario-based uncertainties by country. Maps show the magnitude (mean across model runs) and uncertainty (standard deviation across model runs) for each index in each country. Uncertainties are represented by colour and magnitudes are represented by opacity. Means are expressed in units of each index, while standard deviations are expressed as proportions of the total range of each index across the scenarios and countries (so that a standard deviation of 0.1 spans 1/10th of the difference between the lowest and highest mean values of the index produced for any of the countries in any of the runs). The index ‘People flooded in a 1 in 100 year flood event’ is presented at cell scale rather than country scale in order to show the large variation between cells. The scenarios are labelled as follows: ICA Icarus, RotS Riders on the Storm, SIG Should I Stay or Should I Go, WatW We are the World

Full size image

Biodiversity vulnerability was consistently found to be highest in Germany, France and Greece, but uncertainty levels were very different in each country. The average food per capita was even more variable, being high but highly uncertain in north-eastern Europe, especially in Should I Stay or Should I Go, high but relatively certain in Spain and low and certain in much of the rest of Europe. Land use intensity did not correlate with food production, and was high and certain in the UK and central Europe under all scenarios, highly uncertain in Hungary and Estonia in Icarus and Riders on the Storm, and high in the Netherlands and Sweden in Should I Stay or Should I Go. Absolute values and uncertainties for land use diversity spanned wide ranges, but were consistently high in the Netherlands and Lithuania, with the greatest differences between scenarios occurring in Bulgaria, Sweden and Switzerland. The water exploitation index, in contrast, had very consistent values and uncertainties across the scenarios, and was both highest and most uncertain in southern Europe and, to a lesser extent, Belgium and the Netherlands. The number of people flooded in a 1 in 100 year flood event varied enormously between cells, with uncertainty being nearly equal to, and strongly correlated with, magnitude. Both were greatest in coastal cells in Belgium and the Netherlands, where a combination of circumstances poses considerable but unpredictable risks.

4 Discussion

The results presented here capture the total uncertainties in six metrics describing the impacts of climate change as simulated by the CLIMSAVE IAP. The metrics were chosen to represent a range of impacts on human and natural systems, and for their relevance for policy-makers and other users of the IAP. As such, their robustness to uncertainties in model and scenario inputs is important to assess. Our findings allow us to judge the reliability of the IAP itself, identify components that may be significant sources of uncertainties, and to contribute to an understanding of the actual relationships between the systems modelled by the IAP.

We find considerable uncertainties in each of the chosen metrics at all scales, with a consequent potential for substantial differences in climate change impacts, adaptation and vulnerability, especially at national or smaller scales. However, we also find clear, peaked distributions of output metric values, which may be used to predict small ranges of values with some defined level of confidence. Bi-modal and tri-modal distributions further suggest that alterations to specific initial conditions can shift and narrow the most likely range of output values.

Surprisingly, the results of the four CLIMSAVE socio-economic scenarios were not found to be consistently distinct from one another at European scale (either in their output values or uncertainties), and none were inherently more uncertain than the others - even where they describe politically and economically unstable systems, as in Should I Stay or Should I Go. Harrison et al. (2014b) and Kok et al. (2014) explain that these scenarios were designed to provide plausible, representative examples of future socio-economic change, and the similarities in their results suggest that any differences in the impacts of climate change that they might generate are not robust to the uncertainties that they contain. It has been suggested that there are limits to societal abilities to adapt to climate change (Adger et al. 2008), and so the use of plausible scenarios may mean that impacts are inevitably similar. Emissions levels and climatic sensitivity to emissions are certainly thought to have more (and perhaps more predictable) influence on the impacts of climate change (e.g. Parry et al. 2004), although other studies have found greater, if uncertain, sensitivity to socio-economics than we do (e.g. Holman et al. 2005). The Biodiversity vulnerability index, in particular, was found to be almost completely insensitive to socio-economic scenario, primarily due to the meta-model’s use of very few socio-economic inputs, and despite its position at the end of the meta-model sequence where it is subject to the propagation of uncertainties or errors in the preceding IAP components. It is notable that our findings are not the consequence of an assumption that GDP will always increase (e.g. Mendelsohn et al. 2000; Arnell et al. 2004; as challenged by e.g. Hallegatte et al. 2007), as GDP change takes negative values in some cases (Table S1, Online Resource 1).

Despite these similarities, each scenario produces non-uniform distributions of the chosen metrics that differ to some extent (e.g. in their mean or standard deviation), showing that data, model and scenario uncertainties do not entirely erode either the differences between scenarios or the existence of basically predictable relationships between input assumptions and measured outputs. Scenarios are not intended to have equal (or even known) probabilities, but within scenarios it is possible to predict specific outcomes with a defined level of confidence. This predictability, together with the nature of the most likely outcomes, reveals much about the probable future impacts of climate change as well as modelling issues that would otherwise be hidden.

In terms of specific impacts, our findings allow us to conclude that food production is highly predictable at European scale, given some socio-economic assumptions, and that it will increase with political stability except under the most dysfunctional system (Should I Stay or Should I Go, where a substantial increase in land use intensity leads to uncertain gains). Uncertainty about food production was greatest in north-eastern Europe, where climate and socio-economic uncertainties are maximised, particularly with respect to the amount of agricultural abandonment (e.g. Olesen and Bindi 2002). The European intensity of land use also correlated with political instability in the scenarios, being lowest where political and economic change was slow and well-planned (especially in We are the World), although uncertainty tended to be both greater and more widespread in the same circumstances. Land use diversity appeared to be predictable and independent of intensity, but variability at national scale suggest that large regional changes may occur. Biodiversity vulnerability was also relatively insensitive to scenario at the European level, suggesting that overall declines in biodiversity are unlikely to be arrested. This finding, though, is strongly dependent on the selection of species considered by the index, which may overemphasise species associated with agricultural habitats. Water exploitation was, broadly, lower in the more stable scenarios, and particularly uncertain in southern Europe, where uncertainty in water availability is also likely to be maximised.

The observation that land use intensity is negatively related to political stability is consistent with recognised links between rapid political, economic or demographic change and agricultural intensification (e.g. Lambin et al. 2001). Sensitivity analysis (Kebede et al. 2014) shows that land use diversity has fewer strong sensitivities than most other outputs, being strongly (negatively) affected only by five parameters, that relate to climate, population and agricultural productivity. The potential for these to have confounding effects may explain the lack of large variations in these findings (e.g. Olesen and Bindi 2002). The sensitivity of biodiversity is similar, with precipitation providing the only strong negative effects, which might explain the observed bi-modal distributions. Other studies confirm the likelihood of continuing biodiversity decline in Europe, despite uncertainties in climate projections (Bellard et al. 2012). Uncertainty about water exploitation in south-eastern Europe is probably due to strong sensitivity to GDP and precipitation change in this region, particularly under the politically unstable scenarios (Kebede et al. 2014), and reflects genuine uncertainties about water availability and use (Schröter et al. 2005). The number of people flooded in a 1 in 100 year flood event is strongly sensitive to the level of flood protection (Kebede et al. 2014), almost certainly explaining the observed tri-modal form, and we find that the lower the average number of people flooded (and hence, the greater the level of flood protection), the greater the level of certainty. This suggests that intervention can both lower the range and magnitude of impacts, even if both remain largest in areas where precipitation, population and effects of sea level are greatest (e.g. Nicholls 2004).

These findings also have implications for the utility and reliability of the CLIMSAVE IAP, which appears to be capable of predicting specific impacts of climate change that vary consistently with model inputs in a logical manner. The most politically and economically unstable scenarios (Should I Stay or Should I Go and Icarus) do not always generate the most uncertain outcomes, which suggests that the IAP can handle the relatively extreme input values they generate without compromising output reliability. However, the similarity of the results of We are the World and Riders on the Storm suggests either that the political stability they assume can overcome differences in their socio-economic backdrop, or that the IAP overstates the influence of political intervention in some way. In either case, the general correlation between magnitude and uncertainty of each metric suggests that uncertainties are likely to be more strongly related to modelled processes than to underlying data.

While it is impossible to rule out model uncertainties in any of the above results, we find no evidence that the IAP misrepresents known relationships or exaggerates uncertainties about the processes it models. It is important to recognise that very few, if any, of the modelled processes are fully deterministic or understood (Stainforth et al. 2005; Pereira et al. 2010; Dawson et al. 2011), and the uncertainties in model outputs therefore appear to be as small as can be hoped for. So, for example, the low sensitivity of water exploitation to scenario assumptions highlights the relatively strong impact that climatic factors have on this metric (Schröter et al. 2005).

Nevertheless, it is important to note that this analysis does not capture or distinguish all of the potential errors and uncertainties in model outputs. In particular, the use of simplified meta-models in the IAP (which is necessary to enable modelling across the spatial and sectoral range covered and to allow the rapid exploration of a range of possible scenarios) may introduce unrecognised inaccuracies (Ackerman et al. 2009). Conversely, meta-models may be preferable to more specialised models because they use stylised relationships that avoid unquantifiable uncertainties implicit in detailed descriptions of incompletely understood processes. In either case, though, it is possible that unexplored assumptions are made across a suite of models or meta-models, and that these introduce biases into results (Masson and Knutti 2011). An uncertainty analysis of the kind presented here depends upon accurate identification and quantification of underlying uncertainties relating to such assumptions, and our findings therefore rely on previous analysis of IAP input uncertainties (Dunford et al., in review). We also deliberately investigate the extent and form of uncertainties rather than individual possible outcomes, and the number of separate IAP parameterisations we run is therefore not large enough to detect extreme but very low-probability impacts that may nevertheless have substantial policy implications (e.g. Gerst et al. 2013). The potential importance of these low-probability events justifies further, targeted studies that consider fewer parameters but use much larger sample sizes.

The difficulty of investigating all potential sources of uncertainty, bias or error should not preclude the careful use of tools such as the CLIMSAVE IAP. The models and meta-models used here have all been subject to rigorous testing in the past (e.g. Audsley et al. 2014; Mokrech et al. 2014; Wimmer et al. 2014), and any remaining scope for inaccuracies does not mean that they are actually present. Uncertainties are inevitably large and difficult to fully describe in integrated modelling approaches, but decisions about climate change must be made, notwithstanding a lack of knowledge about its precise course, magnitude and effects. Methods to reduce or quantify these uncertainties are therefore extremely valuable, and their application to the CLIMSAVE IAP allows more confident, effective use of the tool itself, as well as illuminating the real-world processes that have been modelled.