If the future were easy to predict accurately and convincingly, there would be no need for scenarios. Scenarios exist so that decision makers and those who provide them with information can make statements about the future that claim less confidence than do predictions, projections, and forecasts (Parson et al. 2007). The climate and global change communities often face what the literature calls “wicked” problems (Rittel and Webber 1973) and deeply uncertain future conditions (Morgan et al. 2009). Not surprisingly, these communities have embraced scenarios as one important method for addressing such challenges (O'Neill et al. 2008).

Despite their prevalence, fundamental questions remain about how scenarios should be developed and used (O'Neill et al. 2008). In their review of scenario literature and practice, Bradfield et al. (2006) describe what they call “methodological chaos” engendered by a “plethora of scenario development models and techniques.” Such diversity can prove valuable because different users have different requirements. But it also reflects a lack of clarity as to what scenarios are and how best to use them. While most scenario practice is not well-grounded in the relevant behavioral and decision sciences (Garb et al. 2008), there does exist a small “evaluative scenario literature” (European Environmental Agency (EEA) 2009) that documents some significant challenges faced when using scenarios and that helps point the way towards solutions.

This paper proposes a particular conceptualization of scenarios that aims to address many of the challenges faced when using scenarios to inform contentious policy debates. The concept envisions scenarios as illuminating the vulnerabilities of proposed policies, that is, as concise summaries of the future states of the world in which a proposed policy would fail to meet its goals. Such scenarios emerge from a decision support process that asks those involved with a decision to begin with a proposed policy, to seek to understand the conditions under which it would fail, and to then use this information to identify and evaluate potential alternative policies that are robust over a wide range of future conditions. Statistical cluster analyses applied to databases with the results of many simulation model runs can help identify scenarios as part of this process. In this conceptualization, scenarios become contingent on the particular policies considered, which may prove useful to decision makers, but perhaps disruptive to some existing institutional processes for developing scenarios.

Drawing on themes from the decision support literature and on successful example applications, this paper argues that this scenario concept can remedy methodological challenges faced by the climate and global change communities. As one example, scenarios are often used to support two distinct tasks: a decision structuring task that involves defining the scope of the problem, the goals, and the options under consideration and a choice task that involves picking among a menu of existing decision options. But these tasks require different attributes so that scenarios intended for one may perform poorly for the other. In particular, placing probabilities on scenarios may prove more or less useful depending on whether one aims primarily to support decision structuring or choice. Scenarios that illuminate vulnerabilities can, however, effectively support both types of tasks.

This paper first reviews some of the challenges faced when using scenarios to inform climate-related decisions, describes the proposed approach to address these challenges, illustrates the approach with three applications, and concludes with some thoughts on implications for the provision of climate information and for future scenario processes.

1 Challenge of informing climate-related decisions with scenarios

Scenarios can have many uses: informing decisions under uncertainty, scoping and exploring weakly understood issues, and integrating knowledge from diverse domains (Parson 2008). This paper focuses on scenarios intended to inform decisions, that is, cases where scenarios are part of a process of climate-related decision support. Following National Research Council (2009) this paper defines climate-related decisions as “choices by individuals or organizations, the results of which can be expected to affect climate change or to be affected by climate change and its interactions with ecological, economics, and social systems” and decision support as a “set of processes intended to create the conditions for production and appropriate use of decision-relevant information.”

The global and climate change scenario literature employs many definitions. Parson et al. (2007) define a scenario as a “description of potential future conditions developed to inform decision-making under uncertainty,” and note that they are most useful for those uncertainties resistant to formal analytic methods (Parson 2008). Scenarios differ from other types of statements about the future because they claim less confidence than those other statements; are multi-dimensional, that is contain information about a wide range of socio-economic and biophysical factors; are schematic, that is highlight essential details but not so much as to distract from large-scale patterns; and come in sets of two of more. This paper employs the Parson et. al. definition because it is decision-focused and, in contrast to some others, agnostic about the methodology used to produce the scenarios.

Informing decisions is not the most common use of climate scenarios, which most frequently provide standardized inputs for scientific studies and assessments (Parson et al. 2007). This paper will not directly address such scholarly uses, though it will suggest important tensions between the attributes of scenarios intended to coordinate research efforts and those intended to support decision-making. It is also useful to note that many scenarios initially generated to help coordinate research are ultimately for decision support. For instance, the SRES scenarios (Nakicenovic et al. 2000), originally developed by the Intergovernmental Panel on Climate Change (IPCC) primarily to help coordinate across multiple research efforts, have since been used in innumerable decision-focused analyses.

The climate community, like most scenario practitioners, draws predominantly from the intuitive logics (also called “scenario axis”) school that originated at RAND in the 1960s and is now often associated with the scenario groups at Shell Oil and the Global Business Network (Borjeson et al. 2006; Bradfield et al. 2006; Bishop et al. 2007). As described by Schwartz (1996), one of the method’s developers, the intuitive logics process includes three key elements: a decision the scenarios are meant to inform, identification of a small number of key driving forces most relevant to that decision, and storylines. In this primarily qualitative approach, practitioners compile lists of key factors in the external environment that may affect the decision and the key driving forces that may influence those factors. Using their expert judgment, practitioners then select a small number (often two) of the most important driving forces using a pair of criteria: importance to the success of the decision and the degree of uncertainty surrounding them. These driving forces (the scenario axes) define a small set of scenarios (often four) and are used to craft a narrative for each scenario and check for internal consistency.

The climate community has adapted the intuitive logics framework into what has been called a “story and simulation” approach, in which qualitative storylines about how relevant events might unfold in the future are used to parameterize quantitative models of biophysical and socio-economic processes (Garb et al. 2008). The simulations capture the well-characterized factors of the system, such as the response of the climate system to particular levels of atmospheric greenhouse gas concentrations and while the storylines capture less predictable factors such as future changes in technology and human values. The IPCC, Millennium Ecosystem Assessment, and many regional and national studies have all used this approach (Rounsevell and Metzger 2010).

1.1 How scenarios can provide effective decision support

Scenarios aim to address a fundamental challenge. Decision makers often face a multiplicity of plausible futures. They bring differing expectations and values that influence their judgments about the future while significant gaps in scientific knowledge make it difficult to adjudicate with confidence among these views. But decision makers have limited cognitive bandwidth, so need some concise summary of the futures they face. In such contexts, scenarios aim to usefully focus decision makers’ attention on a diverse but limited subset of the conditions that might await them.

The decision support literature, as reviewed for instance in National Research Council (2009), provides a useful foundation for understanding and evaluating the effectiveness of various scenario approaches. Of particular relevance here, this literature describes the five key elements of any effective decision process: 1) defining the problem in a way that opens it up to thoughtful consideration, 2) defining the objectives to be achieved, 3) laying out the alternative actions that might be taken in an attempt to achieve the objectives, 4) estimating the consequences of each alternative, and 5) evaluating the tradeoffs among the options in terms of their ability to meet the objectives. To understand how scenarios can support decisions, it proves useful to group the last two elements into the choice task, that is, selecting among a menu of available options, and the first three elements into the decision-structuring task, that is, defining the scope of the problem, goals, and options under consideration.

In addition, the decision support literature distills the criteria for effective decision support, which, in brief, should aim to improve: i) the usefulness of information so that the intended users regard it as credible, legitimate, actionable, and salient; ii) the relationships among knowledge producers and users, helping these parties to engage in mutual learning and ‘coproduction of knowledge’ while increasing mutual understanding, respect, and trust; and iii) the quality of the decision, which should include all five elements described above and be regarded by the parties as having been improved by the support received. The evaluation criteria Hulme and Dessai (2008) propose for scenarios–predictive, decision, and learning success–are consistent with this more general framework.

Finally, the decision support literature differentiates between decision support products and as processes, a distinction echoed in the scenario literature (O'Neill and Nakicenovic 2008). The decision support literature suggests that processes, that is the ways in which analysts and users interact to construct, disseminate, and apply climate scenarios, may prove at least as important as the content of the scenarios themselves.

A small “evaluative scenario literature,” reviewed in European Environmental Agency (EEA) (2009), attempts to assess the extent to which, and by which mechanisms, scenarios and the processes that create them contribute to effective environmental and climate-related decisions. This literature uses empirical evidence gathered through means such as case studies, surveys, ethnography, and psychology experiments to evaluate how scenarios contribute to one or more of the above criteria for effective decision support.

As one key theme, this literature points to scenarios’ ability to expand the range of futures and options that decision makers consider. This structuring task helps decision makers to anticipate the consequences of their actions in a future similar to the one that actually occurs. But cognitive and organizational barriers can inhibit this crucial task. Decision makers can prove over-confident (Morgan and Henrion 1990), neglect risks they believe they cannot control (Kahneman and Lovallo 1993), strive to enhance their stature by appearing more confident than they actually are (Treverton 2001), and strategically emphasize uncertainty to sway opinion for or against actions they favor. These behaviors can all make effective decisions illusive, in particular by inhibiting full consideration of the consequences of alternative actions and by blocking creative efforts aimed at expanding the range of options under consideration.

Scenarios aim to counteract such behaviors by presenting a set of plausible and contrasting futures worthy of consideration, but without any firm claims about likelihood. Participants to a decision may find such scenarios psychologically less threatening than predictions, allowing them to explore the implications of uncomfortable or contentious statements about the future before committing themselves to the necessity of accepting those implications. By presenting multiple views of the future without privileging among them, scenarios can also help facilitate communication and collaboration among individuals with differing expectations and values. Individuals who disagree with each other can still find scenarios that validate their own worldviews, thereby increasing the legitimacy and credibility of the process. “It is precisely because scenarios do not aim to predict the future, but rather bound it,” Schoemaker (1993) writes “that a consensus building approach can work even if faced with starkly different viewpoints.”

Much of the scenario literature focuses on private sector firms, with a strong focus on scenarios’ contributions to organizational learning. Managers face a flood of information, and in uncertain times may pay attention to familiar but inconsequential trends while neglecting the novel and important. An effective scenario process can provide managers with a common conceptual framework that helps them identify the important signals in the noise, and to “scan, encode, update, and understand the future as it unfolds” (Schoemaker 1993). The narrative storylines used by the intuitive logics school help facilitate this process. Storylines, with their memorable descriptions of an inexorable flow of events, help make unexperienced futures sufficiently real to capture managers’ attention and imaginations (see related discussion in March et al. 1991). At their best, such scenarios can change decision makers’ assumptions about how the world works, “compelling them to reorganize their mental models of reality” (Wack 1985). Intuitive logics school proponents claim that their combinations of driving forces and storylines help decision makers to improve their ability to learn and adjust policy over time by focusing attention on the future conditions that might suggest the need for such adjustments (van der Heijden 1996).

1.2 How scenarios can fail

The evaluative scenario literature also suggests that in many cases scenarios do not always perform as intended. The intuitive logics methods that work well with small groups of decision makers well-known to the scenario developers may not always translate well into public debates with diverse groups of participants who have differing information needs, policy preferences, expectations about the future, and values (European Environmental Agency (EEA) 2009). Global change scenarios can generate controversy because they frame issues in particular ways and serve as proxies for the need to take action (Parson 2008). Those who disagree with the policy implications may find the choice of scenarios and their key drivers arbitrary or reflecting particular interests and values (Parson et al. 2007), and may attempt to discredit the scenarios and the legitimacy of the process that produced them. Scenarios may also be sufficiently ambiguous that they generate an illusion of communication. In one striking example, an ethnographic study traces how participants in a government-sponsored scenario exercise coalesced into three groups with contradictory interpretations of the scenario axes, each publishing separate reports on their findings (van’t Klooster and van Asselt 2006).

Global change scenarios may also fail to provide effective decision support because they are only weakly connected to potential users’ concerns and worldviews. For instance, climate scenarios may focus on long-term trends with little apparent relevance to users’ near term decisions. They may lack the spatial and temporal details needed by decision makers who are concerned with local impacts and adaptation. Scenario exercises may also omit potentially important future surprises or discontinuities and thereby fail to adequately expand the range of futures that decision-makers consider. To mention just two of numerous examples in the literature, the Hart Rudman Commission (U.S. Commission 1999), which warned of a 9/11-scale terrorist attack, surveyed 20 scenario studies of U.S. national security and found that all focused on extrapolations of current concerns rather than on surprise and discontinuities. Van Notten et al. (2005) surveyed 22 scenario studies and found that none of those that used simulation models considered surprise. Postma and Liebl (2005) suggest such patterns reflect a fundamental tension in which the goal of creating a small number of detailed, internally self-consistent scenarios tends to systematically exclude surprising or paradoxical developments as inconsistent or logically impossible.

The long-standing debate on whether and when to place probabilities on scenarios (Grubler and Nakicenovic 2001; Schneider 2001; Dessai and Hulme 2004; Morgan and Keith 2008) reflects the tension between the requirements for choice and for decision structuring tasks. Private sector managers often avoid this tension by using scenarios primarily for the latter task (Schoemaker 1993), but the public sector finds it more difficult to confine them to that role.

Those who favor placing probabilities on climate and global change scenarios argue that decision makers require information about the relative likelihood of each scenario in order to choose among alternative decision options and that the experts creating scenarios have a responsibility to provide the best available information on such likelihoods. If the experts fail to provide such information, it is argued, decision makes will obtain it from other, potentially less accurate, sources. The evaluative scenario literature supports this claim, noting that many users of the SRES scenarios, which did not have any probabilities attached, often used simple devices such as counting scenarios or using a uniform distribution to develop such probabilities on their own (Parson et al. 2007). Placing probabilities on scenarios may also address some of the challenges of scenario use, for instance, countering the tendency of scenario narratives to make previously unconsidered futures appear more likely than they actually are (Morgan and Keith 2008).

Those opposed to placing probabilities on scenarios argue that such information will undercut their cognitive and organizational benefits. In contentious situations, the probabilities can become the focus of controversy and make it easier for decision makers to reject the analysis entirely or to focus on a too-narrow range of possible futures and responses. In situations where decision makers accept the probabilities, the scenarios could lose the ability to convey information about the imprecision in the probability estimates—leading to overconfidence and policies unnecessarily vulnerable to surprise. As Herman Kahn reportedly stated, “the most likely scenario isn’t.”

The global change scenario literature proposes solutions to this and other challenges. However, in seeking to address some problems, many of these proposals aggravate others. For instance, making scenarios more detailed to make them more relevant will also make them less likely and thus less representative of the full range of plausible futures. Attempts to validate scenarios (Rounsevell and Metzger 2010) so that users find them more credible will reduce the scenarios’ ability to include surprises and discontinuities and will undermine the sense of mere plausibility that enables groups of individuals with differing world-views to engage with the scenarios. The attempt to fuse precise and imprecise information through the “story and simulation” approach may fall victim to the process March and Simon (1958) called “uncertainty absorption,” where information has its caveats stripped away as it travels from producers to users. The latter interpret the information as far more certain than intended by the former. As noted by Parson (2008), scenario exercises that begin with stories and simulations often have their stories fade away leaving behind only the quantitative projections.

2 Scenarios that illuminate vulnerabilities

To address these challenges, this paper recommends choosing scenarios that provide concise summaries of the future states of the world in which a proposed policy would fail to meet its goals. Such scenarios illuminate the vulnerabilities of proposed policies.

This scenario concept emerges from a set of decision support approaches that begin with a specific decision under consideration by a specific community of users and then use questions relevant to these decisions to organize information about future climate and socio-economic conditions. The literature offers several names for such approaches, including “context-first” (Ranger et al. 2010), “decision-scaling” (Brown 2010), and “assess risk of policy” (Lempert et al. 2004; Carter et al. 2007; Dessai and Hulme 2007). All share the central idea of beginning with a proposed policy or policies; identifying future conditions where the policy fails to meet its goals; and then organizing available information about the future to help policy makers to identify potential policy responses to those vulnerabilities and to decide whether and when to adopt these responses. This ordering of analytic steps stands in contrast to the commonly practiced alternative that begins with quantitative statements about climate and socio-economic factors deemed of broad relevance and then uses these projections to help decision-makers rank the desirability of alternative decision options. Such approaches, which follow the conceptual structure of traditional probabilistic decision and risk analysis and that are sometimes characterized in the literature as “science-first” or “predict-then-act,” work well when there is widespread consensus among parties to the decision on the framing of their challenge and the scientific evidence that underlie the projected likelihood of future conditions. But such approaches can prove problematic when these conditions do not hold (Sarewitz and Pielke 2000; Morgan et al. 2009).

Our group has used a set of methods and supporting analytic tools called robust decision making (RDM) (Lempert et al. 2003; Lempert and Collins 2007) to begin a decision support process with a decision to be examined rather than with uncertainties to be characterized. RDM seeks strategies that are robust, that is, which perform well compared to the alternatives over a wide range of plausible futures under conditions of deep uncertainty, defined here as the situation where decision makers do not know or do not agree on the structure of the model relating action to consequences and on the probability distributions describing key inputs to the model(s) (Lempert et al. 2003; Morgan et al. 2009; National Research Council 2009). Of particular interest here, the approach includes a unique quantitative step called “scenario discovery” (Groves and Lempert 2007; Bryant and Lempert 2010) that formalizes and implements analytically the concept of scenarios that illuminate vulnerabilities.

RDM has been described in detail elsewhere (Lempert et al. 2003, 2006; Lempert and Collins 2007) and will be illustrated with several examples below. In brief, the analytics involve running a simulation over many (hundreds to thousands) of cases to create a database of model results. Each entry in the database records some specific set of assumptions about the future state of the world and the resulting estimate of how a proposed policy would perform (Bankes 1993). For example, an RDM analysis for a water management agency might run the agency’s planning models for each of a wide range of assumptions about future demand, climate conditions, and regulatory conditions and in each case record the agency’s reliability and cost of operations. A scenario discovery cluster analysis on the resulting database then summarizes the common characteristics of those cases where the plan fails to meet its goals. If each uncertain model input parameter is seen as a potential stressor, then scenario discovery identifies the combinations of a small number of stressors that best explain the vulnerabilities of the proposed policy.

RDM follows an interactive series of steps consistent with the “deliberation with analysis” decision support process often recommended in the literature (National Research Council 2009). Deliberation with analysis begins with many participants to a decision working together to define their objectives and other parameters, working with experts to generate and interpret decision relevant information, then revisiting choices and objectives based on this information. As shown in Fig. 1, the RDM process begins with a decision structuring exercise that defines the goals, uncertainties, and policies under consideration. Analysts next use simulation models to generate a large database of simulation model runs. Scenario discovery algorithms operating on this database then suggest for the participants’ consideration scenarios that illuminate vulnerabilities of the policies. This information then helps decision makers identify potential responses to those vulnerabilities and evaluate whether these choices are worth adopting. Within the context of this decision support process, a cluster analysis on a database on simulation model runs can merit the name “scenario” because of the way in which the results can be used to help focus decision makers’ attention on the uncertain future conditions most important to the challenges they face and to help identify and evaluate potential improvements and alternatives to the proposed strategy (Bryant and Lempert 2010).

Fig. 1
figure 1

Iterative steps in Robust Decision Making (RDM) analysis. Shaded boxes indicate steps with deliberation among participants to the decision, boxes with dark borders indicate steps with quantitative analysis, and shaded boxes with dark borders indicate steps where results of quantitative analysis directly facilitate deliberations

Full size image

Parson et al. (2007) identify three broad classes of users of climate scenarios: national governments concerned with emission reductions policies; those concerned with impacts, adaptation, and vulnerability (IAV); and energy resource managers. Three example applications, one from each of these groups, now illustrate this scenario discovery process and the scenarios it generates.

2.1 Example scenarios for national governments

As shown in Table 1, the first example aims to inform the United State government’s investments in low-carbon energy technology research. Stabilizing atmospheric concentrations of greenhouse gases will require widespread deployment of new energy-related technologies and research can help reduce the cost of meeting any target stabilization level. This example used the Global Change Assessment Model (GCAM) to project costs to the U.S. economy of meeting a 550 ppm stabilization target contingent on assumptions regarding future improvements in the cost and performance of eight low-carbon energy supply and demand technologies (McJeon et al. 2011).

Table 1 Example scenarios that illuminate vulnerabilities for three types of users
Full size table

The analysis ran GCAM for many combinations of technology assumptions and used scenario discovery to identify the key combinations that would result in high economic costs. This particular analysis focused on questions of interactions among technologies in energy markets, where some technologies are substitutes and others complements. The example did not engage federal decision makers in an iterative decision support exercise and only considered uncertainties about future technology performance rather than the wider range of future economic, political, and other uncertainties that might influence a nation’s energy technology research portfolio.Footnote 1 Nonetheless, this simplicity contributes to a useful initial example.

The analysis generated a database of model results using a full combinatorial design over three assumed levels of cost and performance for nuclear, and two levels each for solar, wind, carbon capture and storage (CCS), buildings, transportation, industry, and a group of “other” technologies, which yields of 384 cases (384 = 3*27). In each case GCAM reports the cost of achieving the emissions reduction goal with the assumed technology suite. High cost cases are defined as those where the drop in aggregate consumption exceeds some threshold.

Scenario discovery currently employs the Patient Rule Induction Method (PRIM) (Friedman and Fisher 1999), a bump-hunting algorithm that seeks to identify the range of values for a small number of uncertain model input parameters that best predict the cases in the database where the policy fails to meet its goals. The parameters chosen by PRIM are analogous to the scenario axes generated by the intuitive logics approach. PRIM seeks to optimize three metrics: density, the fraction of cases in the data cluster in which the policy fails; coverage, the fraction of all cases in the database in which the policy fails contained within the data cluster; and interpretability, the ease with which users can understand the information conveyed by the scenario and typically measured heuristically by using only a small number of parameters to define the cluster. PRIM identifies the combination of model input parameters that best predict the cases of interest by peeling away faces of the input space to generate smaller and smaller regions of progressively higher density. Improving any of the three measures often negatively impacts the others. PRIM thus provides users with visualizations that show alternative scenarios with differing balances among density, coverage, and interpretability. Users can then choose the scenario that best serves their needs.

In this example PRIM offers scenarios which constrain one (CCS only), two (CCS and transportation), and three technology parameters (CCS, transportation, and buildings) with density/coverage of 40 %/100 %, 65 %/81 %, and 94 %/58 %, respectively. For each scenario, the other technology assumptions (including wind, solar, and nuclear) prove less important. Figure 2 displays the scenario defined by two parameter constraints. Its story is simple: for policy makers crafting an energy research R&D portfolio to help the U.S. achieve stabilization at 550 ppm, the cost and performance of CCS and transportation technologies must improve, otherwise meeting such a concentration target could prove unacceptably costly, irrespective of assumptions about other technologies.

Fig. 2
figure 2

Scenario (bold lines) showing conditions leading to high costs of meeting 550 ppm greenhouse gas concentration target. FIX, REF, and ADV labels indicate technology performance that remains at current levels, follows best-estimate projections, and exceeds best-estimate projections, respectively. Outer circle in each cell shows all cases, while inner circle shows high cost cases

Full size image

Before proceeding to the other two examples, it is useful to situate this scenarios illuminating vulnerabilities concept in a broader context. The disaster risk management, resilience, and climate change adaptation literatures define vulnerability as the susceptibility to loss or damage (Adger 2006; Fussel 2007) and climate adaptation studies often employ some type of vulnerability analysis (IPCC 2007). For instance, Kirshen et al. (2008) provided decision makers in New York City plots that showed how the frequency of large-scale flooding would increase along the New York coastline due to different assumptions about sea level rise. The IPCC’s Reasons for Concern approach (Smith et al. 2009) aims to summarize the deleterious impacts implied by various greenhouse gas emissions trajectories.

This concept of vulnerability is used similarly here, albeit with a focus on particular policy and the specific sets of conditions under which it fails to achieve its goals. That is, the proposed scenario concept takes a decision theoretic, actor-oriented view that seeks to inform particular agents who can agree among one another to make choices based on their various preferences, their institutional interests, and the power and capabilities they have available. In contrast, much of the resilience and vulnerability literature takes a system view that seeks to understand the fate of multiple, interacting agents and the social, ecological, and geophysical systems within which they interact (Olsson et al. 2006; Walker et al. 2006; Berkes 2007; Nelson et al. 2007). This latter view often aims to deepen scientific understanding of the interactions among coupled human and natural systems, while the former aims to provide decision support. Despite these different purposes, the term vulnerability captures a similar idea whether applied to systems or policies.

2.2 Example scenarios for water managements (IAV)

The second example, from the IAV community, examines the conditions under which a water agency’s long-range plan would fail to meet reliability goals. This example introduces several features in addition to those above: climate information, socio-economic factors, the use of probabilistic information to inform a choice task, and analytics used within a decision support process involving the agency’s management and stakeholders.

Like most water agencies in the arid American west, Southern California’s Inland Empire Utilities Agency (IEUA) is legally required to prepare multi-decadal plans demonstrating the ability to ensure their community’s access to water. IEUA in 2005 completed an urban water management plan (UWMP) that aimed to increase ground water use by 75 % and recycled water use by 600 % by 2025 (IEUA 2005). But this plan did not consider the potential impacts of climate change, so in 2007 RAND helped the agency revisit its 2005 UWMP (Groves et al. 2008a, b, c; National Research Council 2009; Lempert and Groves 2010; Means et al. 2010).

The exercise used a simulation model, based on the WEAP (Water Evaluation and Planning) platform,Footnote 2 to project the performance of the agency’s 2005 UWMP out to 2030.Footnote 3 These projections depend on assumptions about future climate conditions, future socio-economic factors such as the agency’s ability to implement its new ground water and recycling programs, demand growth in the region, and supplies of imported water. A series of stakeholder workshops helped to define the plans to consider, their goals, and the uncertain factors of potential concern (Groves et al. 2008a, b, c).

The analysis generated a database of 200 simulation model runs with a statistically representative sample of cases over the multi-dimensional space defined by the WEAP model’s uncertain input parameters. This sample included 200 different daily weather sequences (temperature and precipitation) extending out to the year 2030 each generated by statistical downscaling projections from a multi-climate model ensemble (Groves et al. 2008a, b, c). The sample also considered a range of values for parameters describing the agency’s ability to implement its plan, future trends in demand, and other key factors. The agency management and their constituents considered those cases with shortages as ones where its plan had failed. Scenario discovery algorithms applied to this database found that of the six uncertain parameters considered, only a specific combination of three of them would cause the 2005 UWMP to suffer future shortages. As shown in Fig. 3, these are: a 10 % or larger decrease in precipitation, a 3 % or larger drop in the agency’s ability to capture precipitation as ground water, and a failure of the recycling program to meet its ambitious goals. This cluster, labeled the Dry, Flashy, Low-Recycling scenario, explains the main vulnerability of IEUA’s 2005 UWMP. If all three of these conditions were to occur over the next few decades, the plan runs serious risks of shortages. Otherwise, the plan should fare reasonably well.

Fig. 3
figure 3

Dry, Flashy, Low-Recycling scenario, as identified by scenario discovery process, that summarizes the future conditions where the IEUA’s 2005 Urban Water Management Plan would fail to provide a reliable water supply

Full size image

Analysts then compared the 2005 UWMP’s performance to that of three alternative plans over two scenarios: Dry, Flashy, Low-Recycling and Favorable Conditions, the latter being all those states of the world not contained in the former set of cases. The three alternatives add differing combinations of additional efficiency and ground water replenishment to the base case strategy, choices informed by information about the key drivers of the current plan’s vulnerabilities. As shown in Table 2, IEUA faces a tradeoff. Adopting measures to hedge against the scenario where the 2005 UWMP suffers shortages can lead to over-investment in the scenario where the plan meets its goals. Using a measure of the comparative costs of shortages and over-investment, the analysis suggests that IEUA might invest now in additional efficiency if the Dry, Flashy, Low-Recycling scenario has probability greater than 25 %. The best available evidence, including probability estimates from the AOGCM ensemble and expert elicitations for the socio-economic factors with IEUA staff and stakeholders, suggests a best-estimate likelihood for the scenario of 30 % (Groves et al. 2008b). Thus none of the alternatives in Table 2 seem robust. Based on this information, analysts worked with the agency managers to develop several alternative adaptive-decision strategies, which make specified future investments contingent on specified future observations, and repeated the RDM analysis with this more robust set of policies (Lempert and Groves 2010).

Table 2 Comparative performance of IEUA water management plans over scenarios identified by scenario discovery analyses. Shaded cells show surpluses the agency considered too high or too low
Full size table

2.3 Example scenarios for energy resource managers

The third example examines how Israel’s Ministry of National Infrastructures can best integrate natural gas into the country’s energy mix (Popper et al. 2009). This example introduces an analysis with multiple goals and one where the concept of scenarios that illuminate vulnerabilities helped facilitate deliberations among high level policy makers—in this case ministers in Israel’s coalition government–with strongly differing priorities among these goals.

Until 2004, Israel’s electricity supply drew heavily (75 %) from imported coal. Recent discoveries of large offshore gas fields offered the opportunity to shift the country’s energy mix towards natural gas. In a series of engagements with government officials, the analysis examined the risks and opportunities of different strategies for pursuing such a shift by conducting several iterations of the process in Fig. 1. The first iteration began with simple resource development strategies and used scenario discovery to understand the combinations of future conditions that would cause each strategy to fail according to one or more of the four performance measures: system cost, greenhouse gas emissions, land use, and unmet demand. Initially the analysis sought to understand what strategy for energy infrastructure expansion and fuel mix would be most robust to the uncertainties present.

The analysts subsequently used this information to craft four alternative strategies for achieving the level of supply of natural gas implied by the selected energy generation strategy. Each depended on investments in production from Israel’s domestic deep water reserves (DDW) with investments in facilities for imports of liquefied natural gas (LNG). The strategies varied in the priority and timing of these investments. Three of the strategies were adaptive, that is, adjusting investments over time in response to new information.

The analysis evaluated these strategies over many future states of the world. As expected, all four were much more robust than the strategies considered in the first iteration, that is, each performed well across a broad range of futures for each of the performance measures. Scenario discovery analysis identified one key uncertain parameter–the ratio of costs associated with LNG to those associated with DDW supplies—that explained most of the variation in performance among these strategies. As shown in Fig. 4, if policy makers knew with high confidence the value of this cost ratio, they might choose a strategy focused on DDW only (if the ratio were high) or a joint strategy that emphasized DDW (if the ratio were low). Note that each strategy is evaluated based on its performance compared to the other three, in contrast to the types of criteria used in the other two examples. In this example scenarios illuminate vulnerabilities based on a regret-based criteria (Savage 1954), as opposed to vulnerabilities evaluated according to absolute performance thresholds.

Fig. 4
figure 4

Comparative cost of four Israeli natural gas development strategies, as a function of the key driver differentiating the cost of those strategies

Full size image

Given deep uncertainty about the value of the LNG-DDW cost ratio, the analysis suggests Israel pursue investments in DDW production and prepare for, but not yet complete, construction of facilities for LNG imports. This analysis and its framing was effective at facilitating discussions on these contentious issues within the Israeli government. For instance, the analysis allowed consideration (before the events of the Arab Spring) of the impacts of surprises, such as a cut off of Egyptian gas supplies. This surprise did not change the ordering of the strategies.

3 Benefits of scenarios that illuminate vulnerabilities

Many climate-related decisions are best addressed with an iterative risk management approach, which recognizes that anticipating and responding to climate change does not constitute a single set of judgments at any one time, but rather an ongoing process of assessment, action, reassessment, and response that may continue indefinitely (IPCC 2007; America’s Climate Choices (ACC) 2010). Scenarios can contribute importantly to similar processes of organizational learning and response in business applications, but often prove less successful in the situations with the numerous and diverse stakeholders common to many climate-related decisions. In the above examples, however, scenarios did help implement an iterative risk management approach among diverse stakeholders. Here we explore some cognitive and organizational mechanisms that may enable scenarios designed to illuminate the vulnerabilities of policies to contribute successfully to such processes.

First, using such scenarios within a decision support process like RDM can provide a clean separation between those attributes that contribute to decision structuring and to choice tasks. For the former task, the scenarios retain the sense of possibility rather than prediction that makes them less threatening to those individuals holding worldviews potentially inconsistent with the scenarios’ implications. The IEUA water management and Israel energy resource examples engaged local stakeholders and government officials with deeply held skepticism regarding, respectively, climate change and the risks of natural gas. Nonetheless the scenarios allowed these individuals to evaluate potential responses to futures where these factors were prominent before committing themselves to the necessity of a response.

Such scenarios can subsequently support a choice task when the analysis identifies thresholds for values that would favor one policy over another. The IEUA example used a probability threshold, suggesting a likelihood for the Dry, Flashy, and Low-Recycling scenario beyond which the agency might consider augmenting its current plan. The Israel energy example identified a key cost parameter where values above some threshold would favor certain energy investments. In both cases, such information facilitated deliberations on the choice of a strategy.

By enabling both types of tasks, scenarios that illuminate vulnerabilities can contribute to consensus-building among stakeholders. Such scenarios can allow individuals with different values and expectations to discuss without prejudice potential responses to future conditions where a preferred strategy might fail. If such discussions help identify a robust strategy, one that performs reasonably well across all participants’ values and expectations, the participants may agree on this choice.

Second, the proposed scenario concept and the cluster-finding analytics that helps identify them, can produce scenarios whose meaning is unambiguous and which are difficult to reject as arbitrary or biased–even in situations where participants have strongly divergent views about what futures are most important, likely, hopeful, and interesting. Individuals pay more heed to information that is proximate (that is, clearly affects them) and actionable (that is, they can do something about) (Pidgeon and Fischhoff 2011). Many scenario exercises fail on these grounds because they explicitly or implicitly take on the predictive task of providing a comprehensive summary of the multiplicity of plausible futures. In contrast, the process proposed here asks a much narrower question and one of inescapable general interest–in what futures might an organization’s proposed strategies fail? Scenario discovery analytics then provides transparent, reproducible answers with quantifiable measures of merit that can be used to justify the quality of the chosen scenarios (Bryant and Lempert 2010). In perhaps its most public demonstration (though not a climate-related decision), scenario discovery identified the conditions where proposed legislation reauthorizing the United States’ Terrorism Risk Insurance Act (TRIA) would fail to save the taxpayers money (Dixon et al. 2007). The analysis proved more favorable to the legislation than official projections by the Congressional Budget Office and U.S. Treasury Department. A supporter of the bill quoted the scenario’s implications on the floor of the U.S. Senate (Dodd 2007). Opponents editorialized that the analysis was “insidious,” but did not question the choice or immediate implications of the underlying scenarios (Jenkins 2007).

Finally, the scenarios that illuminate vulnerabilities concept provides a specific focus for the interaction between scenario users and developers who must work together to specify the policies under consideration, the goals the policies seek to achieve, and the future conditions that might threaten the policies’ ability to achieve these goals (Bryant and Lempert 2010). For instance, descriptions of the combinations of key driving forces leading to policy failure can facilitate focused deliberations regarding modifications to proposed plans so they might achieve their goals over a wider range of futures. In both the IEUA and Israel energy examples, an iterative process of proposing policies, illuminating vulnerabilities, and crafting responses led to strategies more robust than those originally considered.

Scenario discovery and the decision support process in which it is embedded also provides valuable guidance regarding the value of probabilistic and other information associated with the uncertain inputs to the simulation model. The approach helps participants to a decision usefully debate which uncertainties are important and which are less so. Traditional risk analyses often ask participants to accept initial estimates of the distributions for all the potentially relevant inputs, even those for which the underlying science is weak. Deliberations may become sidetracked by doubts over probabilistic estimates that turn out to be unimportant to the identification of vulnerabilities and judgments about responses. The analysis described here only requires participants to consider such probabilistic information towards the end of the process (in the “Tradeoff Analysis” step in Fig. 1), and only for the most important subset of the uncertainties they face. As an aside, while it has traditionally been analytically difficult to assign probabilities to simulation-model generated scenarios because each represents a single point in a multi-dimensional space of model inputs (Morgan and Keith 2008), scenarios illuminating vulnerabilities represent regions in the space of model inputs, thus making their probability easier to define mathematically.

As one important value of information benefit, choosing scenarios that illuminate vulnerabilities can also improve the ability of model-facilitated exercises to address surprise. The process described here aggressively seeks cases where a proposed strategy fails to meet its goals, including those cases some might regard as surprising. But more importantly, a decision-focused criterion for information relevance provides a much-needed filter for which potential surprises should compete for decision makers’ attention. The Israel energy analysis, for instance, showed that a potential shutoff of Egyptian natural gas did not affect the relative ranking of the strategies under consideration. Highlighting only those futures that affect the robustness of proposed strategies can sufficiently constrain the search for surprises to make the endeavor a productive analytic exercise (Lempert et al. 2002; Lempert 2007).

4 Implications for the provision of climate and other information

In his later work, the philosopher John Rawls advocated a “political, not metaphysical” approach to agreement among parties with diverse expectations and interests, arguing that people might more easily reach consensus on specific actions to undertake rather than on the general principles, comprehensive doctrines, or metaphysical commitments that might lead one to support those actions (Shapiro 2003). Most scenario concepts recognize a diversity of worldviews. But many approaches begin with what we might call a metaphysical ambition, seeking to summarize all that can happen in the future comprehensively enough so that everyone in a large and diverse audience can find truth in the description. In contrast, the concept of scenarios that illuminate vulnerabilities, as part of a decision support process that seeks robust strategies, aims at a narrower goal. The scenarios only seek to summarize those futures most important to identifying vulnerabilities and judging responses in a way that can engage people with differing values and expectations.

This scenario concept has some important implications for the provision of climate and other types of scientific information. These implications flow from a process in which the scenarios depend on the particular policies the users choose to consider. For instance, some government agencies now operate websites that provide local climate projections for communities in their jurisdictions. Generally, these services offer a range of projections for specific hazards, such as temperature, precipitation, or sea level, obtained from different climate models driven by different emission scenarios. Some sites provide probabilistic information to accompany these projections. In the future, such sites might also be organized around a scenarios illuminating vulnerabilities concept and incorporate search engines that would allow users to specify thresholds in one or more climate parameters (for instance precipitation below some threshold and temperate above another) and obtain a summary of the projections and other scientific evidence, probabilistic and otherwise, that suggests future climate might or might not exceed such thresholds. The search portals might be organized to inform decision structuring and choice tasks in distinct steps. As one important component, this conception would require climate information providers to generate a wider range of projections than currently offered to allow users to more thoroughly explore the scientific evidence for and against potential worst cases.

In another example, the IPCC and other organizations currently provide a small number of global scenarios to coordinate research as well as to provide input for decision support. In such contexts, the scenarios aim to serve many different users considering many different policies, complicating any implementation of the scenarios illuminating vulnerabilities concept. In the near-term such exercises might nonetheless more explicitly organize the choice of scenarios around clusters of future states of the world that represent strengths and weaknesses of different generic policy approaches. Rozenberg et al. (2012) offer a step in this direction by using scenario discovery to identify five global socioeconomic scenarios with differing combinations of capacity to mitigate and to adapt to climate change. In the longer-term, scenario providers might shift from supplying a product (a fixed set of scenarios) to offering a service (information resources that enable users to create their own scenarios on demand) (Parson et al. 2007). In this latter vision, users might come to a portal and find tools and information enabling them to generate scenarios with the mix of local and global trends most important to their particular application. For instance, such a portal might help a community considering desalination for water supply to develop a scenario where such an investment yields low return, one that combines wetter-than-expected local climate with higher-than-expected global energy prices.

Offering a scenarios-on-demand service clearly presents an array of technical and conceptual challenges. The former include the design of interfaces, supporting software and analytics, and the provision of databases with a wider range of climate and socio-economic projections than now currently made available. Conceptually, some organizations charged with information provision are proscribed from discussing policies and thus may find it difficult to construct a set of proposed policies around which to organize a scenario exercise. The IPCC SRES exercise suffered this constraint. The need to equitably serve a broad audience would also complicate the design of a system that provides customized scenarios on demand. In addition, offering information services rather than a fixed set of scenarios would highlight the tensions between scenarios intended to inform scientific inquiry and those most useful for decision support. If analysts worldwide increasingly employed scenarios customized for different users, the scientific community would have fewer analyses conducted with common scenarios to study. Perhaps scientists could use statistical and other methods to draw valid comparisons across the differing cases, using a large increase in the number of analyses to compensate for a loss of simple comparability across them.

This paper makes no claim that the scenarios illuminating vulnerabilities concept would prove useful for all scenario exercises, nor is there yet a clear definition of the situations where it might prove more or less useful. The approach builds on the decision support literature and specific mechanisms that seek to overcome cognitive and organizational barriers to the effective use of scenarios. Nonetheless, many of the claims here are based on observations of the successes and failures of particular applications of these concepts. Much needs to be done to test these claims more rigorously and empirically. Nonetheless, climate-related decisions will only become more ubiquitous. The approach proposed here may make scenarios more useful in supporting many such decisions.