Abstract
Expert judgement is an unavoidable element of the process-based numerical models used for climate change projections, and the statistical approaches used to characterize uncertainty across model ensembles. Here, we highlight the need for formalized approaches to unifying numerical modelling with expert judgement in order to facilitate characterization of uncertainty in a reproducible, consistent and transparent fashion. As an example, we use probabilistic inversion, a well-established technique used in many other applications outside of climate change, to fuse two recent analyses of twenty-first century Antarctic ice loss. Probabilistic inversion is but one of many possible approaches to formalizing the role of expert judgement, and the Antarctic ice sheet is only one possible climate-related application. We recommend indicators or signposts that characterize successful science-based uncertainty quantification.
Similar content being viewed by others
References
The Economic and Financial Risks of a Changing Climate: Insights from Leading Experts Workshop Report (AAAS, 2014).
Draper, D. Assessment and propagation of model uncertainty. J. R. Statis. Soc. B 57, 45–97 (1995).
Oppenheimer, M. et al. in Climate Change: Impacts, Adaptation, and Vulnerability. (eds Field, C. B. et al.) 1039–1099 (IPCC, Cambridge Univ. Press, 2014).
Frigg, R., Smith, L. A. & Stainforth, D. A. The myopia of imperfect climate models: the case of UKCP09. Philos. Sci. 80, 886–897 (2013).
Knutti, R. & Hegerl, G. C. The equilibrium sensitivity of the Earth's temperature to radiation changes. Nature Geosci. 1, 735–743 (2008).
Morgan, M. G. & Keith, D. W. Subjective judgements by climate experts. Environ. Sci. Technol. 29, 468–476 (1995).
Zickfeld, K., Morgan, M. G., Frame, D. J. & Keith, D. W. Expert judgements about transient climate response to alternative future trajectories of radiative forcing. Proc. Natl Acad. Sci. USA 107, 12451–12456 (2010).
Church, J. A. et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) (IPCC, Cambridge Univ. Press, Cambridge, 2013).
Morgan, G. M. et al. Best Practice Approaches for Characterizing, Communicating, and incorporating Scientific Uncertainty in Climate Decisions Synthesis and Assessment Product 5.2. (US Climate Change Science Program, 2009).
Mastrandrea, M. D. et al. Guidance note for lead authors of the IPCC Fifth Assessment Report on consistent treatment of uncertainties (IPCC, 2010).
Knutti, R. et al. in Meeting Report of the Intergovernmental Panel on Climate Change: Expert Meeting on Assessing and Combining Multi Model Climate (eds Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M. & Midgley, P. M.) 1–13 (IPCC, 2010).
Moss, R. H. & Schneider, S. H. in Guidance papers on the cross cutting issues of the Third Assessment Report of the IPCC (eds R. Pachauri, T. Taniguchi and K. Tanaka) 33–51 (IPCC, 2000).
Collins . et al. in Climate Change 2013: The Physical Science Basis (eds Stocker, T. F. et al.) Ch. 13 (Cambridge Univ. press, 2013).
Goossens, L. H. J. & Kelly, G. N. Radiation protection dosimetry expert judgement and accident consequence. J. Uncertainty Anal. 90, 295–301 (2000).
Mason, D. & Knutti, R. Predictor screening, calibration, and observational constraints in climate model ensembles: an illustration using climate sensitivity. J. Clim. 26, 887–898 (2013).
Murphy, J. M. et al. A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Phil. Trans. Royal Soc. A 365, 1993–2028 (2007).
Knutti, R., Furrer, R., Tebaldi, C., Cermak, J. & Meehl, G. A. Challenges in combining projections from multiple climate models. J. Clim. 23, 2739–2758 (2010).
Tebaldi, C. & Knutti, R. The use of the multi-model ensemble in probabilistic climate projections. Phil. Trans. R. Soc. A 365, 2053–2075 (2007).
Rougier, J. C., Goldstein, M. & House, L. Second-order exchangeability analysis for multi-model ensembles. J. Am. Statist. Assoc. 108, 852–863 (2013).
Goldstein, M. & Rougier, J. Reified Bayesian modelling and inference for physical systems. J. Statist. Plan. Infer. 139, 1221–1239 (2009).
Murphy, J. M. et al. UK Climate Projections Science Report: Climate Change Projections (Met Office, 2009).
Sexton, D. M. H., Murphy, J. M., Collins, M. & Webb, M. J. Multivariate probabilistic projections using imperfect climate models part I: outline of methodology Clim. Dynam. 38, 2513–2542 (2012).
Sexton, D. M. H. & Murphy, J. M. Multivariate probabilistic projections using imperfect climate models part II: robustness of methodological choices and consequences for climate sensitivity Clim. Dynam. 38, 2543–2558 (2012).
Parker, W. S. Ensemble modeling, uncertainty and robust predictions. WIREs Clim. Change 4, 213–223 (2013).
O'Reilly, J., Oreskes, N. & Oppenheimer, M. The rapid disintegration of predictions: climate science, bureaucratic institutions, and the West Antarctic ice sheet. Social Stud. Sci. 42, 709–731 (2012).
van der Sluijs, J. et al. Anchoring devices in science for policy: the case of consensus around climate sensitivity. Social Stud. Sci. 28, 291–323 (1998).
Houser, T. et al. American Climate Prospectus: Economic Risks in the United States (Rhodium Group, 2014).
Hinkel, J. et al. Coastal flood damage and adaptation costs under 21st century sea-level rise. Proc. Natl Acad. Sci. USA 111, 3292–3297 (2014).
Pattyn, F. et al. Grounding-line migration in plan-view marine ice-sheet models: results of the ice2sea MISMIP3d intercomparison. J. Glaciol. 59, 410–422 (2013).
Durand, G. & Pattyn, F. Reducing uncertainties in projections of Antarctic ice mass loss. Cryos. Discuss. 9, 2625–2654 (2015).
Pollard, D., DeConto, R. M. & Alley, R. B. Potential Antarctic ice sheet retreat driven by hydrofracturing and ice cliff failure. Earth Planet. Sci. Lett. 412, 112–121 (2015).
Bindschadler, R. A. et al. Ice-sheet model sensitivities to environmental forcing and their use in projecting future sea level (the SeaRISE project). J. Glaciol. 59, 195–224 (2013).
Nowicki, S. et al. Insights into spatial sensitivities of ice mass response to environmental change from the searise ice sheet modeling project ii: Greenland. J. Geophys. Res. Earth Surf. 118, 1025–1044 (2013).
Vaughan, D. & Spouge, J. Risk estimation of collapse of the West Antarctic ice sheet. Climatic Change 52, 65–91 (2002).
Rahmstorf, S., Perrette, M. & Vermeer, M. Testing the robustness of semi-empirical sea level projections. Clim. Dynam. 39, 861–875 (2012).
Oppenheimer, M., O'Neill, B. & Webster, M. Negative learning. Climatic Change 89, 155–172 (2008).
Oppenheimer, M. et al. The limits of consensus. Science 317, 1505–1506 (2007).
Bamber, J. L. & Aspinall, W. P. An expert judgement assessment of future sea. level rise from the ice sheets. Nature Clim. Change 3, 424–427 (2013).
Little, C. M., Oppenheimer, M. & Urban, N. M. Upper bounds on twenty-first-century Antarctic ice loss assessed using a probabilistic framework, Nature Clim. Change 3, 654–659 (2013).
Kopp, R. E. et al. Probabilistic 21st and 22nd century sea-level projections at a global network of tide gauge sites. Earth's Future 2, 383–406 (2014).
Jevrejeva, S., Grinsted, A. & Moore, J. C. Upper limit for sea level projections by 2100. Environ. Res. Lett. 9, 104008 (2014).
Kraan, B. C. P. & Bedford. T. J. Probabilistic inversion of expert judgements in the quantification of model uncertainty. Manag. Sci. 51, 995–1006 (2005).
Du, C., Kurowicka, D. & Cooke, R. M. Techniques for generic probabilistic inversion, Comp. Stat. Data Anal. 50, 1164–1187 (2006).
Joughin I., Smith, B. & Medley, B. Marine ice sheet collapse potentially under way for Thwaites Glacier Basin, West Antarctica. Science 344, 735–738 (2014).
Rignot, E., Mouginot, J., Morlighem, M., Seroussi, H. & Scheuchl, B. Widespread, rapid grounding line retreat of Pine Island, Thwaites, Smith, and Kohler glaciers, West Antarctica, from 1992 to 2011. Geophys. Res. Lett. 41, 3502–3509 (2014).
de Vries, H. & van de Wal, R. S. W. How to interpret expert judgement assessments of 21st century sea-level rise. Climatic Change 130, 87–100 (2015).
Horton, B. P., Rahmstorf, S., Engelhart, S. E. & Kemp, A. C. Expert assessment of sea-level rise by AD 2100 and AD 2300. Quatern. Sci. Rev. 84, 1–6 (2014).
Global sea level rise scenarios for the United States National Climate Assessment (Climate Program Office, 2012).
Little, C. M., Urban, N. M. & Oppenheimer, M. Probabilistic framework for assessing the ice sheet contribution to sea level change. Proc. Natl Acad. Sci. USA 110, 3264–3269 (2013).
Jones, J. A. et al. Probabilistic Accident Consequence Uncertainty Assessment Using COSYMA Uncertainty from the Atmospheric Dispersion and Deposition Module EUR 18822EN (European Commission, 2001).
Popper, K. R. The Logic of Scientific Discovery (Hutchinson, 1959).
Lakatos, I. The Methodology of Scientific Research Programmes Philos. Papers Vol. 1 (Cambridge Univ. Press, 1978).
Oreskes, N., Shrader-Frechete, K. & Belitz, K. Verification, validation and confirmation of numerical models in the Earth sciences. Science 26, 641–646 (1994).
Cooke, R. M. Uncertainty in dispersion and deposition in accident consequence modelling assessed with performance-based expert judgement. Rel. Eng. Syst. Saf. 45, 35–46 (1994).
Aspinall, W. P. & Cooke, R. M. in Risk and Uncertainty Assessment in Natural Hazards (eds Hill, L., Rougier, J. C. & Sparks R. S. J.) 64–99 (Cambridge University Press, 2013).
Rasmussen, N. C. et al. Reactor Safety Study: An Assessment of Accident Risks in US Commercial Nuclear Power Plants WASH-1400 (NUREG75/014) (US Nuclear Regulatory Commission, 1975).
Lewis, H. et al. Risk Assessment Review Group Report to the US Nuclear Regulatory Commission NUREG/CR-04000 (Chemical Rubber Company, 1979).
Science and Judgement in Risk Assessment (The National Academies, 1994).
Guidelines for Carcinogen risk assessment EPA/630/P-03/001F (US Environmental Protection Agency, 2005).
Expert Elicitation Task Force White Paper (US Environmental Protection Agency, 2011).
Cooke, R. M. & Goossens, L. H. J. TU Delft Expert judgment data base, special issue on expert judgement. Rel. Eng. Syst. Saf. 93, 657–674 (2008).
Aspinall, W. P. A route to more tractable expert advice. Nature 463, 294–95 (2010).
Eggstaff, J. W., Mazzuchi, T. A. & Sarkani, S. The effect of the number of seed variables on the performance of Cooke's classical model. Rel. Eng. Syst. Saf. 121, 72–82, (2014).
Aspinall, W. P. in Statistics in Volcanology. (eds Mader, H. M., Coles, S. G., Connor, C. B. & Connor, L. J.) 15–30 (Geological Society, 2006).
Cooke, R. M. et al. A probabilistic characterization of the relationship between fine particulate matter and mortality: elicitation of European experts. Environ. Sci. Technol. 41, 6598–6605 (2007).
Tuomisto, J. T., Wilson, A., Cooke, R. M., Tainio, M. & Evans J. S. Mortality in Kuwait due to PM from oil fires after the Gulf War: combining expert elicitation assessments. Epidemiol. 16, S74–S75 (2005).
Evans J. S., Wilson A., Tuomisto J. T., Tainio M. & Cooke R. M. What risk assessment can tell us about the mortality impacts of the Kuwaiti oil fires. Epidemiol. 16, S137–S138 (2005).
Cooke, R. M. et al. Out-of-sample validation for structured expert judgement of Asian carp establishment in Lake Erie. Integr. Environ. Assess. Manag. 10, 522–528 (2014).
Hoffmann, S. et al. Research synthesis methods in an age of globalized risks: lessons from the global burden of foodborne disease expert elicitation. Risk Analysis 36, 191–202 (2015).
Koch, B. J. et al. Suburban watershed nitrogen retention: estimating the effectiveness of storm water management structures. Elementa: Sci. Anthropocene 3, 1–18 (2015).
Wittmann, M. E., Cooke, R. M., Rothlisberger, J. D. & Lodge, D. M. Using structured expert judgement to assess invasive species prevention: Asian carp and the Mississippi — Great Lakes hydrologic connection. Environ. Sci. Technol. 48, 2150–2156 (2014).
Wittmann, M. E. et al. Structured expert judgement to forecast species invasions: bighead and silver carp in Lake Erie. Cons. Biol. 29, 187–197 (2014).
Tyshenko, M. G. et al. 2010 expert elicitation for the judgement of prion disease risk uncertainties using the classical model and EXCALIBUR J. Toxicol. Environ. Health A. 74, 261–285 (2011).
Gerstenberger, M. C., McVerry, G. H., Rhoades, D. A. & Stirling, M. W. Seismic hazard modeling for the recovery of Christchurch, New Zealand. Earthquake Spectra 30, 17–29 (2014).
Christophersen, A., Nicol, A. & Gerstenberger, M. C. The Feasibility of Using Seed Questions for Weighting Expert Opinion in CCS Risk Assessment CO2CRC Report RPT11–2868 (Cooperative Research Centre for Greenhouse Gas Technologies, 2011).
Gerstenberger, M. C. et al. in 11th International Conference on Greenhouse Gas Control Technologies (eds Dixon, T. & Yamaji, K.) 2775–2782 (Elsevier, 2013).
Acknowledgements
We thank J. Hall (Oxford University, UK), K. Keller (Pennsylvania State University, USA), R. Kopp (Rutgers University, USA), J. Rougier (University of Bristol, UK), D. Sexton (Met Office Hadley Centre, UK) and C. Tebaldi (National Center for Atmospheric Research, USA) for either helpful comments on an earlier draft of the manuscript, or useful discussions of the issues raised, or both.
Author information
Authors and Affiliations
Contributions
M.O., C.M.L. and R.M.C. designed the research, conducted analysis of data and results, and contributed to writing, editing and revision. C.M.L. and R.M.C. performed statistical modelling.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Information
Supplementary Information (PDF 607 kb)
Supplementary Information
Supplementary Information (XLS 88 kb)
Rights and permissions
About this article
Cite this article
Oppenheimer, M., Little, C. & Cooke, R. Expert judgement and uncertainty quantification for climate change. Nature Clim Change 6, 445–451 (2016). https://doi.org/10.1038/nclimate2959
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nclimate2959
- Springer Nature Limited
This article is cited by
-
A climate risk index for marine life
Nature Climate Change (2022)
-
Evaluating process-based integrated assessment models of climate change mitigation
Climatic Change (2021)
-
Burning embers: towards more transparent and robust climate-change risk assessments
Nature Reviews Earth & Environment (2020)
-
Implications of ambiguity in Antarctic ice sheet dynamics for future coastal erosion estimates: a probabilistic assessment
Climatic Change (2020)
-
Wisdom of the experts: Using survey responses to address positive and normative uncertainties in climate-economic models
Climatic Change (2020)