Abstract
This chapter discusses a Bayesian methodology to integrate model verification, validation, and calibration activities for the purpose of overall uncertainty quantification in model-based prediction. The methodology is first developed for single-level models and then extended to systems that are studied using multilevel models that interact with each other. Two types of interactions among multilevel models are considered: (1) Type-I, where the output of a lower-level model (component and/or subsystem) becomes an input to a higher-level system model, and (2) Type-II, where parameters of the system model are inferred using lower-level models and tests (that describe simplified components and/or isolated physics). The various models; their inputs, parameters, and outputs; experimental data; and various sources of model error are connected through a Bayesian network. The results of calibration, verification, and validation with respect to each individual model are integrated using the principles of conditional probability and total probability and propagated through the Bayesian network in order to quantify the overall system-level prediction uncertainty. For Type-II model, the relevance of each lower-level output to the system-level quantity of interest is quantified by comparing Sobol indices, thus measuring the extent to which a lower-level test represents the characteristics of the system so that the calibration results can be reliably used in the system level. The proposed methodology is illustrated with numerical examples that deal with heat conduction and structural dynamics.
Similar content being viewed by others
References
Urbina, A., Mahadevan, S., Paez, T.L.: Quantification of margins and uncertainties of complex systems in the presence of aleatoric and epistemic uncertainty. Reliab. Eng. Syst. Saf. 96(9), 1114–1125 (2011)
Sankararaman, S., Ling, Y., Mahadevan, S.: Uncertainty quantification and model validation of fatigue crack growth prediction. Eng. Fract. Mech. 78(7), 1487–1504 (2011)
Sankararaman, S., Mahadevan, S.: Model parameter estimation with imprecise and unpaired data. Inverse Probl. Sci. Eng. 20(7), 1017–1041 (2012)
Sankararaman, S., Mahadevan, S.: Model validation under epistemic uncertainty. Reliab. Eng. Syst. Saf. 96(9), 1232–1241 (2011)
Ling, Y., Mahadevan, S.: Quantitative model validation techniques: new insights. Reliab. Eng. Syst. Saf. 111, 217–231 (2013)
Sankararaman, S., Mahadevan, S.: Likelihood-based representation of epistemic uncertainty due to sparse point data and/or interval data. Reliab. Eng. Syst. Saf. 96(7), 814–824 (2011)
Jeffrey, H.: Theory of Probability. Oxford University Press, Oxford (1998)
Sankararaman, S., Ling, Y., Shantz, C., Mahadevan, S.: Uncertainty quantification in fatigue crack growth prognosis. Int. J. Progn. Heal. Manag. 2(1), 15 (2011)
Sankararaman, S., Ling, Y., Shantz, C., Mahadevan, S.: Inference of equivalent initial flaw size under multiple sources of uncertainty. Int. J. Fatigue 33(2), 75–89 (2011)
Sankararaman, S., Ling, Y., Mahadevan, S.: Statistical inference of equivalent initial flaw size with complicated structural geometry and multi-axial variable amplitude loading. Int. J. Fatigue 32(10), 1689–1700 (2010)
Sankararaman, S., McLemore, K., Mahadevan, S., Bradford, S.C., Peterson, L.D.: Test resource allocation in hierarchical systems using bayesian networks. AIAA J. 51(3), 537–550 (2013)
Mullins, J., Li, C., Mahadevan, S., Urbina, A.: Optimal Selection of Calibration and Validation Test Samples under Uncertainty. In: IMAC XXXII, Orlando, pp. 391–401 (2014)
Li, C., Mahadevan, S.: Sensitivity Analysis for Test Resource Allocation. In: IMAC XXXIII, Orlando (2015)
Mullins, J., Li, C., Sankararaman, S., Mahadevan, S.: Probabilistic integration of validation and calibration results for prediction level uncertainty quantification: application to structural dynamics. In: 54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Boston (2013)
Kennedy, M.C., O’Hagan, A.: Bayesian calibration of computer models. J. R. Stat. Soc. 63(3), 425–464 (2001)
Sankararaman, S., Mahadevan, S.: Comprehensive framework for integration of calibration, verification and validation. In: 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, Honolulu, pp. 1–12 (2012)
Li, C., Mahadevan, S.: Uncertainty quantification and output prediction in multi-level problems. In: 16th AIAA Non-Deterministic Approaches Conference, National Harbor (2014)
Li, C., Mahadevan, S.: Role of calibration, validation, and relevance in multi-level uncertainty integration. Reliab. Eng. Syst. Saf. 148, 32–43 (2016)
Sankararaman, S., Mahadevan, S.: Likelihood-based approach to multidisciplinary analysis under uncertainty. J. Mech. Des. 134(3), 031008 (2012)
Babuska, I., Oden, J.T.T.: Verification and validation in computational engineering and science: basic concepts. Comput. Methods Appl. Mech. Eng. 193(36–38), 4057–4066 (2004)
Roy, C.J.: Review of code and solution verification procedures for computational simulation. J. Comput. Phys. 205(1), 131–156 (2005)
AIAA: Guide for the verification and validation of computational fluid dynamics simulations. American Institute of Aeronautics and Astronautics (AIAA), no. AIAA G-077-1998 (1998)
Defense Modelling and Simulation Office, Verification, Validation, and accreditation (VV & A) recommend practices guide, Alexandia (1998)
Oberkampf, W.L., Blottner, F.G.: Issues in computational fluid dynamics code verification and validation. AIAA J 36(5), 687–695 (1998)
Oberkampf, W.L., Trucano, T.G.G.: Verification and validation in computational fluid dynamics. Prog. Aerosp. Sci. 38(3), 209–272 (2002)
Benay, R., Chanetz, B., Delery, J.: Code verification/validation with respect to experimental data banks. Aerosp. Sci. Technol. 7(4), 239–262 (2003)
Roy, C.J., Oberkampf, W.L.: A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Comput. Methods Appl. Mech. Eng. 200(25), 2131–2144 (2011)
Roache, P.J.: Verification of codes and calculations. Aiaa J. 36(5), 696–702 (1998)
Roache, P.J.: Verification and Validation in Computational Science and Engineering. Hermosa Publishers, Albuquerque (1998)
Oberkampf, W.L., Trucano, T.G., Hirsch, C.: Verification, validation, and predictive capability in computational engineering and physics. Appl. Mech. Rev. 57(5), 345–384 (2004)
Roy, C.J., McWherter-Payne, M.A., Oberkampf, W.L.: Verification and validation for laminar hypersonic flowfields, part 1: verification. Aiaa J. 41(10), 1934–1943 (2003)
Rebba, R., Mahadevan, S., Huang, S.: Validation and error estimation of computational models. Reliab. Eng. Syst. Saf. 91(10–11), 1390–1397 (2006)
Liang, B., Mahadevan, S.: Error and uncertainty quantification and sensitivity analysis in mechanics computational models. Int. J. Uncertain. Quantif. 1(2), 147–161 (2011)
Rangavajhala, S., Sura, V.S., Hombal, V.K., Mahadevan, S.: Discretization error estimation in multidisciplinary simulations. AIAA J. 49(12), 2673–2683 (2011)
Ferziger, J., Peric, M.: Computational Methods for Fluid Dynamics. Springer, Berlin (1996)
Ainsworth, M., Oden, J.T.T.: A posteriori error estimation in finite element analysis. Comput. Methods Appl. Mech. Eng. 142(1–2), 1–88 (1997)
Oberkampf, W.L., DeLand, S.M., Rutherford, B.M., Diegert, K.V., Alvin, K.F.: Error and uncertainty in modeling and simulation. Reliab. Eng. Syst. Saf. 75(3), 333–357 (2002)
Haldar, A., Mahadevan, S.: Probability, Reliability, and Statistical Methods in Engineering Design. John Wiley, New York (2000)
Ghanem, R., Spanos, P.D.: Polynomial chaos in stochastic finite elements. J. Appl. Mech. 57(1)(89), 197–202 (1990)
Buhmann, M.D.: Radial Basis Functions: Theory and Implementations, vol. 12. Cambridge university press, Cambridge/New York (2003)
Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. MIT, Cambridge (2006)
Richards, S.A.: Completed Richardson extrapolation in space and time. Commun. Numer. Methods Eng. 13(7), 573–582 (1997)
Xu, P., Su, X., Mahadevan, S., Li, C., Deng, Y.: A non-parametric method to determine basic probability assignment for classification problems. Appl. Intell. 41(3), 681–693 (2014)
Babuska, I., Rheinboldt, W.C.: A posteriori error estimates for the finite element method. Int. J. Numer. Methods Eng. 12(10), 1597–1615 (1978)
Demkowicz, L., Oden, J.T., Strouboulis, T.: Adaptive finite elements for flow problems with moving boundaries. part I: variational principles and a posteriori estimates. Comput. Methods Appl. Mech. Eng. 46(2), 217–251 (1984)
Rasmussen, C.E.: Evaluation of Gaussian processes and other methods for non-linear regression. PhD dissertation, University of Toronto, 1996
Rasmussen, C.E.: The infinite Gaussian mixture model. In: NIPS, Denver, vol. 12, pp. 554–560 (1999)
Rasmussen, C.E.: Gaussian processes in machine learning. In: Bousquet, O., VonLuxburg, U., Ratsch, G. (eds.) Advanced Lectures on Machine Learning, vol. 3176, pp. 63–71 (2004)
Santner, T.J., Williams, B.J., Notz, W.I.: The design and analysis of computer experiments. Springer, Dordrecht/New York (2013)
Bichon, B.J., Eldred, M.S., Swiler, L.P., Mahadevan, S., McFarland, J.M.: Efficient global reliability analysis for nonlinear implicit performance functions. Aiaa J. 46(10), 2459–2468 (2008)
McFarland, J.M.: Uncertainty Analysis for Computer Simulations throuth Validation and Calibraion. Vanderbilt University, Nashville (2008)
Cressie, N.: Spatial Statistics. John Wiley, New York (1991)
Chiles, J.-P., Delfiner, P.: Geostatistics: Modeling Spatial Uncertainty, vol. 344. Wiley-Interscience, New York (1999)
Wackernagel, H.: Multivariate Geostatistics: An Introduction with Applications. Springer, Berlin/New York (2003)
Trucano, T.G., Swiler, L.P., Igusa, T., Oberkampf, W.L., Pilch, M.: Calibration, validation, and sensitivity analysis: what’s what. Reliab. Eng. Syst. Saf. 91(10–11), 1331–1357 (2006)
Seber, G.A.F., Wild, C.J.: Nonlinear Regression. Wiley, New York (1989)
Edwards, A.W.F.: Likelihood. Cambridge University Press, Cambridge, UK (1972)
Pawitan, Y.: In all likelihood: statistical modelling and inference using likelihood. Oxford University Press, Oxford/New York (2001)
Leonard, T., Hsu, J.: Bayesian Methods. Cambridge University Press, Cambridge (2001)
Lee, P.: Bayesian Statistics, 3rd edn. Arnold, London (2004)
Malinverno, A., Briggs, V.A.: Expanded uncertainty quantification in inverse problems: hierarchical Bayes and empirical Bayes. Geophysics 69(4), 1005–1016 (2004)
Park, I., Amarchinta, H.K., Grandhi, R.V.: A Bayesian approach for quantification of model uncertainty. Reliab. Eng. Syst. Saf. 95(7), 777–785 (2010)
Oliver, T.A., Moser, R.D.: Accounting for uncertainty in the analysis of overlap layer mean velocity models. Phys. Fluids 24(7), 075108 (2012)
Arendt, P.D., Apley, D.W., Chen, W.: Quantification of model uncertainty: calibration, model discrepancy, and identifiability. J. Mech. Des. 134(10), 100908 (2012)
Ling, Y., Mullins, J.G., Mahadevan, S.: Options for the inclusion of model discrepancy in Bayesian calibration. In: 16th AIAA Non-Deterministic Approaches Conference, National Harbor. American Institute of Aeronautics and Astronautics (2014)
Liu, F., Bayarri, M.J., Berger, J.O.: Modularization in Bayesian analysis, with emphasis on analysis of computer models. Bayesian Anal. 4(1), 119–150 (2009)
Sankararaman, S.: Uncertainty Quantification and Integration in Engineering Systems. Vanderbilt University, Nashville (2012)
Gilks, W.R., Richardson, S., Spiegelhalter, D.J.: Markov chain Monte Carlo in practice. Chapman and Hall, London (1996)
Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21(6), 1087–1092 (1953)
Gilks, W.R., Wild, P.: Adaptive rejection sampling for Gibbs sampling. Appl. Stat. 41(2), 337–348 (1992)
Neal, R.M.: Slice sampling. Ann. Stat. 31(3), 705–767 (2003)
American Society of Mechanical Engineers: Guide for Verification and Validation in Computational Solid Mechanics, p. 53. American Society of Mechanical Engineers, New York (2006)
Coleman, H.W., Stern, F.: Uncertainties and CFD code validation. J. Fluids Eng. Asme 119(4), 795–803 (1997)
Oberkampf, W.L., Barone, M.F.: Measures of agreement between computation and experiment: validation metrics. J. Comput. Phys. 217(1), 5–36 (2006)
Ferson, S., Oberkampf, W.L., Ginzburg, L.: Model validation and predictive capability for the thermal challenge problem. Comput. Methods Appl. Mech. Eng. 197(29–32), 2408–2430 (2008)
Hills, R.G., Leslie, I.H.: Statistical validation of engineering and scientific models: validation experiments to application. Sandia National Labs., Albuquerque/Livermore (2003).
Urbina, A., Paez, T.L., Hasselman, T.K., Wathugala, G.W., Yap, K.: Assessment of model accuracy relative to stochastic system behavior. In: Proceedings of 44th AIAA Structures, Structural Dynamics, Materials Conference, Norfolk, pp. 7–10 (2003)
Gelfand, A.E., Dey, D.K.: Bayesian model choice: asymptotics and exact calculations. J. R. Stat. Soc. Ser. B-Methodol. 56(3), 501–514 (1994)
Geweke, J.: Bayesian model comparison and validation. Am. Econ. Rev. 97(2), 60–64 (2007)
Zhang, R.X., Mahadevan, S.: Bayesian methodology for reliability model acceptance. Reliab. Eng. Syst. Saf. 80(1), 95–103 (2003)
Mahadevan, S., Rebba, R.: Validation of reliability computational models using Bayes networks. Reliab. Eng. Syst. Saf. 87(2), 223–232 (2005)
Rebba, R., Mahadevan, S.: Computational methods for model reliability assessment. Reliab. Eng. Syst. Saf. 93(8), 1197–1207 (2008)
Sankararaman, S., Mahadevan, S.: Assessing the reliability of computational models under uncertainty. In: 54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Boston, pp. 1–8 (2013)
Thacker, B.H., Paez, T.L.: A simple probabilistic validation metric for the comparison of uncertain model and test results. In: ASME Verification and Validation Symposium, Las Vegas (2013)
Liu, Y., Chen, W., Arendt, P., Huang, H.-Z.: Toward a better understanding of model validation metrics. J. Mech. Des. 133(7), 071005 (2011)
Roache, P.J.: Fundamentals of Verification and Validation. Hermosa Press, Socorro (2009)
Oberkampf, W.L., Roy, C.C.J.: Verification and Validation in Scientific Computing. Cambridge University Press, New York (2010)
O’Hagan, A.: Fractional Bayes Factors for Model Comparison. J. R. Stat. Soc. 57(1), 99–138 (1995)
Jiang, X., Mahadevan, S.: Bayesian risk-based decision method for model validation under uncertainty. Reliab. Eng. Syst. Saf. 92(6), 707–718 (2007)
Cha, S.: Comprehensive survey on distance/similarity measures between probability density functions. Int. J. Math. Model. METHODS Appl. Sci. 1(4) (2007)
De Maesschalck, R., Jouan-Rimbaud, D., Massart, D.L.: The Mahalanobis distance. Chemom. Intell. Lab. Syst. 50(1), 1–18 (2000)
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S.: Global Sensitivity Analysis: The Primer. John Wiley, Chichester (2008)
Sobol’, I.M.: Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55(1–3), 271–280 (2001)
Li, C., Mahadevan, S.: Global sensitivity analysis for system response prediction using auxiliary variable method. In: 17th AIAA Non-Deterministic Approaches Conference, Kissimmee (2015)
Li, C., Mahadevan, S.: Relative contributions of aleatory and epistemic uncertainty sources in time series prediction. Int. J. Fatigue 82, 474–486 (2016)
Singhal, A.: Modern information retrieval: a brief overview. IEEE Data Eng. Bull. 24(4), 35–43 (2001)
Van Horn, K.S.: Constructing a logic of plausible inference: a guide to Cox’s theorem. Int. J. Approx. Reason. 34(1), 3–24 (2003)
Sankararaman, S., Mahadevan, S.: Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliab. Eng. Syst. Saf. 138, 194–209 (2015)
Li, C., Mahadevan, S.: Uncertainty quantification and integration in multi-level problems. In: IMAC XXXII, Orlando, vol. 3 (2014)
Rosenblatt, M.: Remarks on some nonparametric estimates of a density function. Ann. Math. Stat. 27(3), 832–837 (1956)
Red-Horse, J.R.R., Paez, T.L.L.: Sandia National Laboratories validation workshop: Structural dynamics application. Comput. Methods Appl. Mech. Eng. 197(29–32), 2578–2584 (2008)
Chopra, A.K.: Dynamics of Structures: Theory and Applications to Earthquake Engineering, 4th edn. Prentice Hall, Englewood Cliffs (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this entry
Cite this entry
Mahadevan, S., Sankararaman, S., Li, C. (2016). Multilevel Uncertainty Integration. In: Ghanem, R., Higdon, D., Owhadi, H. (eds) Handbook of Uncertainty Quantification. Springer, Cham. https://doi.org/10.1007/978-3-319-11259-6_8-1
Download citation
DOI: https://doi.org/10.1007/978-3-319-11259-6_8-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Online ISBN: 978-3-319-11259-6
eBook Packages: Springer Reference MathematicsReference Module Computer Science and Engineering