Abstract
Stochastic simulations applied to black-box computer experiments are becoming more widely used to evaluate the reliability of systems. Yet, the reliability evaluation or computer experiments involving many replications of simulations can take significant computational resources as simulators become more realistic. To speed up, importance sampling coupled with near-optimal sampling allocation for these experiments is recently proposed to efficiently estimate the probability associated with the stochastic system output. In this study, we establish the central limit theorem for the probability estimator from such procedure and construct an asymptotically valid confidence interval to quantify estimation uncertainty. We apply the proposed approach to a numerical example and present a case study for evaluating the structural reliability of a wind turbine.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Ankenman B, Nelson BL, Staum J (2010) Stochastic kriging for simulation metamodeling. Oper Res 58(2):371–382
Asmussen S, Binswanger K, Højgaard B (2000) Rare events simulation for heavy-tailed distributions. Bernoulli 6(2):303–322
Balesdent M, Morio J, Brevault L (2016) Rare event probability estimation in the presence of epistemic uncertainty on input probability distribution parameters. Methodol Comput Appl Probab 18(1):197–216
Blanchet J, Lam H (2012) State-dependent importance sampling for rare-event simulation: An overview and recent advances. Surv Oper Res Manag Sci 17(1):38–59
Blanchet J, Lam H (2014) Rare-event simulation for many-server queues. Math Oper Res 39(4):1142–1178
Blanchet J, Glynn P, Lam H (2009) Rare event simulation for a slotted time M/G/s model. Queueing Syst 63(1-4):33–57
Botev ZI, Kroese DP (2008) An efficient algorithm for rare-event probability estimation, combinatorial optimization, and counting. Methodol Comput Appl Probab 10(4):471–505
Byon E, Ntaimo L, Ding Y (2010) Optimal maintenance strategies for wind power systems under stochastic weather conditions. IEEE Trans Reliab 59(2):393–404
Byon E, Choe Y, Yampikulsakul N (2016) Adaptive learning in time-variant processes with application to wind power systems. IEEE Trans Autom Sci Eng 13 (2):997–1007
Cannamela C, Garnier J, Iooss B (2008) Controlled stratification for quantile estimation. Ann Appl Stat 2(4):1554–1580
Chang CS, Heidelberger P, Juneja S, Shahabuddin P (1994) Effective bandwidth and fast simulation of ATM intree networks. Perform Eval 20(1):45–65
Choe Y, Byon E, Chen N (2015) Importance sampling for reliability evaluation with stochastic simulation models. Technometrics 57(3):351–361
Choe Y, Pan Q, Byon E (2016) Computationally efficient uncertainty minimization in wind turbine extreme load assessments. ASME J Solar Energy Eng 138 (4):041,012–041,012–8
Coles SG (2001) An introduction to statistical modeling of extreme values. Springer, London
Gatto R, Baumgartner B (2016) Saddlepoint approximations to the probability of ruin in finite time for the compound Poisson risk process perturbed by diffusion. Methodol Comput Appl Probab 18(1):217–235
Geweke J (2005) Contemporary Bayesian econometrics and statistics. Wiley, Hoboken
Glasserman P, Li J (2005) Importance sampling for portfolio credit risk. Manag Sci 51(11):1643–1656
Glynn PW, Iglehart DL (1989) Importance sampling for stochastic simulations. Manag Sci 35(11):1367–1392
Heidelberger P (1995) Fast simulation of rare events in queueing and reliability models. ACM Trans Model Comput Simul (TOMACS) 5(1):43–85
Hoeffding W, Robbins H (1948) The central limit theorem for dependent random variables. Duke Math J 15(3):773–780
Hong LJ, Juneja S, Liu G (2017) Kernel smoothing for nested estimation with application to portfolio risk measurement. Oper Res 65(3):657–673
Ibragimov IA (1975) A note on the central limit theorems for dependent random variables. Theory Probab Appl 20(1):135–141
International Electrotechnical Commission (2005) IEC/TC88, 61400-1 ed. 3, Wind Turbines - Part 1: Design Requirements
Jonkman BJ (2009) TurbSim user’s guide: version 1.50. Technical Report NREL/TP-500-46198, National Renewable Energy Laboratory. Golden, Colorado
Jonkman JM, Buhl ML Jr (2005) FAST User’s Guide. Technical Report NREL/EL-500-38230, National Renewable Energy Laboratory. Golden, Colorado
Juneja S, Shahabuddin P (2006) Rare-event simulation techniques: an introduction and recent advances. Handbooks Oper Res Manag Sci 13:291–350
Kahn H, Marshall AW (1953) Methods of reducing sample size in Monte Carlo computations. J Oper Res Soc Amer 1(5):263–278
Kawai R (2008) Adaptive Monte Carlo variance reduction for Lévy processes with two-time-scale stochastic approximation. Methodol Comput Appl Probab 10(2):199–223
Koopman SJ, Shephard N, Creal D (2009) Testing the assumptions behind importance sampling. J Econ 149(1):2–11
Kroese DP, Taimre T, Botev ZI (2011) Handbook of Monte Carlo methods. Wiley, New York
Lee G, Byon E, Ntaimo L, Ding Y (2013) Bayesian spline method for assessing extreme loads on wind turbines. Ann Appl Stat 7(4):2034–2061
Manuel L, Nguyen HH, Barone MF (2013) On the use of a large database of simulated wind turbine loads to aid in assessing design standard provisions In: Proceedings of the 51st AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition. Grapevine, Texas
Moriarty P (2008) Database for validation of design load extrapolation techniques. Wind Energy 11(6):559–576
Plumlee M, Tuo R (2014) Building accurate emulators for stochastic simulations via quantile kriging. Technometrics 56(4):466–473
Rigby RA, Stasinopoulos DM (2005) Generalized additive models for location, scale and shape. J Royal Stat Soc: Ser C (Appl Stat) 54(3):507–554
Rosenblatt M (1956) A central limit theorem and a strong mixing condition. Proc Natl Acad Sci 42(1):43–47
Roussas GG (1994) Asymptotic normality of random fields of positively or negatively associated processes. J Multivar Anal 50(1):152–173
Rubinstein R (1999) The cross-entropy method for combinatorial and continuous optimization. Methodol Comput Appl Probab 1(2):127–190
Sadowsky JS (1991) Large deviations theory and efficient simulation of excessive backlogs in a GI/GI/m queue. IEEE Trans Autom Control 36(12):1383–1394
Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6(2):461–464
Soleimanzadeh M, Wisniewski R, Kanev S (2012) An optimization framework for load and power distribution in wind farms. J Wind Eng Ind Aerodyn 107-108:256–262
Staid A, Guikema SD, Nateghi R, Quiring SM, Gao MZ (2014) Simulation of tropical cyclone impacts to the US power system under climate change scenarios. Clim Chang 127(3-4):535–546
Steckley SG, Henderson SG, Ruppert D, Yang R, Apley DW, Staum J (2016) Estimating the density of a conditional expectation. Electron J Stat 10 (1):736–760
Sun Y, Apley DW, Staum J (2011) Efficient nested simulation for estimating the variance of a conditional expectation. Oper Res 59(4):998–1007
Withers CS (1981) Central limit theorems for dependent variables. I. Z Wahrscheinlichkeitstheorie verwandte Geb 57(4):509–534
Zhang N, Apley DW (2014) Fractional Brownian fields for response surface metamodeling. J Qual Technol 46(4):285
Zhang N, Apley DW (2016) Brownian integrated covariance functions for Gaussian process modeling: Sigmoidal versus localized basis functions. J Amer Stat Assoc 111 (515):1182–1195
Acknowledgements
The authors greatly appreciate an editorial board member and an anonymous reviewer for their thorough review and comments that helped improve the manuscript greatly. This work was partially supported by the National Science Foundation (Grant No. CMMI-1362513, CMMI-1523453, CMMI-1542020, CAREER CMMI-1653339, and IIS-1741166) and the University of Michigan MCubed Grant.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Choe, Y., Lam, H. & Byon, E. Uncertainty Quantification of Stochastic Simulation for Black-box Computer Experiments. Methodol Comput Appl Probab 20, 1155–1172 (2018). https://doi.org/10.1007/s11009-017-9599-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11009-017-9599-7