Bayesian Methodology for Uncertainty Quantification in Complex Engineering Systems

Abstract

This chapter presents a Bayesian methodology for system-level uncertainty quantification and test resource allocation in complex engineering systems. The various component, subsystem, and system-level models, the corresponding parameters, and the model errors are connected efficiently using a Bayesian network. This provides a unified framework for uncertainty analysis where test data can be integrated along with computational models across the entire hierarchy of the overall engineering system. The Bayesian network is useful in two ways: (1) in a forward problem where the various sources of uncertainty are propagated through multiple levels of modeling to predict the overall uncertainty in the system-level response; and (2) in an inverse problem where the model parameters of multiple subsystems are calibrated simultaneously using test data. Test data available at multiple data are first used to infer model parameters, and then, this information is propagated through the Bayesian network to compute the overall uncertainty in the system-level prediction. Then, the Bayesian network is used for test resource allocation where an optimization-based procedure is used to identify tests that can effectively reduce the overall uncertainty in the system-level prediction. Finally, the overall Bayesian methodology for uncertainty quantification and test resource allocation is illustrated using three different numerical examples. While the first example is mathematical, the second and third examples deal with practical applications in the domain of structural mechanics.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

During the past 20 years, there has been an increasing need and desire to design and build engineering systems with increasingly complex architectures and new materials. These systems can be multi-level, multi-scale, and multi-disciplinary in nature, and may need to be decomposed into simpler components and subsystems to facilitate efficient model development, analysis, and design. The development and implementation of computational models is not only sophisticated and expensive, but also based on physics which is often not well understood. Therefore, when such models are used to design and analyze complex engineering systems, it is necessary to ensure their reliability and safety.

In order to facilitate efficient analysis and design, computational models are developed at the component-level, subsystem-level, and system-level. Each individual model may correspond to isolated features, or isolated physics, or simplified geometry of the original system. Typically, along the hierarchy of a multi-level system, the complexity of the governing physics increases, and hence, the complexity of the model increases, the cost of testing increases, and hence, the amount of available experimental data decreases. At the system-level, full-scale testing may not even be possible to predict the system performance under actual operating conditions. It is essential to quantify the overall uncertainty in the system-level prediction using the models and data available at all levels. The field of “quantification of margins and uncertainties” (QMU) has the goal of enabling this overall capability (Helton and Pilch 2011). This analysis is helpful to estimate the reliability and adjudge the safety of the overall engineering system.

An important challenge in this regard is to efficiently connect all of the models and experimental data available across the hierarchy of the entire system. This is not straightforward because there are several sources of uncertainty—physical variability, data uncertainty, and model uncertainty—at each level of the overall system. Further, the issue is complicated by the presence of individual model inputs, parameters, outputs, and model errors, all of which may be uncertain. It is important to use a computational approach that can not only facilitate integration across multiple levels but also provide a fundamental framework for the treatment of uncertainty.

This can be accomplished through the use of a Bayesian network that serves as an efficient and powerful tool to integrate multiple levels of models (including inputs, parameters, outputs, and errors of each and every model), the associated sources of uncertainty, and the experimental data at all different levels of hierarchy. The Bayesian network is based on Bayes’ theorem, and can efficiently integrate all of the aforementioned information using the principles of conditional probability and total probability. It can be used for uncertainty propagation (forward problem), where the system-level prediction uncertainty is quantified by propagating all the sources of uncertainty at lower levels through the Bayesian network. The Bayesian network is also useful for model parameter calibration (inverse problem), where the data available at all levels can be simultaneously used to calibrate the underlying parameters at different levels of models. Different types of sampling techniques can be used in conjunction with a Bayesian network; while Monte Carlo-sampling-based approaches are often used for uncertainty propagation (forward problem), Markov Chain Monte Carlo-based approaches are often used for model parameter estimation (inverse problem). Since sampling methods require several hundreds of thousands of evaluations of the system models that may be complicated to evaluate, the Bayesian approach can also include cost-efficient surrogate models like Gaussian process, as demonstrated by Kennedy and O’Hagan (2001). This chapter explains the fundamentals of Bayesian methodology and illustrates the use of Bayesian networks for uncertainty quantification in complex multi-level, multi-physics engineering systems. Methods for uncertainty quantification, model parameter estimation, and system-level uncertainty quantification are presented in detail.

Finally, it is also illustrated as to how these capabilities of Bayesian networks can be exploited to guide test resource allocation in hierarchical systems (Sankararaman et al. 2013). Test data available at multiple levels of system hierarchy are used for model parameter calibration, which in turn leads to a reduction of uncertainty in the model parameters; this reduced uncertainty is represented through the posterior distributions of the model parameters. When these posterior distributions are propagated through the Bayesian network, the uncertainty in the system-level response also decreases. Thus, testing can be used to evolve the system performance prediction and a common concern is to select that test design which leads to maximum reduction in the uncertainty (usually expressed through variance) of the system performance prediction. The tests need to be selected and designed with adequate precision (measurement error and resolution), and the simulations need to be developed with adequate resolution (model fidelity) to achieve the project requirements. This can be performed by embedding the Bayesian network within an optimization algorithm where the decision variables correspond to the test data. This formulation is very interesting because, in the past, model parameters have been calibrated with available test data; the difference now is that it is required to perform Bayesian calibration and assess the reduction in uncertainty in the system-level response even before actual testing is performed.

Two types of questions are of interest: (1) what tests to do? and (2) how many tests to do? Tests at different levels of the system hierarchy have different costs and variance reduction effects. Hence, the test selection is not trivial and it is necessary to identify an analytical procedure that helps in the optimum test resource allocation. However, current practice for this is, at best, based on simplified analysis and relevant experience, and at worst based on ad hoc rules, any of which may or may not result in truly conservative estimates of the margin and uncertainty. For multi-level systems, a rational test selection procedure should also incorporate the information from component-level and subsystem-level tests towards overall system level performance prediction. Recently, Sankararaman et al. (2013) developed an optimization-based methodology to identify the tests that will lead to maximum reduction in the system-level uncertainty, while simultaneously minimizing the cost of testing. This methodology is presented in detail, towards the end of this chapter, and illustrated using numerical examples.

2 Fundamentals of Bayesian Methodology

The Bayesian methodology is based on subjective probabilities, which are simply considered to be degrees of belief and quantify the extent to which the “statement” is supported by existing knowledge and available evidence. This may be contrasted with the frequentist approach to probability (classical approach to statistics), according to which probabilities can be assigned only in the context random physical systems and experiments. Uncertainty arising out of the model parameter estimation procedure is expressed in terms of confidence intervals, and it is not statistically meaningful to assign probability distributions to estimation parameters, since they are assumed to be “deterministic but unknown” in the frequentist methodology. This is a serious limitation, since it is not possible to propagate uncertainty after parameter estimation, which is often necessary in the case of model-based quantification of uncertainty in the system-level response. For example, if the uncertainty in the elastic modulus had been estimated using a simple axial test, this uncertainty cannot be used for quantifying the response in a plate made of the same material. Another disadvantage of this approach is that, when a quantity is not random, but unknown, then the tools of probability cannot be used to represent this type of uncertainty (epistemic). The subjective interpretation of probability, on which the Bayesian methodology relies upon, overcomes both of these limitations.

In the Bayesian approach, even deterministic quantities can be represented using probability distributions which reflect the subjective degree of the analyst’s belief regarding such quantities. As a result, probability distributions can even be assigned to parameters that need to be estimated, and therefore, this interpretation facilitates uncertainty propagation after parameter estimation; this aspect of the Bayesian methodology is helpful for uncertainty integration across multiple models after inferring the underlying model parameters.

For example, consider the case where a variable is assumed to be normally distributed and it is desired to estimate the mean and the standard deviation based on available point data. If sufficient data were available, then it is possible to uniquely estimate these distribution parameters. However, in some cases, data may be sparse and therefore, it may be necessary to quantify the uncertainty in these distribution parameters. Note that this uncertainty is an example of epistemic uncertainty; the quantities may be estimated deterministically with enough data. Though these parameters are actually deterministic, the Bayesian methodology can calculate probability distributions for the distribution parameters, which can be easily used in uncertainty propagation. The fundamentals of Bayesian philosophy are well established in several textbooks (Calvetti and Somersalo 2007; Lee 2004; Leonard and Hsu 2001; Somersalo and Kaipio 2004), and the Bayesian approach is being increasingly applied to engineering problems in recent times, especially to solve statistical inverse problems. This section provides an overview of the fundamentals of the Bayesian approach, and later sections illustrate the application of Bayesian methods to uncertainty quantification in complex engineering systems.

2.1 Bayes Theorem

Though named after the eighteenth century mathematician and theologian Thomas Bayes (Bayes and Price 1763), it was the French mathematician Pierre-Simon Laplace who pioneered and popularized what is now called Bayesian probability (Stigler 1986, 2002). For a brief history of Bayesian methods, refer to Fienberg (2006). The law of conditional probability is fundamental to the development of Bayesian philosophy:

$$\displaystyle{ P(AB) = P(A\vert B)P(B) = P(B\vert A)P(A) }$$

(1)

Consider a list of mutually exclusive and exhaustive events A _i (i = 1 to n) that together form the sample space. Let B denote any other event from the sample space such that P(B) > 0. Based on Eq. (1), it follows that:

$$\displaystyle{ P(A_{i}\vert B) = \frac{P(B\vert A_{i})P(A_{i})} {\mathop{\sum }\limits_{j}P(B\vert A_{j})P(A_{j})} }$$

(2)

What does Eq. (2) mean? Suppose that the probabilities of events A _i (i = 1 to n) are known to be equal to P(A _i ) (i = 1 to n) before conducting any random experiments. These probabilities are referred to as prior probabilities in the Bayesian context. Suppose that a random experiment has been conducted and event B has been observed. In the light of this data, the so-called posterior probabilities P(A _i  | B) (i = 1 to n) can be calculated using Eq. (2).

The quantity P(B | A _i ) is the probability of observing the data conditioned on A _i . It can be argued that event B has “actually been observed,” and there is no uncertainty regarding its occurrence, which renders the probability P(B | A _i ) meaningless. Hence, researchers “invented” new terminology in order to denote this quantity. In earlier days, this quantity was referred to as “inverse probability,” and since the advent of Fisher (Aldrich 2008; Jeffreys 1998) and Edwards (1984), this terminology has become obsolete, and has been replaced by the term “likelihood.” In fact, it is also common to write P(B | A _i ) as L(A _i ).

2.2 Bayesian Inference

The concept of Bayes theorem can be extended from the discrete case to the continuous case. Consider the context of statistical inference where a set of parameters $\boldsymbol{\theta }$ needs to be inferred. All the current knowledge regarding this parameter is represented in the form of a prior distribution denoted by $f'(\boldsymbol{\theta })$. The choice of the prior distribution reflects the subjective knowledge of uncertainty regarding the variable before any observation. It is assumed that the prior distribution is able to explain the data with some degree of uncertainty; in other words, there exists a nonempty set E such that $\forall$ θ ∈ E, the prior probability density function (PDF) and likelihood values evaluated $\forall$ θ ∈ E are both nonzero.

Measurement data (D) is collected on a quantity which depends on the parameter ($\boldsymbol{\theta }$). This information is then used to update the distribution of θ to arrive at the posterior distribution ($f''(\boldsymbol{\theta })$), as:

$$\displaystyle{ f''(\boldsymbol{\theta }) = \frac{L(\boldsymbol{\theta })f'(\boldsymbol{\theta })} {\int L(\boldsymbol{\theta })f'(\boldsymbol{\theta })d\boldsymbol{\theta }} }$$

(3)

In Eq. (3), $L(\boldsymbol{\theta })$ is the likelihood function of $\boldsymbol{\theta }$ and is proportional to $P(D\vert \boldsymbol{\theta })$, i.e. probability of observing the data D conditioned on the parameter $\boldsymbol{\theta }$. Typically, data D is available in the form of independent, paired input–output combinations ($\boldsymbol{x}_{i}$ versus x _i , where i varies from 1 to n), where the input $\boldsymbol{X}$ and output Y are related to one another as:

$$\displaystyle{ y = G(\boldsymbol{x},\boldsymbol{\theta }) }$$

(4)

Considering an observation error $\epsilon$ that is assumed to follow a Gaussian distribution with zero mean and standard deviation σ, the likelihood can be written as:

$$\displaystyle{ L(\boldsymbol{\theta }) =\prod _{ i=1}^{n} \frac{1} {\sigma \sqrt{(2\pi )}}\text{ }\exp -\bigg [\frac{(y_{i} - G(\boldsymbol{x_{i}},\boldsymbol{\theta }))^{2}} {2\sigma ^{2}} \bigg] }$$

(5)

Note that the above equation assumes that the numerical value of σ is known. If this quantity is unknown, σ may be considered to be an argument to the likelihood function and updated along with θ. Equation (5) is substituted in Eq. (3), and the posterior distribution ($f''(\boldsymbol{\theta })$) can be computed.

Note that the denominator on the RHS of Eq. (3) is simply a normalizing constant, which ensures that $f''(\boldsymbol{\theta })$ is a valid PDF, i.e., the integral of the PDF is equal to unity. So, Eq. (3) is sometimes written as:

$$\displaystyle{ f''(\boldsymbol{\theta }) \propto L(\boldsymbol{\theta })f'(\boldsymbol{\theta }) }$$

(6)

The posterior in Bayesian inference is always known only up to a proportionality constant and it is necessary generate samples from this posterior for uncertainty analysis. When there is only one parameter, the proportionality constant can be calculated through one-dimensional integration. Often, multiple parameters may be present, and hence, multi-dimensional integration may not be affordable to calculate the proportionality constant. Therefore, a class of methods popularly referred to as Markov Chain Monte Carlo (MCMC) sampling is used to generate samples from the Bayesian posterior. In general, these methods can be used when it is desired to generate samples from a PDF which is known only up to a proportionality constant. The topic of MCMC will be discussed in detail later in this chapter, in Sect. 3.

2.3 Notes on the Likelihood Function

The likelihood function is defined as the probability of observing data conditioned on the parameters, i.e. $L(\boldsymbol{\theta }) = P(D\vert \boldsymbol{\theta })$; note that, since the data (D) has actually been observed, the terminology “probability of observing the data” is physically meaningless. Therefore, as explained earlier in Sect. 2.1, this quantity was renamed as “the likelihood.” The likelihood function does not follow the laws of probability, and must not be confounded with probability distributions or distribution functions. In fact, Edwards (1984) explains that the likelihood function is meaningful only up to a proportionality constant; the relative values of the likelihood function are alone significant and the absolute values are not of interest.

The concept of likelihood is used in the context of both physical probabilities (frequentist) and subjective probabilities, especially in the context of parameter estimation. In fact, Edwards (1984) refers to the likelihood method as the third or middle way.

From a frequentist point of view (the underlying parameters are deterministic), the likelihood function can be maximized in order to obtain the maximum likelihood estimate of the parameters. According to Fisher (1912), the popular least-squares-based optimization methodology is an indirect approach to parameter estimation and one can “solve the real problem directly” by maximizing the “probability of observing the given data” conditioned on the parameter θ (Aldrich 1997; Fisher 1912). Further, it is also possible to construct likelihood-based confidence intervals for the inferred parameters (Pawitan 2001).

On the other hand, the likelihood function can also be interpreted using subjective probabilities. Singpurwalla (2006, 2007) explains that the likelihood function can be viewed as a collection of “weights” or “masses” and therefore is meaningful only up to a proportionality constant (Edwards 1984). In other words, if L(θ ⁽¹⁾) = 10, and L(θ ⁽²⁾) = 100, then it is 10 ten times more likely for θ ⁽²⁾ than θ ⁽¹⁾ to correspond to the observed data. The entire likelihood function can be used in Bayesian inference, as in Eq. (3), in order to obtain the entire PDF of the parameters.

3 MCMC Sampling

The class of MCMC methods can be used to generate samples from an arbitrary probability distribution, especially when the CDF is not invertible or when the PDF is known only up to a proportionality constant. In Sect. 2.2, it was explained that the latter is the case in Bayesian inference, where the objective is to compute the posterior distribution. Therefore, MCMC sampling can be used to draw samples from the posterior distribution, and these samples can be used in conjunction with kernel density estimation (Rosenblatt 1956) procedure to construct the posterior distribution.

There are several algorithms which belong to the class of MCMC sampling methods. Two such algorithms, the Metropolis algorithm (Metropolis et al. 1953) and the slice sampling (Neal 2003) algorithm are discussed below.

3.1 The Metropolis Algorithm

Assume that a function that is proportional to the PDF is readily available, as f(x); this means that f(x) is not a valid PDF because ∫ f(x)dx ≠ 1. For the purpose of illustration, consider the one-dimensional case, i.e. $x \in \mathbb{R}$. The following steps constitute the algorithm in order to generate samples from the underling PDF. Note that the function f(x) is always evaluated at two points and the ratio is only considered; the effect of the unknown proportionality constant is therefore nullified.

• 1. Set i = 0 and select a starting value x ₀ such that f(x ₀) ≠ 0.

• 2. Initialize the list of samples X = { x ₀}.

• 3. Repeat the following steps; each repetition yields a sample from the underlying PDF.

  • (a) Select a prospective candidate from the proposal density $q(x^{{\ast}}\vert x_{i})$. The probability of accepting this sample is equal to $\frac{f(x^{{\ast}})} {f(x_{i})}$.

  • (b) Calculate acceptance ratio α = min $\big(1, \frac{f(x^{{\ast}})} {f(x_{i})} \big)$.

  • (c) Select a random number u, uniformly distributed on [0, 1].

  • (d) If u < α, then set $x_{i+1} = x^{{\ast}}$, otherwise set $x_{i+1} = x_{i}$.

  • (e) Augment the list of samples in X by x _i+1.

  • (f) Increment i, i.e. $i = i + 1$.

• 4. After the Markov chain converges, the samples in X can be used to construct the PDF of X using kernel density estimation.

The common practice is to generate a few hundreds of thousands of samples and discard the first few thousand samples to ensure the convergence of the Markov Chain.

The Metropolis algorithm (Metropolis et al. 1953) assumes that the proposal density is symmetric, i.e. $q(x^{{\ast}}\vert x_{i}) = q(x_{i}\vert x^{{\ast}})$. A generalization of this algorithm assumes asymmetric proposal density functions $q_{1}(x^{{\ast}}\vert x_{i})$ and $q_{2}(x_{i}\vert x^{{\ast}})$; this algorithm is referred to as Metropolis–Hastings algorithm (Hastings 1970). The only difference is that the probability of accepting the prospective candidate is calculated as $\frac{f(x^{{\ast}})q_{ 2}(x_{i}\vert x^{{\ast}})} {f(x_{i})q_{1}(x^{{\ast}}\vert x_{i})}$.

3.2 Slice Sampling

Consider the same function f(x), i.e. the PDF of X, known up to a proportionality constant. The steps of the slice sampling algorithm are as follows:

• 1. Set i = 0 and select a starting value x ₀ such that f(x ₀) ≠ 0.

• 2. Draw a random number y from the uniform distribution [0, f(x)].

• 3. Consider the set f ⁻¹[y, ∞); note that this set may not be convex, especially when the target distribution is multi-modal. Select a sample which is uniformly distributed on this set. Assign $i = i + 1$, and call this sample x _i .

• 4. Repeat Steps 1–3 to generate multiple samples of X and construct the PDF of X using kernel density estimation.

In contrast with the previously discussed Metropolis algorithm, the slice sampling algorithm is not a acceptance–rejection algorithm.

3.3 MCMC Sampling: Summary

In addition to the above algorithms, other MCMC sampling algorithms such as Gibbs sampling (Geman and Geman 1984), multiple-try Metropolis (Liu et al. 2000), and Metropolis-within-Gibbs (Roberts and Rosenthal 2006) are also discussed in the literature. One critical disadvantage of MCMC sampling approaches is that they may require several hundreds of thousands of samples, and in turn, several hundreds of thousands of evaluations of G in Eq. (4), which may be computationally prohibitive. Therefore, it is common in engineering to replace G (which may be a complicated physics-based model) with an inexpensive surrogate, such as the Gaussian process surrogate model.

4 Gaussian Process Surrogate Modeling

The use of sampling techniques involves repeated evaluations of mathematical models, which may be computationally intensive. One approach to overcome this computational difficulty is to make use of surrogate models to replace the original physics-based model. A few evaluations of the original model are used to train this inexpensive, efficient surrogate model. Different types of surrogate modeling techniques such as polynomial response surface (Rajashekhar and Ellingwood 1993), polynomial chaos expansion (Ghanem and Spanos 1990), support vector regression (Boser et al. 1992), relevance vector regression (Tipping 2001), and Gaussian process interpolation (Bichon et al. 2008; Rasmussen 2004; Santner et al. 2003) have been investigated in the literature.

The Gaussian process interpolation is a powerful technique based on spatial statistics and is increasingly being used to build surrogates to expensive computer simulations for the purposes of optimization and uncertainty quantification (Bichon et al. 2008; Rasmussen 2004; Santner et al. 2003). The GP model is preferred in this research because (1) it is not constrained by functional forms; (2) it is capable of representing highly nonlinear relationships in multiple dimensions; and (3) can estimate the prediction uncertainty which depends on the number and location of training data points.

The basic idea of the GP model is that the response values Y evaluated at different values of the input variables X are modeled as a Gaussian random field, with a mean and covariance function. Suppose that there are m training points, x ₁, x ₂, $x_{3}\ \ldots \ x_{m}$ of a d-dimensional input variable vector, yielding the output values y(x ₁), y(x ₂), $y(x_{3})\ \ldots \ y(x_{m})$. The training points can be compactly written as x _T vs. y _T where the former is a m × d matrix and the latter is a m × 1 vector. Suppose that it is desired to predict the response (output values y _P ) corresponding to the input x _P , where x _P is p × d matrix; in other words, it is desired to predict the output at n input combinations simultaneously. Then, the joint density of the output values y _P can be calculated as:

$$\displaystyle{ p(y_{P}\vert x_{P},x_{T},y_{T};\Theta ) \sim N(m,S) }$$

(7)

where $\Theta$ refers to the hyperparameters of the Gaussian process, which need to be estimated based on the training data. The prediction mean and covariance matrix (m and S, respectively) can be calculated as:

$$\displaystyle\begin{array}{rcl} m& =& K_{PT}(K_{TT} +\sigma _{ n}^{2}I)^{-1}y_{ T} \\ S& =& K_{PP} - K_{PT}(K_{TT} +\sigma _{ n}^{2}I)^{-1}K_{ TP}{}\end{array}$$

(8)

In Eq. (8), K _TT is the covariance function matrix (size m × m) amongst the input training points (x _T ), and K _PT is the covariance function matrix (size p × m) between the input prediction point (x _P ) and the input training points (x _T ). These covariance matrices are composed of squared exponential terms, where each element of the matrix is computed as:

$$\displaystyle{ K_{ij} = K(x_{i},x_{j};\Theta ) = -\frac{\theta } {2}\left [\sum _{q=1}^{d}\frac{(x_{i,q} - x_{j,q})^{2}} {l_{q}} \right ] }$$

(9)

Note that all of the above computations require the estimate of the hyperparameters $\Theta$; the multiplicative term (θ), the length scale in all dimensions (l _q , q = 1 to d), and the noise standard deviation (σ _n ) constitute these hyperparameters ($\Theta =\{\theta,l_{1},l_{2}\ \ldots \ l_{d},\sigma _{n}\}$). As stated earlier, these hyperparameters are estimated based on the training data by maximizing the following log-likelihood function:

$$\displaystyle{ \log \text{ }p\big(y_{T}\vert x_{T};\Theta \big) = -\frac{y_{T}^{T}} {2} \big(K_{TT} +\sigma _{ n}^{2}I\big)^{-1}y_{ T} -\frac{1} {2}\log \vert \big(K_{TT} +\sigma _{ n}^{2}I\big)\vert + \frac{d} {2}\log (2\pi ) }$$

(10)

Once the hyperparameters are estimated, then the Gaussian process model can be used for predictions using Eq. (8). Note that the “hyperparameters” of the Gaussian process are different from the “parameters” of a generic parametric model (e.g., linear regression model). This is because, in a generic parametric model, it is possible to make predictions using only the parameters. On the contrary, in the case of the Gaussian process model, all the training points and the hyperparameters are both necessary to make predictions, even though the hyperparameters may have estimated previously. For details of this method, refer to Bichon et al. (2008); Chiles and Delfiner (1999); Cressie (1991); McFarland (2008); Rasmussen (1996, 2004); Santner et al. (2003), and Wackernagel (2003).

An important issue in the construction of the Gaussian process model is the selection of training points. In general, the training points may arise out of field experiments or may be generated using a computer code. Model parameter estimation considers the latter case and hence, there is no noise in the data, thereby eliminating σ _n from the above equations. Adaptive techniques can be used to select training points for the GP model, in order to construct the response surface to a desired level of accuracy or precision. Since the GP model is capable of estimating the variance in model output, a variance minimization algorithm proposed by McFarland (2008) identifies the next training point at the input variable value which corresponds to the largest variance. This selection algorithm is repeated and training points are adaptively identified until the estimated variance is below a desired threshold. Alternatively, another training point selection algorithm has been developed by Hombal and Mahadevan (2011), where the focus is to select successive training points so that the bias error in the surrogate model is minimized.

Once the training points are selected and the surrogate model is constructed, it can be used for several uncertainty quantification activities such as uncertainty propagation [through Monte Carlo simulation (MCS)], inverse analysis and parameter estimation (through MCMC simulation), and sensitivity analysis. It must be noted that the replacement of a complex computer simulation with an inexpensive surrogate leads to approximations; therefore, it is important to include the effect of this approximation in the procedure for overall uncertainty quantification (Sankararaman 2012).

5 Bayesian Networks

The previous sections of this chapter discussed certain fundamental concepts of the Bayesian methodology, in general. The Bayesian inference approach for model parameter estimation was presented, and the use of MCMC sampling and the importance of using Gaussian process surrogate models were explained. Most of this discussion dealt with single-level models that may represent a component or a subsystem. Since complex engineering systems consist of many components and subsystems, it is important to integrate information across all components and subsystems, in order to compute the overall uncertainty in the system-level prediction. This goal can be accomplished through the use of a Bayesian network.

A Bayesian network (Heckerman 2008; Jensen 1996) is an acyclic, graphical representation of conditional probability relationships between uncertain quantities. Each uncertain quantity is represented as a node and successive links are connected to each other using unidirectional arrows that express dependence in terms of conditional probabilities. Disconnected nodes imply independence between the corresponding random variables. Figure 1 shows a conceptual Bayesian network that aids in uncertainty quantification across multiple levels of models and observed data. Circles correspond to uncertain variables and squares represent observed data. A solid line arrow represents a conditional probability link, and a dashed line arrow represents the link of a variable to its observed data if available.

Fig. 1

Bayesian network illustration

In Fig. 1, a system level output Z is a function of two subsystem level quantities Y ₁ and Y ₂; in turn, Y ₁ is a function of subsystem-level input X ₁ and model parameter $\Theta _{1}$, and similarly, Y ₂ is a function of subsystem-level input X ₂ and model parameter $\Theta _{2}$. For example, in a beam deflection study, the applied force is an input, the elastic modulus is a model parameter, while the deflection is measured and a model is built to predict the same. Experimental data $D_{Y _{1}}$ and $D_{Y _{2}}$ are available for comparison with the respective model predictions Y ₁ and Y ₂.

5.1 Uncertainty Propagation: Forward Problem

In the forward problem, the probability distributions of the inputs (X ₁ and X ₂) and model parameters ($\Theta _{1}$ and $\Theta _{2}$) are known or assumed, and these distributions are used to calculate the PDF of Y ₁ and Y ₂, which in turn are used to calculate the PDF of the system-level output Z, as:

$$\displaystyle\begin{array}{rcl} f_{Z}(z)& =& \text{ }\int f_{Z}(z\vert y_{1},y_{2})f_{Y _{1}}(y_{1})f_{Y _{2}}(y_{2})f_{Y _{1}}(y_{1})f_{Y _{2}}(y_{2})dy_{1}dy_{2} \\ f_{Y _{1}}(y_{1})& =& \text{ }\int f_{Y _{1}}(y_{1}\vert x_{1},\theta _{1})f_{X_{1}}(x_{1})f_{\Theta _{1}}(\theta _{1})dx_{1}d\theta _{1} \\ f_{Y _{2}}(y_{2})& =& \text{ }\int f_{Y _{2}}(y_{2}\vert x_{2},\theta _{2})f_{X_{2}}(x_{2})f_{\Theta _{2}}(\theta _{2})dx_{2}d\theta _{2} {}\end{array}$$

(11)

Note that uppercase letters are used to denote random variables and the corresponding lowercase letters are used to denote realizations of those random variables. Equation (11) can be solved using methods of uncertainty propagation such as MCS, first-order reliability method (FORM), and second-order reliability method (SORM) (Haldar and Mahadevan 2000).

5.2 Inference: Inverse Problem

In the inverse problem, the probability densities of the model parameters ($\Theta _{1}$ and $\Theta _{2}$ in Fig. 1) can be updated based on the observed data ($D_{Y _{1}}$ and $D_{Y _{2}}$) using Bayes’ theorem as:

$$\displaystyle{ f_{\Theta _{1},\Theta _{2}}(\theta _{1},\theta _{2}\vert D_{Y _{1}},D_{Y _{2}}) = CL(\theta _{1},\theta _{2})f_{\Theta _{1}}(\theta _{1})f_{\Theta _{2}}(\theta _{2}) }$$

(12)

In Eq. (12), the prior distributions of the model parameters $\Theta _{1}$ and $\Theta _{2}$ are given by $f_{\Theta _{1}}(\theta _{1})$ and $f_{\Theta _{2}}(\theta _{2})$, respectively. The choice of the prior distribution reflects the subjective knowledge of uncertainty regarding the variable before any testing. It is assumed that the prior distribution is able to explain the data with some degree of uncertainty; in other words, there exists a nonempty set E such that $\forall$ $\{\Theta _{1},\Theta _{2}\}\ \in E$, the prior PDF ($f_{\Theta _{1}}(\theta _{1})f_{\Theta _{2}}(\theta _{2})$) and likelihood ($L(\theta _{1},\theta _{2})$) values evaluated at $\{\Theta _{1},\Theta _{2}\}$ are both non-zero.

The joint posterior density of the parameters is given by $f_{\Theta _{1},\Theta _{2}}(\theta _{1},\theta _{2}\vert D_{Y _{1}},D_{Y _{2}})$. The likelihood function $L(\theta _{1},\theta _{2})$ is calculated as the probability of observing the given data ($D_{Y _{1}},D_{Y _{2}}$), conditioned on the parameters being updated, i.e. $\Theta _{1}$ and $\Theta _{2}$. The likelihood function accounts for the uncertainty in the inputs X ₁ and X ₂. For details of the likelihood function, refer to Edwards (1984), Pawitan (2001), and Sankararaman (2012). As explained earlier in Sect. 3, Eq. (12) can be evaluated by generating samples of model parameters ($\Theta _{1}$ and $\Theta _{2}$) through MCMC sampling.

6 Test Resource Allocation

Typically, a multi-level, multi-physics system has several parameters that influence the overall system-level output, and the uncertainty in these parameters can be updated by tests at multiple levels of the system and multiple types of physics coupling. When the posterior distributions of the parameters are propagated through the system model to calculate the overall system-level output, the posterior variance of the overall system-level prediction can be computed. With more acquisition of data, a decreasing trend can be observed in the variance of the system-level output.

Two types of questions need to be answered: (1) What type of tests to do (which component, isolated physics, etc.)? and (2) How many repetitions of each type? Each type of test has a different testing cost and an associated reduction in the variance of system-level prediction. Further, the same type of test may need to be repeated on nominally identical specimens of the same component or subsystem. Such repetition is performed in order to account for the effect of natural variability across nominally identical specimens; while each repetition may have the same monetary cost, the associated reduction in the variance of system-level prediction may be different.

The test conducted on one subsystem is assumed to be statistically independent of another test on another subsystem; in other words, one type of test is independent of any other type. Further, for a given type of test, the repetitions across multiple replicas are also considered to be independent. It is assumed that a model is available to predict the quantity being measured in each type of test; the model may have several outputs but only that output which is measured is of concern. The overall objective is to identify how many tests of each type must be performed so as to achieve the required reduction in the variance of the system-level output. If there are several system-level outputs, either an aggregate measure or the most critical output can be considered. However, multi-objective optimization formulations to simultaneously reduce the variance of more than one system-level output have not yet been addressed in the literature.

6.1 Sensitivity Analysis

The method of sensitivity analysis has been used to quantify the sensitivity of model output to parameters. While derivative-based methods only compute local sensitivities, the method of global sensitivity analysis (Saltelli et al. 2008) to apportion the variance in the system-level output to the various sources of uncertainty, and thereby guide in the reduction of system-level prediction uncertainty.

The first step of the resource allocation methodology is to use sensitivity analysis and identify those parameters that have a significant influence on the variance of the overall system-level prediction. Once the “important” parameters are identified, only those tests that aid in reducing the uncertainty in these important parameters can be performed. For example, consider a system-level output that is very sensitive to the uncertainty in the parameters of sub-system-I but not sensitive to the parameters of sub-system-II, then it is logical to perform more sub-system-I tests than sub-system-II tests. Note that this procedure for test identification is only a preliminary approach. This approach can answer the question—“which tests to do?” In order to answer the question, “how many tests to do?”, it is necessary to quantify the decrease in variance that may be caused due to a particular test. The effect of a particular test on variance reduction can be quantified by using Bayesian updating. Therefore, the resource allocation methodology first uses sensitivity analysis for selection of calibration parameters and then uses Bayesian updating to quantify the effect of a test on the variance of system-level prediction.

6.2 Optimization Formulation

In order to solve the resource allocation problem and identify the number of tests to be performed for each type, the optimization problem can be formulated in two ways, as explained below.

In the first formulation shown in Eq. (13), the goal is to minimize the variance of the system-level output subject to satisfying a budget constraint.

$$\displaystyle\begin{array}{rcl} \mathop{\text{ Minimize }}\limits_{N_{\mathrm{test}}}& \text{ } & E(\mathrm{Var}(R)) \\ \text{s.t.}\sum _{i=1}^{q}(C_{ i}N_{i})& \leq & \text{Total Budget} \\ N_{\mathrm{test}}& =& \text{ }\ [N_{1},N_{2}\ldots N_{q}]{}\end{array}$$

(13)

In Eq. (13), q refers to the number of different types of possible tests. The cost of the ith (i = 1 to q) type of test is equal to C _i , and N _i (decision variable) denotes the number of repetitions of the ith type of test. Let D _i denote all the data collected through the ith type of test. Let N _test denote the vector of all N _i ’s and let $\boldsymbol{D}$ denote the entire set of data collected from all q types of tests.

Alternatively, the resource allocation problem can be formulated by minimizing the cost required to decrease the variance of the system-level output below a threshold level, as:

$$\displaystyle\begin{array}{rcl} \mathop{\text{ Minimize }}\limits_{N_{\mathrm{test}}}& \text{ } & \sum _{i=1}^{q}(C_{ i}N_{i}) \\ \text{s.t. }E(\mathrm{Var}(R))& \leq & \text{ Threshold Variance } \\ N_{\mathrm{test}}& =& \text{ }\ [N_{1},N_{2}\ldots N_{q}] {}\end{array}$$

(14)

Sankararaman et al. (2013) pursued the first formulation (Eq. (13)) because the threshold level for the variance is assumed to be unknown. Using $\boldsymbol{D}$, the model parameters are calibrated and the system-level response ($R(\boldsymbol{D})$) is computed. The optimization in Eq. (13) calculates the optimal values of N _i , given the cost values C _i , such that the expected value of variance of the system-level prediction (E(Var(R))) is minimized, while the budget constraint is satisfied.

This optimization formulation uses E(Var(R)) as the objective function because R is a function of $\boldsymbol{D}$, which is not available before testing. Hence, random realizations of the test data set ($\boldsymbol{D}$) are generated; each random realization is used to compute $\mathrm{Var}(R\vert \boldsymbol{D})$, and the expectation over such random realizations is calculated to be the objective function, as:

$$\displaystyle{ E(\mathrm{Var}(R)) =\int \mathrm{Var}(R\vert \boldsymbol{D})f(\boldsymbol{D})d\boldsymbol{D} }$$

(15)

where $f(\boldsymbol{D})$ is the density considered for the test data. Assuming that one type of test is performed independent of the another (i.e., a subsystem-level test is independent of a material-level test), Eq. (15) can be written as:

$$\displaystyle{ E(\mathrm{Var}(R)) =\int \mathrm{Var}(R\vert D_{1},D_{2}\ldots D_{q})f(D_{1})f(D_{2})\ldots f(D_{q})dD_{1}dD_{2}\ldots dD_{q} }$$

(16)

where f(D _i ) is the density considered for the data obtained through the ith test. Before any testing is done, all prior knowledge regarding the model parameters and the mathematical models constitute the only information available for the calculation of f(D _i ). Therefore, f(D _i ) is calculated as:

$$\displaystyle{ f(D_{i}) =\int f(y_{i}\vert \boldsymbol{\theta }_{i})f'(\boldsymbol{\theta }_{i})d\boldsymbol{\theta }_{i} }$$

(17)

where y _i represents the output of the mathematical model corresponding to the ith type of test, $\boldsymbol{\theta }_{i}$ represents the underlying parameters, and $f'(\boldsymbol{\theta }_{i})$ represents the prior knowledge regarding those parameters. Note that Eq. (17) is simply an uncertainty propagation problem, where the other sources of uncertainty (such as physical variability in inputs, solution approximation errors, data uncertainty) can also be included in the computation of $f(y_{i}\vert \boldsymbol{\theta }_{i})$.

Equations (15)–(17) are implemented using a numerical algorithm, where a finite number of realizations of $\boldsymbol{D}$ are generated and E(Var(R)) is computed over these realizations. Then, E(Var(R)) can be minimized using the optimization in Eq. (13), and the ideal combination of tests can be identified.

Note that an inequality constraint (for the budget), and not an equality constraint, is considered in Eq. (13). This means that the optimal solution which minimizes E(Var(R)) need not necessarily exhaust the budget. Consider the simple case where there are two possible test types (C ₁ = 2 and C ₂ = 3), and the budget is equal to six cost units. There are two test combinations which exhaust the budget: (1) [N ₁ = 3, N ₂ = 0], and (2) [N ₁ = 0, N ₂ = 2]. Suppose that these two combinations lead to a value of E(Var(R)) which is greater than that achieved through the test combination [N ₁ = 1, N ₂ = 1]. Then, obviously the combination [N ₁ = 1, N ₂ = 1] must be selected because it achieves the goal of reducing E(Var(R)) even though it may not exhaust the budget.

6.3 Solution of the Optimization Problem

Equation (13) is a complicated integer optimization problem, where Bayesian updating and forward propagation need to be repeated for each random realization of the test data in order to evaluate the objective function, thus increasing the computational cost several fold. In spite of the use of Gaussian process surrogate models to replace the expensive system model, high computing power is still needed to solve the optimization problem.

Integer optimization is sometimes solved using an approximation method, where the integer constraint is first relaxed, and the integers nearest to the resulting optimal solution are used in further solution of the original (un-relaxed) problem. Unfortunately, this approach is not applicable to the solution of Eq. (13), since the objective function (system-level prediction variance) is defined and computed only for integer-valued decision variables (number of tests). It is meaningless to have a non-integer number of tests.

A multi-step procedure for solving the optimization problem was proposed by Sankararaman et al. (2013). Within each step, the global optimal solution is computed using an exhaustive search process, whereas across steps, a greedy algorithm is pursued. The step size is chosen in cost units, and additional steps are added until the budget constraint is satisfied.

Let the size of the first step be equal to ϕ ¹ cost units; the globally optimal testing combination for this cost ( = ϕ ¹) is denoted by $N_{\mathrm{test}}^{1}$, and is calculated using exhaustive search, as:

$$\displaystyle{ \begin{array}{c} \mathop{\text{ Minimize }}\limits_{N_{\mathrm{test}}^{1}}E(\mathrm{Var}(R))\\ \\ \\ \\ \text{s.t.}\sum _{i=1}^{q}(C_{i}N_{i}^{1}) \leq \phi ^{1}\\ \\ \\ \\ N_{\mathrm{test}}^{1} = [N_{1}^{1},N_{2}^{1}\ldots N_{q}^{1}] \end{array} }$$

(18)

The optimization procedure in the second stage is dependent on the optimal solution from the first stage, i.e. $N_{\mathrm{test}}^{1}$. In general, the optimization for the jth stage, given the solution in the previous stage (i.e., $N_{\mathrm{test}}^{j-1}$), is performed for cost = ϕ ^j. Note that $\mathop{\sum }\limits_{j}\phi ^{j} =$ Total budget. The jth optimization is formulated as:

$$\displaystyle{ \begin{array}{c} \mathop{\text{ Minimize }}\limits_{N_{\mathrm{test}}^{j,\mathrm{new}}}E(\mathrm{Var}(R))\\ \\ \\ \\ \text{s.t. }\sum (C_{i}N_{i}^{j,\mathrm{new}}) \leq \phi ^{j}(i = 1\text{ to }q)\\ \\ \\ \\ N_{\mathrm{test}}^{j} = N_{\mathrm{test}}^{j-1} + N_{\mathrm{test}}^{j,\mathrm{new}}\\ \\ \\ \\ N_{\mathrm{test}}^{j,\mathrm{new}} = [N_{1}^{j,\mathrm{new}},N_{2}^{j,\mathrm{new}}\ldots N_{q}^{j,\mathrm{new}}] \end{array} }$$

(19)

As seen in Eq. (19), the decision variables for the jth stage are $N_{\mathrm{test}}^{j,\mathrm{new}}$, i.e. those tests which need to be performed in the jth stage; therefore the total number of tests is equal to the sum of $N_{\mathrm{test}}^{j,\mathrm{new}}$ and $N_{\mathrm{test}}^{j-1}$, i.e. the optimal number of tests in the previous stage. The same procedure is repeated until no additional test can be performed with the budget constraint satisfied.

The selection of step size for a given budget is an important issue. The true global optimal solution can be calculated by considering one step whose size is equal to the entire budget. However, due to the large number of possible testing combinations, this approach may be computationally infeasible. In a practical problem, several steps are considered, and the step sizes must be chosen judiciously based on (1) the costs of each type of test; (2) time required for each Bayesian update; (3) number of random realizations of data needed to compute E(Var(R)); and (4) the test combinations that are suitable for the chosen step size; a very small step size may not even include an expensive type of test.

6.4 Summary of the Optimization Methodology

The various steps of the optimization-based methodology for test resource allocation are summarized below:

• 1. Construction of the Bayesian network: The first step is to identify the various component-level, subsystem-level, and system-level models. Each model has an output quantity and correspondingly, a test can be performed to measure this quantity. All the models are connected through the Bayesian network, and the data available across the nodes is also indicated. The model errors, if available, can also be included in the Bayesian network. Though solution approximation errors can be calculated prior to testing and included in the Bayes network, model form error cannot be calculated before testing. It must be noted that the Bayesian network, due to its acyclic nature, cannot account for feedback coupling between models. When the system-level response is a coupled physics-based solution, the overall coupled solution is directly included in the Bayesian network instead of considering the individual physics separately.

• 2. Sensitivity analysis: The next step is to perform global sensitivity analysis and identify the “important” parameters that significantly contribute to the uncertainty in the system-level response. Then, those tests which can aid in the reduction of uncertainty in these “important” parameters are selected for consideration in the optimization for test resource allocation.

• 3. Bayesian updating: The next step is to perform Bayesian updating and calibrate parameters for a particular realization of measurement data. Then, this needs to be repeated by generating multiple realizations of measurement data in order to compute the expected value of variance, as in Eq. (15). Due to the required computational expense, the original physics models can be replaced with Gaussian process surrogates. Though this does not lead to analytical calculation of the posterior, it increases the computational efficiency several fold.

• 4. Resource allocation optimization: The final step is to perform the resource allocation optimization using the multi-step procedure developed in Sects. 6.2 and 6.3. It may be useful to verify that the resultant solution is actually optimal by computing E(Var(R)) for few other N _test values.

The rest of this chapter illustrates the optimization-based test resource allocation methodology using three different numerical examples. While Sect. 7 deals with a simple mathematical example, Sects. 8 and 9 consider multi-physics and multi-level engineering systems, respectively.

7 Illustration Using Mathematical Example

This subsection presents a simple illustrative example to illustrate the optimization-based methodology for test resource allocation. In order to focus on this objective, simple mathematical relationships are chosen (even the system-level response has no coupling), and measurement errors are assumed to be negligible. Other features such as coupled system response, measurements errors, and solution approximation errors (while replacing the underlying physics-based model with a Gaussian process approximation) are considered later in Sects. 9 and 8.

The Bayesian network for this problem is exactly the same as that in Fig. 1. There are four independent quantities and three dependent quantities; the numerical details of this problem are specified in Table 1. The notation N(μ, σ) is used to represent a normally distributed quantity with mean μ and standard deviation σ. Two types of tests (on two different lower levels) can be done and this information is used to update the uncertainty in the system-level response based on the tests.

Table 1 Numerical details

Probability distributions are assumed to be available for the inputs X ₁ and X ₂; if this information was not available, and only sparse and/or interval data was available for the inputs, then the likelihood-based method developed in Sankararaman and Mahadevan (2011) can be used to construct a probability distributions for them. The variance of the system-level response quantity Z before conducting any test (i.e., by propagating the above distributions of X ₁, X ₂, $\Theta _{1}$, and $\Theta _{2}$ through the models) is 142 units. The objective is calculate the number of tests on Y ₁ and Y ₂ (N ₁ and N ₂), that will lead to a minimum variance in Z, subject to a total budget of $50 cost units. Since there are only two parameters, global sensitivity analysis is not necessary, and hence, both $\Theta _{1}$ and $\Theta _{2}$ are chosen for calibration. The optimization-based methodology discussed in Sect. 6 is used for this purpose; five different stages are considered and the available budget in each stage is considered to be $10. The results of test prioritization are given in Table 2 (the optimal value in each stage is indicated in bold) and Fig. 2.

Table 2 Resource allocation: results

Fig. 2

Variance vs. cost

At the end of the optimization procedure, the optimal combination is found to be four tests on Y ₁ and 2 tests on Y ₂. Further, this solution was verified by considering all other combinations (exhaustive search) of N ₁ and N ₂ and computing the corresponding E(Var(R)); for this illustrative example, this verification is numerically affordable. However, for practical examples, a few random values of $N_{\mathrm{test}} = [N_{1},N_{2}]$ (if not all) can be considered and it can be verified if the estimated solution is really optimal.

8 Numerical Example: Multi-Physics System

8.1 Description of the Problem

This coupled-physics thermal vibration example illustrates a laboratory experiment which can be used to study and simulate the behavior in solar arrays of telescopes and spacecraft booms (Thornton 1996). In this experiment, a thin walled circular tube is rigidly attached at its top and supports a concentrated mass at its bottom. The tube and the mass are initially at rest and a constant heat flux is applied on one side along the length of the tube. The application of the heat flux causes an increase in the temperature on the incident surface while the unheated side remains at the initial temperature. The temperature gradient causes the beam to bend away from the lamp, due to the thermal moment. The displacement of the beam, in turn, changes the distribution of temperature along the length of the beam, leading to a change in the temperature gradient and the thermal moment, which in turn affects the flexural behavior. Thus the combination of heat transfer and flexural mechanics leads to oscillations of the beam. The setup of this experiment is shown in Fig. 3.

Fig. 3

Thermally induced vibration

The temperature at the tip mass (T _m ) is given by the following differential equation:

$$\displaystyle{ \frac{\partial T_{m}} {\partial t} + \frac{T_{m}} {\tau } = \frac{T^{{\ast}}} {\tau } \bigg(1 -\frac{v(x,t)} {\beta ^{{\ast}}} \bigg) }$$

(20)

In Eq. (20), v(x, t) represents the displacement of the beam as a function of length and time. Thornton (1996) explains how to calculate the parameters T ^∗, τ, β ^∗ as a function of the incident solar flux (S).

The displacement v(x, t) can be related to the displacement of the tip mass V (t) as:

$$\displaystyle{ v(x,t) =\Bigg (\frac{3x^{2}} {2l^{2}} - \frac{x^{3}} {2l^{3}}\Bigg)V (t) }$$

(21)

The tip mass displacement V (t), in turn, depends on the forcing function as follows:

$$\displaystyle{ \ddot{V } + 2\xi \omega _{0}\dot{V } +\Bigg (\omega _{0}^{2} + \frac{6g} {5l} \Bigg)V = \frac{F(t)} {m} }$$

(22)

In Eq. (5), ξ is the damping ratio, and ω ₀ is the angular frequency. The forcing function F(t) depends on the thermal moment which in turn depends on the temperature, thereby casing coupling between the thermal equation and the structural equation. These relations are shown in the following equations:

$$\displaystyle{ F(t) = -\frac{3} {l^{3}}\int _{0}^{l}\int _{ 0}^{x}M(u,t)dudx }$$

(23)

$$\displaystyle{ M(x,t) =\int E\alpha T_{m}(x,t)cos(\Phi )ydA }$$

(24)

In Eq. (24), E is the elastic modulus, α is the coefficient of thermal expansion, $\Phi$ is the angle of incident flux on the cross section, y is the distance from the center of the cross section and the integral is over the area of the cross section A. Refer Thornton (1996) for a detailed description of this problem.

The overall objective of test resource allocation is to minimize the variance of the system-level output (R), which is defined to be the ratio of displacement amplitudes at two different time instants for the coupled system when the incident solar flux (S) is 2,000 W/m². If R < 1, the system is stable with oscillations diminishing as a function of time. If R > 1, the system is unstable, commonly referred to as flutter, an undesirable scenario. While a Gaussian process model is constructed to calculate the multi-physics response R, individual physics predictions are performed using the above described physics-based models.

There are several parameters (both thermal and structural) in the above equations that can be calibrated using test data. The method of sensitivity analysis is used to identify five parameters, which significantly contribute to the uncertainty in the system-level prediction. The prior means are based on Thornton (1996), and the assumed coefficients of variation (CoV) are tabulated in Table 3; note that the radius being a geometric property has a lower CoV. The calibrated parameters are then used to quantify the uncertainty in R.

Table 3 Calibration quantities: thermal vibration problem

The calibration parameters need to be estimated during test data; four different types of tests are considered, as shown in Table 4. The total budget available for testing is assumed to be $2,000. It is assumed that the entire multi-disciplinary system cannot be tested.

Table 4 Types of tests: thermal vibration problem

The calibration quantities, the model predictions, and the test data are connected through a Bayesian network, as shown in Fig. 4.

Fig. 4

Thermal vibration: Bayesian network

In the Bayesian network in Fig. 4, “Temp” refers to temperature, “Accn” refers to the acceleration, “Disp” refers to the displacement, and “Amp” refers to the amplitude of vibration. Measurement errors (ε) are assumed to have a standard deviation that is equal to 10 % of the model prediction. This Bayesian network is used for uncertainty quantification, Bayesian updating and resource allocation.

8.2 Resource Allocation

The objective is to calculate the number of tests that lead to maximum reduction in variance in R. Let N _test denote the number of tests, where $N_{\mathrm{test}} = [N_{m_{1}},N_{m_{2}},N_{F},N_{T}]$; where $N_{m_{1}}$ is the number of material level temperature tests, $N_{m_{2}}$ is the number of material level pluck tests, N _F is the number of flexural subsystem tests, and N _T is the number of thermal subsystem tests. Let $D = [D_{m_{1}},D_{m_{2}},D_{F},D_{T}]$ denote the test measurements. The optimization problem for resource allocation can be formulated as shown in Eq. (25)

$$\displaystyle\begin{array}{rcl} & \mathop{\text{ Minimize}}\limits_{N_{\mathrm{test}}}E(\mathrm{Var}(R)) & \\ & \text{s.t.}100(N_{m_{1}} + N_{m_{2}}) + 500(N_{F} + N_{T}) \leq 2000& \\ & N_{\mathrm{test}} = [N_{m_{1}},N_{m_{2}},N_{F},N_{T}] &{}\end{array}$$

(25)

The above optimization is solved using the multi-stage optimization procedure discussed in Sects. 6.2 and 6.3. Four stages and a budget of $500 for each stage are considered, thereby accounting for the total budget of $2,000. Each stage has eight options (as against two in the mathematical example in Sect. 7); only the optimal solution in each stage is shown.

Note that Table 5 expresses the expectation of variance of R in terms of percentage of the variance before any testing; this variance is equal to 5. 69 × 10⁻⁷; since R is a ratio, this variance is dimensionless.

Table 5 Resource allocation results: thermal vibration problem

For a $2,000 budget, it is seen that one temperature test, nine pluck tests, one thermal subsystem test and one flexural subsystem test are required to achieve the maximum reduction in the variance of R. The results show that while it is useful to do all the tests, repeating the pluck test which calibrates structural damping, is not only cheap but also leads to effective decrease in the variance of R. The decrease of variance with cost is shown in Fig. 5.

Fig. 5

Decrease of variance with cost. (a) Variance with cost; (b) log(Variance) with cost

It is seen that the reduction in variance using the last $1,000 (i.e., from $1,000 to $2,000) was much smaller when compared to the reduction in variance using the initial $1,000. Such information is very useful for budgeting purposes, since all the above computation (and practical resource allocation) is done before any test is actually conducted.

9 Numerical Example: Multi-Level System

9.1 Description of the Problem

A three-level structural dynamics developed at Sandia National Laboratories (Red-Horse and Paez 2008) is considered as shown in Fig. 6.

Fig. 6

Multi-level structural dynamics problem. (a) Level 1; (b) level 2; (c) level 3

The first level (component) consists of a single spring-mass-damper. Three such spring-mass dampers are integrated to form a spring-mass-damper subsystem in the second-level. In the third level, the integrated spring-mass-damper subsystem is mounted on a beam to form the overall system.

The models to represent the first two levels are straightforward (Chopra 1995). Red-Horse and Paez (2008) describe in detail the modeling and simulation of the overall system (third-level). The overall objective is to test resource allocation to minimize the variance of the system level output (R) which is defined to be the maximum acceleration of mass m ₃, when a random force is applied as specified in Red-Horse and Paez (2008). The first-level and second-level responses are computed using physics-based models while the third-level and system-level responses are computed by constructing two Gaussian process surrogate models.

In this numerical example, the stiffness values of the three masses, i.e. k ₁, k ₂, and k ₃ are all the parameters that need to be calibrated with test data; since all parameters are calibrated, sensitivity analysis is not used in this example. The numerical values (in SI units) of three calibration parameters are summarized in Table 6.

Table 6 Model parameters: structural dynamics problem

The mass of the beam is taken to be 0.1295. Further numerical details of the beam are given in Red-Horse and Paez (2008).

Data for calibration is assumed to be available through five different types of tests. The details of these different types of tests are provided in Table 7. For each test, a sinusoidal load (amplitude  = 10,000 and angular velocity = 10 rad s⁻¹) is used. For the first and second level tests, the sinusoidal load is applied at the base; for the third level test, the sinusoidal load is applied as specified in Red-Horse and Paez (2008).

Table 7 Model parameters: structural dynamics problem

The model predictions, experimental data, and the calibration quantities are connected using the Bayesian network, shown in Fig. 7. The corresponding experimental errors are denoted by ε ₁₁, ε ₁₂, ε ₁₃, ε ₂, and ε ₃, respectively, and assumed to be equal to 10 % of the prediction.

Fig. 7

Bayesian network

This Bayesian network can be used for uncertainty quantification, Bayesian updating, and resource allocation, as explained below.

9.2 Resource Allocation

In the resource allocation problem, testing is yet to be done and hence realizations of future experimental data are generated randomly. Then, E(Var(R)) is computed so as to identify which set of tests will lead to the maximum reduction in variance. Let $N_{\mathrm{test}} = [N_{m_{1}},N_{m_{2}},N_{m_{3}},N_{2},N_{3}]$. The optimization problem for resource allocation can be formulated as shown in Eq. (26).

$$\displaystyle\begin{array}{rcl} & \mathop{\text{Minimize }}\limits_{N_{\mathrm{test}}}E(\mathrm{Var}(R)) & \\ & \text{s.t. }100(N_{m_{1}} + N_{m_{2}} + N_{m_{3}}) + 500N_{2} + 1000N_{3} \leq 1000& \\ & N_{\mathrm{test}} = [N_{m_{1}},N_{m_{2}},N_{m_{3}},N_{2},N_{3}] &{}\end{array}$$

(26)

First, the resource allocation is solved for a budget of $1000. There are 54 possible testing combinations and out of these 54, ten testing combinations lead to the same minimum variance of system-level output R, approximately 0.8 % of the variance before testing. These combinations are given in Table 8. The value of E(Var(R)) for these ten cases are close enough that it is not possible to determine whether the difference is due to reality or due to sampling/numerical errors.

Table 8 Resource allocation results: structural dynamics problem

It is a subjective decision as to which one of these ten test combinations is selected. However, all ten combinations unanimously suggest that no tests are needed for the overall system and one test is needed for the second level three spring-mass-damper subsystem. The first four rows in Table 8 suggest that testing is not needed for the first spring-mass-damper. However, it may be desirable to have at least one test for each component, and hence one amongst the latter six options may be preferred.

It was also found that an extra budget of $1,000 caused no further reduction in the variance of R. If the available budget is $2,000, a subjective decision may be made to conduct the full system test (which costs $1,000) in order to further improve the confidence in uncertainty quantification.

10 Conclusion

This chapter presented a Bayesian methodology for uncertainty quantification in complex engineering systems consisting of multiple physics behavior and multiple levels of integration. The various component, subsystem, and system models, and their inputs, parameters, and outputs, and experimental data were efficiently connected through a Bayesian network. Further, the various sources of uncertainty—physical variability, data uncertainty, and model uncertainty were also included in the Bayesian network. The Bayesian network was used for three different tasks: (1) calibrate the parameters of models at multiple levels using all available test data from multiple levels; (2) propagate the various sources of uncertainty (including the previously estimated model parameters) through the Bayes network to predict the overall uncertainty in the system-level response; and (3) aid in resource allocation for test selection, in order to identify the most effective tests to reduce the overall uncertainty in the system-level prediction. The procedure for test resource allocation required Bayesian calibration and assessment of system-level prediction uncertainty even before actual testing was performed. This was achieved by generating multiple samples of test data and estimating the expected reduction in variance of the system-level prediction.

The algorithm for test resource allocation leads to several insights. A lower level test can easily isolate individual components and hence, the model parameters can be effectively updated, leading to a significant reduction in the variance of the system-level prediction. However, such a test would not account for the interaction between the higher level models and the corresponding parameters. In contrast, a higher level test would readily include the effects of interaction between multiple subsystem-level and component-level models. However, the calibration of parameters across multiple models may be difficult and may not lead to a significant reduction in the variance of the system-level prediction. The optimization-based test resource allocation procedure trades off between lower level tests and higher level tests by accounting not only for the resultant reduction in variance of the system-level prediction but also for the costs involved in testing.