Achieving Robust Design through Statistical Effect Screening

Abstract

In the verification and validation of numerical models, statistical effect screening is useful to identify model parameters that influence predictions. Studies such as model calibration and uncertainty propagation can then be limited to the most influential variables, leading to potentially significant computational savings. Contrary to local, derivative-based sensitivity analysis, statistical effect screening assesses the influence on predictions exercised by individual variables, or combinations of variables, as they vary between their lower and upper bounds. Often, however, no formal distinction is made between design parameters and calibration variables. Design parameters are defined, here, as dimensions of the design space controlled by the analyst, while calibration variables are introduced by modeling choices. We argue that uncertainty introduced by design parameters should be treated differently from uncertainty originating from calibration variables, because it is of a different nature, and despite the fact that the two sets can interact to change the model predictions. This work proposes a method for effect screening to identify the influential design parameters while simultaneously being robust to the epistemic uncertainty of calibration variables. The objective is to screen out, that is, remove from analysis, the least influential design parameters while making sure that the ranking reached is not vulnerable to the uncertainty of calibration variables that might interact with design parameters. The robustness criterion guarantees the importance of design parameters within the design space, and irrespective of values taken by the calibration variables. The Morris one-at-a-time screening method is utilized to explore the design space and identify the most influential design parameters, while info-gap decision theory offers a criterion to guarantee that the sensitivity coefficients obtained are robust to the epistemic uncertainty introduced by calibration variables. The proposed method is applied to the NASA Multidisciplinary Uncertainty Quantification Challenge Problem, which is a black-box code for aeronautic flight guidance that requires a total of 21 calibration variables and 14 design parameters. The application demonstrates that a large number of variables can be handled without having to formulate any simplifying assumption about the potential coupling between calibration variables and design parameters. It further suggests that, when design optimization is focused on the most influential design parameters, as defined by our criterion of robustness to calibration variables, a better-performing design can be found. Because the computational cost of Morris sensitivity analysis only increases linearly with the number of variables, we conclude that the method proposed can be applied to even larger-dimensional problems. (Publication approved for unlimited, public release on October 9, 2013, LA-UR-13-27839, Unclassified.)

14.1 Introduction

Many applications in engineering analysis and design have benefited from the use of numerical models to supplement or replace the costly design-build-test paradigm. With the help of numerical simulations, the number of exploratory designs can be efficiently reduced down to the most feasible ones, which can then be proposed for experimental testing. For example, finite element (FE) models have proven useful to explore stress distributions generated from various shapes of dental implants [1], impact energy that vehicles can withstand due to rolling over [2], and material stacking arrangements in the structural design of a composite wind turbine blade [3]. As such, optimization techniques have been used to efficiently search for the design with the best performance. Here, “performance” is defined as the design criterion used to evaluate the extent to which the system analyzed meets its requirements. While it is important for these optimization techniques to identify designs that optimize performance, it has also been acknowledged that the design should be robust to uncertainties encountered in both the operating conditions of the system and development of the numerical model. For example, in the structural design of buildings, uncertainties can arise due to the difference between what is considered in design and what is experienced in implementation, such as idealized load cases compared to in-service loads and differences of the manufactured structure from design specifications [4]. The basic motivation for robust design is to improve the quality of a product by ensuring that target performance requirements are met even when the design parameters vary away from their best-case specifications [5].

Most techniques developed for simulation-based robust design optimize input parameters to determine the design that minimizes the standard deviation of performance and produces its most desirable mean value [6]. The most straightforward way to estimate statistics, such as the mean and standard deviation, is through Monte Carlo sampling of the probability distribution law of input parameters. Clearly, Monte Carlo simulations can be computationally expensive. In addition, these input probability distributions often time need to be assumed or approximated. To mitigate these limitations and, specifically, the need for unnecessary assumptions, one alternative is to evaluate the degradation of performance at bounded levels of uncertainty as the design deviates from its nominal setting. It is the approach that we adopt in this work, proposed in the context of info-gap decision theory (IGDT). IGDT defines a convenient framework, which describes the sources of uncertainty either probabilistically or not, to explore the trade-offs between performance and robustness [7].

Approaches to robust design can be computationally expensive due to the need to evaluate the model at multiple settings around its nominal definition. For high-dimensional and/or computationally expensive models, it is useful to screen, or remove from analysis, parameters that are of low influence to the model output. A powerful tool is to combine a design-of-experiments (DOE) with a screening analysis and, possibly, surrogate modeling [8]. The size of a DOE often scales exponentially, or faster, as the number of model parameters increases. For example, a full-factorial DOE defines discrete values, or “levels” for each parameter, and evaluates all possible combinations of parameters levels. Statistical effect screening, such as analysis of variance, can be used to determine parameters that most significantly change the numerical predictions. For large DOE, parallel computing is a convenient resource to decrease the time-to-solution of the entire DOE [9]. Another alternative is to pursue frameworks that require smaller numbers of model evaluations to understand the predictions. One such method is the Morris one-at-a-time (OAT) sensitivity analysis, which provides an effect screening whereby the DOE scales linearly as the number of model parameters increases. For high-dimensional problems that might depend on several dozen parameters, combining the Morris-based screening with parallel computing offers a real potential for significant computational savings.

One issue with statistical effect screenings for robust design is that parameters identified as influential are not guaranteed to be influential as uncertain variables are varied from their nominal setting. As discussed further in Sect. 14.2, this is particularly true in the case of multiple performance criteria, where compliance of the design pursued is evaluated against many different metrics that must each meet their requirement. For example, the design of automotive performance features multiple criteria, which typically include fuel efficiency, capacity, reliability, cost, maneuverability, vehicle weight and driving comfort [10]. When screening input parameters of numerical models, it is especially important to ensure that the parameters found to be most significant are influential robustly, rather than simply being influential. Robustness means, here, that results of the sensitivity analysis should remain unchanged by potential interactions between the set of input parameters analyzed and other algorithms or variables that, while they may be uncertain, are not the direct focus of the analysis.

Another issue is that sensitivity analysis often makes no formal distinction between design parameters and calibration variables. Design parameters are defined, here, as dimensions of the design space controlled by the analyst, while calibration variables are due to environmental conditions or introduced by modeling choices. For example, design parameters of an aerodynamic simulation of flight test include the flow velocity and pitch angle of the object tested. Calibration variables, on the other hand, might include the air temperature and variables such as artificial viscosity settings of the fluid dynamics solver and structural damping coefficients of a finite element model. The first one (air temperature) is a calibration variable because its value is highly uncertain in the real-world application. The others (artificial viscosity and structural damping) are calibration variables because they originate from specific modeling choices. While a screening analysis seeks to identify which one of the design parameters (flow velocity or pitch angle) most influences the model predictions, it needs to do so in a manner that is robust to our ignorance of calibration variables.

Our point-of-view is that the design parameter uncertainty should be treated differently from uncertainty originating from calibration variables, despite the fact that interactions between the two sets might change the predictions. A different treatment is justified because the two types of uncertainties are fundamentally different. Design parameters are uncertain because many choices are possible; however, once a design is selected they become known exactly. Uncertainty originating from calibration variables, on the other hand, should be propagated through the simulation and accounted for in a parameter study, sensitivity analysis, or design optimization. The fundamental difference between design parameters and calibration variables is addressed, here, by embedding an assessment of robustness with respect to calibration variables within the sensitivity analysis of design parameters.

This manuscript proposes an approach to sensitivity analysis to determine design parameters that most significantly influence the performance while, simultaneously, being robust to uncertainty originating from calibration variables. The novelty is that the analysis focuses on the robustness of predictions rather than performance. No assumption is formulated about probability laws that describe the uncertainty of either design parameters or calibration variables. No assumption is made to simplify the potential interactions between design parameters and calibration variables. Herein, robustness is quantified in the framework of IGDT, making it amenable to cases where uncertainty stems from ignorance, variability, vagueness, or conflict. Sensitivity analysis is performed using the Morris OAT method, where the number of model evaluations scales linearly as the number of design parameters is increased. It provides a convenient alternative for sensitivity analysis when considering models of high dimensionality. To refine the goals of the methodology, Sect. 14.2 provides a hypothetical two-dimensional example to illustrate the difference between performance-optimal and robust-optimal designs. Brief overviews of IGDT, Morris OAT sensitivity analysis, and how they are integrated to formulate the methodology herein, are presented in Sect. 14.3. Application to the NASA Multidisciplinary Uncertainty Quantification Challenge Problem is discussed in Sect. 14.4. Section 14.5 provides the main conclusions and avenues for future work.

14.2 Discussion of Performance-Optimal and Robust-Optimal Designs for Multi-Criteria Simulations

Numerical models are often dependent on design parameters, p, and calibration variables, θ. Here, design parameters are those that the analyst has control over, and can be set during the physical experiments. Calibration variables are input parameters of the numerical simulation whose values are uncertain either because they represent environmental or manufacturing variability or because they are introduced by arbitrary numerical choices. This is shown in Fig. 14.1, where the outputs of interest, g, are obtained by inputting design parameters, p, and uncertain calibration variables, θ, to the system model. Note that p, θ and g can either be scalar or vector quantities.

Fig. 14.1

Mapping of model inputs to model outputs

Equation 14.1 is companion to Fig. 14.1, where the k model outputs, g, depend on the pair of model inputs (p; θ):

$$ {g}_k=f\left(p;\theta \right) $$

(14.1)

Herein, the model is regarded as requirement compliant if the outputs satisfy inequalities such as g < g_C, where g_C expresses critical, user-defined requirements of performance. For a fixed set of design parameters p, the set of calibration variables θ that produces the condition g < g_C is called the safe domain, and the set of calibration variables that produces g ≥ g_C is the failure domain. The goal is to find the “best” design parameters that minimize the probability of failure, which is the probability of performance integrated over the failure region of calibration variables.

It is our contention that two paradigms exist to determine the “best” design: (1) the performance-optimal paradigm, and (2) the robust-optimal paradigm. These two paradigms carry over to sensitivity analysis. Next, a conceptual two-dimensional example illustrates the difference between performance-optimal and robust-optimal approaches.

Suppose that a system model provides two outputs, g₁ and g₂, as depicted in Fig. 14.2. The function g₁ is dependent on the pair of inputs (p₁; θ) and the function g₂ depends on the pair (p₂; θ), where design parameters p₁ and p₂ are different. For simplicity, Fig. 14.2 depicts the functions g₁ and g₂ using a constant, baseline value of the calibration variable, that is, θ = θ₀. It is assumed that the design parameters p₁ and p₂ are represented on the same scale (x-axis). The shaded region depicts the failure domain where the outputs g₁ and g₂ are unable to satisfy the inequality g_k < g_C. It is further assumed that the maximum output value is used to define the system performance and determine if the output predicted by the numerical simulation is in the safe or failure domain:

Fig. 14.2

Nominal model outputs g₁ and g₂ at a calibration variable “θ = θ₀ = constant”

$$ {g}_{Max}= max\left({g}_1;{g}_2\right) $$

(14.2)

The conceptual illustration in Fig. 14.2 is drawn such that the design parameter p₂ is overall more influential than the first parameter p₁. This is because the function g₂ (blue solid line) varies over a wider range than g₁ (green solid line) as p₁ and p₂ are changed from their lower bounds to their upper bounds. A screening analysis would typically rank p₂ as the most influential design parameter, followed by p₁. This ranking is unchanged whether one analyzes the entire design space or only the requirement-compliant designs, shown in Fig. 14.2 as the un-shaded region.

Figure 14.3 depicts the system performance, or maximum output g_Max, over the entire range of design parameters. It shows that either function g₁ or g₂ defines performance, depending on which one is greater in Eq. 14.2. When considering the entire design space (safe and failure regions combined), it is clear that the design parameter p₂ is more influential because function g₂ exhibits a larger variation as p₂ varies than the variation of function g₁ as p₁ varies. A statistical effect screening that considers only the requirement-compliant region would find that the design parameter p₂ influences performance more than its counterpart p₁. It is also likely that the statistical effect screening would identify an interaction coupling between p₁ and p₂, as evidenced in Fig. 14.3 by the fact that the performance (maximum value) is sometimes defined by function g₁ and other times by function g₂.

Fig. 14.3

System performance, as defined in Eq. 14.2, and requirement-compliant region

A first observation, therefore, is that it is important to “filter out” design points located in the failure region such that sensitivity analysis can be restricted to the requirement-compliant, or safe, region. Not doing so offers the danger of identifying design parameters as significant when, in fact, they inherit their influence from the failure region. When attempting to improve a design, computational resources should not be wasted to explore parameter variations in regions where designs are not sought in the first place.

Figure 14.3 also illustrates what is meant by performance-optimal. It is assumed that “better” designs correspond to smaller values of the performance metric defined in Eq. 14.2. By definition, the performance-optimal design is the one that, while located within the safe region, minimizes g_Max. This is formulated as a conventional optimization over the space of design parameters. The point labeled p^(Optimal) in Fig. 14.3 indicates the smallest value of g_Max.

Now consider that one wishes to account for uncertainty associated with the calibration parameter. For example, the nominal calibration variable, θ = θ₀, can be varied to a different setting, $\uptheta =\widehat{\uptheta}$. How these variations are chosen will be discussed in more depth in Sect. 14.3.1. The performance, that was originally requirement-compliant, might degrade as the calibration parameter is allowed to vary, or it might improve. A degradation of performance is illustrated in Fig. 14.4 by the dashed lines that represent, on the left side, the output function g₁ at $\uptheta =\widehat{\uptheta}$ and, on the right side, the output g₂ at the same setting of $\uptheta =\widehat{\uptheta}$.

Fig. 14.4

Effect of output functions g₁ and g₂ of varying the calibration variable θ

Figure 14.5 compares the performance functions, g_Max, at the nominal setting θ = θ₀ of the calibration variable (solid line) and modified setting $\uptheta =\widehat{\uptheta}$ (dashed line). For the modified value of the calibration variable, notice that the maximum output is now dominated by function ĝ₁, whereas at the nominal setting θ = θ₀ the output was dominated by function g₂. The modified performance features a minimum value, labeled p^(Robust) in Fig. 14.5. This design point, however, does not offer as much performance as our nominal solution p^(Optimal). An unwanted consequence of p^(Optimal), however, is that it can enter the failure domain once the calibration variable deviates from its nominal setting, whereas p^(Robust) is formulated to account for the change in the calibration variable. Also observed in Fig. 14.5 is a visual comparison of the difference, Δp, between p^(Optimal) and p^(Robust).

Fig. 14.5

Selection of a design solution that is robust to the calibration variable θ

This approach, that searches for the best-possible performance as uncertainty introduced by calibration variables is exercised, leads to the concept of robust-optimal design. In the example, the robust-optimal design is pursued when the calibration variable is varied from θ = θ₀ to $\uptheta =\widehat{\uptheta}$, but is sub-optimal when compared to the performance-optimal solution, as illustrated in Fig. 14.5:

$$ {g}_{Max}\left({p^{(Robust)}}_{;}{\theta}_0\right)\ge {g}_{Max}\left({p^{(Optimal)}}_{;}{\theta}_0\right) $$

(14.3)

Selecting the design point p^(Robust) means that performance predicted by the numerical model remains within the safe domain as the calibration variable changes from θ = θ₀ to $\uptheta =\widehat{\uptheta}$, or vice-versa. The performance-optimal solution, on the other hand, offers the risk that performance might not be requirement-compliant when the calibration variable is modified. The concept can easily be generalized to explore the variation of several calibration variables within a space denoted by U, further defined in Sect. 14.3.1. Then, the robust-optimal paradigm searches for the worst-case performance across all designs that, while being requirement-compliant, give the best-possible performance over the uncertainty space U of calibration variables. Mathematically, it requires two embedded optimization problems:

$$ {p}^{(Robust)}=\underset{\mathrm{Compliant}\kern1em \mathrm{Designs},\kern0.5em p}{ \arg \kern0.5em \max}\left\{\underset{\theta \kern0.5em \in \kern0.5em U}{\kern0.5em \min }{g}_{\max },\kern0.5em \mathrm{such}\kern0.5em \mathrm{that}\kern0.5em {g}_{\max}\le {g}_C\right\} $$

(14.4)

where the inner optimization searches for the best-possible performance at a fixed design and over the calibration space U, and the outer optimization explores the worst-possible case over all designs.

Now that we have stressed the difference between performance-optimal and robust-optimal designs, we go back to the discussion of sensitivity analysis to conclude with a second observation. Just like a performance-optimal design offers the risk of not delivering the expected performance due to uncertain calibration variables, a sensitivity ranking of design parameters is vulnerable to the same uncertainty, especially if the calibration variables interact with design parameters to significantly change the simulation predictions. In our conceptual example, the function g₂ dominates performance at the nominal setting of calibration variable (recall Fig. 14.3). Thus, the design parameter p₂ would be found more influential, and selected for design optimization. However, when the value of the calibration variable is changed, the reverse occurs: the modified function ĝ₁ dominates performance (recall Fig. 14.5), leading to design parameter p₁ being ranked number one in sensitivity. Clearly, sensitivity analysis cannot be executed at every possible combination of calibration variables to identify these potential influence-ranking reversals. To deal with this challenge, we propose the concept of robust-optimal sensitivity where a robustness criterion, much like the one conceptually illustrated in Eq. 14.4, is embedded within the sensitivity analysis.

14.3 Sensitivity Analysis Using the Robustness Performance

The methodology proposed to screen for design parameters that are influential using a robust-optimal criterion is introduced next. First, Sect. 14.3.1 provides an overview of IGDT, which is the criterion used herein to quantify robustness. Next, Sect. 14.3.2 introduces concepts of the Morris OAT sensitivity analysis. Using the Morris method is particularly appealing because the size of its DOE scales linearly with the number of design parameters, rather than other screening techniques that rely on information volumes that increase exponentially, or faster. Section 14.3.3 discusses how Morris and IGDT are integrated together. An application to the NASA Multidisciplinary Uncertainty Quantification Challenge Problem is given in Sect. 14.4.

14.3.1 Info-Gap Decision Theory for Robustness Analysis

This section provides a brief overview of IGDT, to keep the discussion self-contained as much as possible. Further details of the theory and its implementation can be found in [7]. Supporting a decision using IGDT, irrespective of the discipline or application, requires the combination of three attributes. The first attribute is a model, the second one is a performance criterion, and the third one is a representation of uncertainty using an info-gap model. The three attributes are introduced to explain how decision-making is formulated for our application.

The first two attributes, model and performance criterion, are evoked in the conceptual example of Sect. 14.2. The model can be an analytic formula, numerical function, or black-box software that depends on design parameters, p, and calibration variables, θ, such as illustrated in Fig. 14.1. The performance is a criterion, such as Eq. 14.2, that converts the multi-dimensional model output to a scalar quantity, herein denoted by the symbol R. In Sect. 14.2, R = g_Max is used to define performance. As mentioned earlier, the model is said to be requirement-compliant when the inequality R < R_C is satisfied where R_C is a user-defined, critical requirement of performance.

The third attribute, that warrants more explanation, is the info-gap representation of uncertainty. We postulate that the uncertainty, that we wish to be robust to, originates from the calibration variables. Therefore, the info-gap model applies to the calibration variables. This is not a restrictive assumption since robustness analysis can be applied to other aspects of a simulation that introduce uncertainty or arbitrary assumptions. For example, Van Buren et al. [11] study the extent to which the predictions of wind turbine dynamics are robust to model-form assumption.

The info-gap model, U(α; θ₀), describes how the uncertain calibration variables, θ, vary from their nominal values or settings, θ = θ₀. The nominal values, θ₀, denote the best-available knowledge used as starting point of the analysis. The “distance” between these nominal values, θ = θ₀ and settings $\uptheta =\widehat{\uptheta}$, that we wish to use in the simulation, is defined using a single scalar α called the horizon-of-uncertainty. For example, an info-gap model can be written as:

$$ U\left(\alpha; {\theta}_0\right)=\left\{\widehat{\theta}:\left|\widehat{\theta}-{\theta}_0\right|\le \alpha \right\},\alpha \ge 0 $$

(14.5)

When α = 0, the calibration variables remain fixed at their nominal values, θ = θ₀. Equation 14.5 shows that, as the horizon-of-uncertainty increases, the calibration variables are allowed to venture further and further away from their nominal values to the settings of $\uptheta =\widehat{\uptheta}$. Each vector of values $\uptheta =\widehat{\uptheta}$ considered leads to a different model whose prediction of performance is evaluated against the criterion R < R_C for requirement compliance.

An info-gap analysis of robustness searches for the model that is able to tolerate as much horizon-of-uncertainty α as possible, while meeting the performance requirement R_C. Such a model is robust-optimal because, at a minimum, it delivers the expected level of performance R_C while accepting the largest uncertainty in the calibration variables, as quantified by Eq. 14.5. Mathematically, the robustness can be written as:

$$ \widehat{\alpha}=\underset{\alpha \ge 0}{ \arg \max}\left\{\underset{\widehat{\theta}\in U\left(\alpha; {\theta}_0\right)}{ \max }R\left(p;\widehat{\theta}\right)\le {R}_C\right\} $$

(14.6)

which expresses that robustness $\widehat{\upalpha}$ is the greatest horizon-of-uncertainty α for which the performance requirement is satisfied. Equation 14.6 indicates that estimating the robustness $\widehat{\upalpha}$ involves two nested optimization problems, which is computationally challenging. For their application to wind turbine dynamics, Van Buren et al. [11] implement an algorithm that bypasses this difficulty by focusing on the inner optimization. The reader is referred to this work for further description of the numerical implementation of info-gap robustness.

Equation 14.6 defines the robustness of a model, $\widehat{\upalpha}$, given a performance requirement R_C. Changing the requirement R_C, and repeating the info-gap analysis for several values of R_C, leads to the exploration of the robustness function. Conceptually, the robustness function quantifies the degradation of performance as the horizon-of-uncertainty α increases. Two such functions are plotted in Fig. 14.6 to conceptually compare the robust-optimal and performance-optimal designs of Fig. 14.5. Note that, in Fig. 14.6, performance is shown on the x-axis, thus the failure domain is located horizontally on the right side of the graph, rather than the top portion, as shown previously in Figs. 14.2, 14.3, 14.4, and 14.5.

Fig. 14.6

Comparison of info-gap robustness functions $\widehat{\upalpha}$-versus-R

At the nominal calibration variables, where α = 0, the performance-optimal design provides the best performance. However, as α increases and the uncertain calibration variables are allowed to deviate from their nominal settings, the performance degrades. This degradation is a direct consequence of Eq. 14.6 where, by definition, the robust solution is the worst possible performance over the space U(α; θ₀) of calibration variables. As α increases, U(α; θ₀) includes more and more possibilities and the worst case can only get worse. It leads to the increase of R-values and deterioration of performance illustrated in Fig. 14.6. The figure suggests a number of scenarios. Designs might cross into the failure domain, as more robustness to uncertainty is required. Another situation is that the performance-optimal design (blue solid curve), initially better than the robust-optimal design (green solid curve), degrades faster as α increases. The implication is that, at high levels of uncertainty about the values of calibration variables, the robust-optimal design outperforms the performance-optimal design. Hence, the robustness function maps the trade-offs between performance (on the horizontal axis) and uncertainty (on the vertical axis), which is the information needed to decide if a particular design should be adopted.

14.3.2 Morris One-at-a-Time (OAT) Screening for Sensitivity Analysis

The Morris OAT screening method is a global sensitivity analysis that is particularly attractive to evaluate models that feature many variables or are computationally expensive to execute. The method was first introduced in [12] and later refined [13], and is based on evaluating the performance of “trajectories” within the parameter space of the numerical model. Screening is achieved by ranking the variables according to their influence coefficients. For completeness, a brief overview is provided; the reader is referred to [12–14] for details.

The Morris OAT method samples variables by constructing “trajectories” where one variable is changed at a time. Figure 14.7 illustrates two trajectories in 2D space. For the first trajectory, shown in color blue, a starting point (p₁; p₂) is selected randomly and defines the first execution of the model. A direction p_k is then selected at random, and perturbed by an amount Δp_k. For the first trajectory, this first perturbation is applied to variable p₂. The second run of the model is performed at the setting (p₁; p₂ + Δp₂). The third run is performed at setting (p₁ + Δp₁; p₂ + Δp₂), which completes the trajectory since all variables have now been sampled once. In contrast, the second trajectory shown in color green is perturbed first in the positive p₁-direction, and second in the negative p₂-direction.

Fig. 14.7

Examples of Morris OAT trajectories in two dimensions

Generalizing to D variables (p₁; p₂; … p_D), a trajectory is obtained with (D + 1) simulation runs by defining the starting point of the (k + 1)^th perturbation as the end-point of the kth perturbation. Campolongo et al. [13] recommends constructing between 10 and 50 trajectories. After studying the influence of the number of trajectories for exploratory, “first-pass” effect screening, Hemez [14] suggests ≈50 trajectories. The total number of simulation runs needed to evaluate the resulting Morris OAT design is N · (D + 1), where N is the number of trajectories. The reader is referred to [14] for the discussion of implementation details, such as how to initialize the trajectories, how to select an appropriate perturbation size, Δp_k, or how to handle values (p_k + Δp_k) that might fall outside the lower and upper bounds of the variable.

Once the Morris trajectories are defined, the simulation code is analyzed at these points, and predictions are used to calculate the elementary effect (EE) of perturbing each variable independently as:

$$ E{E}_k^{(r)}=\left|\frac{f\left({p}_1^{(r)},\dots, {p}_k^{(r)}+\varDelta {p}_k^{(r)},\dots, {p}_D^{(r)}\right)-f\left({p}_1^{(r)},\dots, {p}_k^{(r)},\dots, {p}_D^{(r)}\right)}{\Delta {p}_k^{(r)}}\right| $$

(14.7)

In Eq. 14.7, the kth variable is perturbed by an amount Δp_k ^(r) and the other variables are kept unchanged. The notation EE_k ^(r) uses a subscript (●)_k that indicates the variable perturbed and a superscript (●)^(r) that identifies the trajectory, 1 ≤ r ≤ N. The value of EE_k ^(r) indicates the effect on the prediction of varying one variable at a time.

When applied to a nonlinear function, the value of the kth effect changes depending on which point (p₁; p₂; … p_D) is analyzed. To get a consistent picture of which main effect p_k, linear interaction p_p · p_q, or higher-order effect p_k ², p_k ³, etc., are statistically significant, the calculation of values EE_k ^(r) is repeated for multiple trajectories, and statistics are estimated. We use the mean, μ_k(EE), and standard deviation, σ_k(EE), statistics estimated as:

$$ {\mu}_k(EE)=\frac{1}{N}\cdot {\displaystyle \sum_{1\le r\le N}E{E}_k^{(r)}} $$

(14.8)

and:

$$ {\sigma}_k(EE)=\sqrt{\frac{1}{N-1}\cdot {\displaystyle \sum_{1\le r\le N}{\left(E{E}_k^{(r)}-{\mu}_k(EE)\right)}^2}} $$

(14.9)

Values of statistics μ_k(EE) and σ_k(EE) can then be plotted against each other to identify the significant variables. The mean, μ_k, assesses the overall influence of the kth factor on the simulation output, whereas the standard deviation, σ_k, estimates the higher order effects of the kth factor [12, 13]. Higher order effects include those that are nonlinear or involved in interactions with other factors. When both μ_k(EE) and σ_k(EE) are “small,” the corresponding variable p_k is not statistically significant to explain how predictions change. A large value of μ_k(EE), paired with a small value of σ_k(EE), indicates a significant main effect while linear interactions involving variable p_k are not important. “Large” statistics μ_k(EE) and σ_k(EE) indicate significant, linear or nonlinear, effects.

In Sect. 14.4, the influence coefficients μ_k(EE) and σ_k(EE) are used to rank the design parameters p_k for the influence that they exercise on performance. The goal is not to get an “exact” ranking; it is, rather, to find a small number of strongly influential design parameters, such that design optimization can focus on these dimensions for efficiency. As we have seen, the Morris screening method scales the number of model evaluations linearly with the number of design parameters. Also, the user controls the number of trajectories, N. These properties are attractive for high-dimensional problems, or simulations that require significant computational resource. Another advantage is that different trajectories can be computed in parallel to take advantage of parallel or massively serial architectures. The methodology proposed, hence, develops a computationally efficient solution for problems with dozens of variables.

14.3.3 Integration of IGDT Robustness and Morris OAT Sensitivity Analysis

Having provided an overview of IGDT and Morris OAT sensitivity, this section discusses the framework proposed for screening design parameters using a robustness criterion. The first step, as outlined in the flowchart in Fig. 14.8, is to develop the Morris DOE and “filter out” trajectories that are not requirement-compliant (recall the discussion of Fig. 14.3). Doing so ensures that computational resources are focused on the design space of interest. In the flowchart, the first box generates a Morris OAT design composed on N trajectories. In the second box, the DOE is evaluated using the simulation model to determine trajectories that are requirement-compliant. Trajectories with predictions of performance located in the failure domain are disregarded, reducing the number of trajectories available for sensitivity analysis from N to only those that are compliant, denoted by NC.

Fig. 14.8

Flowchart of the procedure to “filter out” designs for the compliant Morris DOE

The compliant Morris DOE with NC trajectories is used next to rank the design parameters. Figure 14.9 compares the strategy where the ranking is determined using the influence that parameters exercise on performance (left side), and the alternate strategy where ranking is based on robustness to calibration variable uncertainty (right side).

Fig. 14.9

Comparison of two strategies for design parameter ranking and screening. (a) Left: Performance-based screening. (b) Right: Robustness-based screening

On the left side, the performance, g = g(p;θ₀), is obtained with calibration variables set to their nominal settings, θ = θ₀. In contrast, the right side of Fig. 14.9 proposes a strategy where model robustness is first determined by solving Eq. 14.6, as outlined in Sect. 14.3.1, at each design point of the Morris trajectories. The robust-optimal solution gives the values of calibration variables, $\uptheta =\widehat{\uptheta}$, that define the performance, ĝ = g(p;$\widehat{\uptheta}$), used for sensitivity analysis. In short, the main difference between the two approaches is the set of calibration variables, performance-optimal (left side, θ = θ₀) or robust-optimal (right side, $\uptheta =\widehat{\uptheta}$), used to evaluate the model performance.

How much the calibration variables are allowed to vary from their nominal settings is determined in an info-gap context by the horizon-of-uncertainty, α. In other words, the robust-optimal solution, ĝ = g(p;$\widehat{\uptheta}$), is searched within an uncertainty space written conceptually as “||θ − θ₀|| ≤ α.” Thus, the robustness can be evaluated for increasing levels of horizon-of-uncertainty, α, as deemed necessary by the analyst. These successive evaluations of robustness are indicated in Fig. 14.9b by a vertical arrow that loops over the horizon-of-uncertainty, α. Once the predictions of performance are obtained for each Morris trajectory, performance-optimal (left side, g = g(p;θ₀)) or robust-optimal (right side, ĝ = g(p;$\widehat{\uptheta}$)), the elementary effects are calculated, followed by the statistics defined in Eqs. 14.8 and 14.9.

The integration of info-gap robustness to Morris OAT trajectories is further illustrated in Fig. 14.10. An info-gap analysis is performed at each design point of Morris trajectories to determine the robust-optimal performance, ĝ. The trajectory is disregarded if any of its design points violates requirement-compliance (see Fig. 14.8). At any level of horizon-of-uncertainty considered, α_i, the robust performance is searched for using genetic algorithm optimization. As indicated in the flowchart of Fig. 14.9, these predictions of robust performance are retained for each design of the DOE to calculate the elementary effects and Morris statistics.

Fig. 14.10

Integration of info-gap robustness to Morris OAT trajectories for sensitivity analysis

14.4 Application to the NASA Multidisciplinary Uncertainty Quantification Challenge Problem

The framework established in Sect. 14.3 is applied to the NASA Multidisciplinary Uncertainty Quantification Challenge (MUQC) problem, which is a black-box problem of a realistic aeronautics application. Section 14.4.1 briefly introduces the MUQC problem formulation and some of the assumptions that are imposed to the problem. Section 14.4.2 compares the rankings of design parameters obtained with the performance-optimal and robust-optimal criteria. To demonstrate the advantage that the robust-optimal ranking can offer over the other ranking, parameters identified as the most influential ones are used to optimize the design, and an unexpected result is discussed in Sect. 14.4.3.

14.4.1 The NASA Langley MUQC Problem

The NASA MUQC problem is formulated as a discipline independent black-box mathematical model to pose challenges to the uncertainty quantification community. The mathematical model produces eight outputs, denoted using a vector, g. The mathematical model is slightly different from the system model described in Eq. 14.1. Here, the output, g, depends on 5 intermediate variables, x, and 14 design parameters, p. The intermediate variables, x, are sub-models that depend on 21 uncertain calibration variables, θ, expressed as follows:

$$ x=h\left(\theta \right),\kern2em g=f\left(p;x\right) $$

(14.10)

Figure 14.11 provides a visual representation of how the 14 design parameters, p, 21 calibration variables, θ, and 5 intermediate variables, x, are combined to predict the 8 performance outputs, g. Inputs are the design parameters, p, and calibration variables, θ, as denoted by the green dashed box. The calibration variables, θ, are used to predict the intermediate variables, x, as denoted by the solid blue box. The design parameters, p, and intermediate variables, x, are used to predict the performance outputs, g. The simulation is considered requirement-compliant when all outputs fulfill the inequality, g_k < 0 for k = 1 … 8. Model performance is, as before, defined as g_Max = max(g_k). The different models of Fig. 14.11 are implemented in a black-box Matlab™ code that executes in a fraction of second.

Fig. 14.11

Mapping of model inputs to model outputs for the NASA MUQC problem

When the design parameters are kept fixed, Eq. 14.10 is defined in a space of dimension 21. The support of all 21 uncertain calibration variables is the interval [0; +1], with exception of variables θ₄ and θ₅ that are described using normal probability laws with expected values between [−5; +5]. The design parameters, p, are controller gains that are not physical quantities. Thus, they are not subjected to restrictions and can assume any sign and any finite value. For the analysis herein, a baseline set of design parameters and calibration variables, provided by the MUQC problem moderators at NASA, is utilized.

Figure 14.11 indicates that the intermediate variables, x, and calibration variables, θ, are somewhat decoupled. For example, variable x₁ is not dependent on the same inputs as x₂, x₃, x₄, or x₅. A Latin Hypercube exploration of calibration variables is performed to ensure that the intermediate variables are continuous functions of their inputs. This is confirmed by the histograms of sampled values shown in Fig. 14.12. Establishing continuity is important to define the lower and upper bounds of intermediate variables, x, used for the analysis of info-gap robustness.

Fig. 14.12

Empirical histograms of intermediate variable outputs, x. (a) Histogram of x₁-values. (b) Histogram of x₂-values. (c) Histogram of x₃-values. (d) Histogram of x₄-values. (e) Histogram of x₅-values

Based on the observation that the intermediate variables vary continuously between their lower and upper bounds in Fig. 14.12, it is decided to assess the robustness of model predictions, g, to uncertainty in intermediate variables, x, rather than robustness to uncertainty in calibration variables, θ. A significant advantage is to reduce the exploration from 21 dimensions to a five-dimensional space. The main drawback of this choice is to ignore that some values of the intermediate variables are more likely than others, as suggested by the histograms of Fig. 14.12. This approach is justified because our goal is to assess the robustness of performance outputs, not to sample their probability laws.

14.4.2 Parameter Ranking Using the Morris Elemental Effect Statistics

The Morris sensitivity analysis of 14 design parameters, p, is first performed with calibration variables held constant at their nominal settings. A total of 1,000 trajectories are constructed to explore the design space defined by ±40 % variation relative to the baseline design parameters. The starting points of trajectories are determined from a Latin Hypercube design-of-experiments. Trajectories are then constructed by moving Δp_k/p_k = 10 %, one parameter at a time, with the parameter and its direction chosen at random. Of the 1,000 trajectories, 388 provide simulation runs that are requirement-compliant for each point within the trajectory. These 388 trajectories define a compliant Morris DOE based on more than the ≈50 trajectories suggested for sensitivity analysis by previous studies.

After completing the model runs, Morris statistics of elemental effects are estimated as shown in Eqs. 14.8 and 14.9. The resulting statistics are plotted in Fig. 14.13a (left side), with mean sensitivity coefficients, μ_k(EE), shown on the horizontal x-axis and standard deviation sensitivity coefficients, σ_k(EE), shown on the vertical y-axis. Figure 14.13a represents the performance-optimal sensitivity analysis where calibration variables are equal to the baseline settings, θ = θ₀. Based on μ_k(EE) only, the three most influential design parameters are (p₁₂; p₁₃; p₁₁). It can be observed that a similar ranking is reached with a composite sensitivity metric, such as the distance-to-origin μ_k ²(EE) + σ_k ²(EE). The large standard deviation of parameter p₉ suggests a coupling or higher-order interactions with the other parameters.

Fig. 14.13

Comparison of mean and standard deviation statistics for Morris sensitivity analysis. (a) Left: Performance-optimal sensitivity analysis. (b) Right: Robust-optimal sensitivity analysis

Next, the design parameter ranking obtained with the performance-optimal criterion is contrasted with the ranking provided by the robust-optimal criterion. The 388 requirement-compliant trajectories are explored through info-gap analysis, as illustrated in Fig. 14.10. It means that the performance values used for sensitivity analysis are ĝ = g(p;$\widehat{\uptheta}$), where the calibration parameters, $\uptheta =\widehat{\uptheta}$, are those of the robust-optimal solution at a given design point, p. Results are shown in Fig. 14.13b (right side) at the horizon-of-uncertainty of α = 1, which corresponds to ±27 % maximum deviation of calibration variables from their nominal settings. (For variables θ₁, θ₁₃, θ₁₄, and θ₁₆, the level of α = 1 corresponds to their upper bounds.) Comparing Fig. 14.13a and b clearly shows that the most influential design parameters are different when the system performance is predicted using the robust-optimal calibration variables.

Table 14.1 reports the design parameter rankings that contribute to 90 %, or more, of the sensitivity influence on model performance. These rankings are based on the normalized μ_k(EE) values listed in columns three to four for the performance-optimal criterion and columns six to seven for the robust-optimal criterion. Three of the four parameters identified to be most influential with the robust-optimal criterion, p₁, p₃, and p₆, are not influential with the other criterion. This suggests that, for this application, considering the performance predicted with the nominal calibration variables, θ = θ₀, might lead to an inadequate screening of design parameters for robust-design optimization.

Table 14.1 Design parameter ranking using Morris statistics with the performance and robustness criteria

Table 14.2 reports which one of the eight outputs, g_k, predicted in the 388 requirement-compliant trajectories, defines the maximum performance used for robust-optimal sensitivity analysis. Each row of the table corresponds to a given horizon-of-uncertainty, α. At the nominal settings (first row, α = 0) where the calibration variables are held constant, θ = θ₀, the maximum value is controlled by the seventh performance output, g₇, for 5,706 of 5,820 total evaluations, or 98 % of the time. As calibration variables are allowed to vary, the design parameters controlling the maximum value start to change. Notably, the fourth performance output, g₄, increasingly influences the maximum value. At the horizon-of-uncertainty of α = 1, the percentage of maximum values controlled by the fourth performance output is ≈82 % (or 4,761 of 5,820 evaluations). It is observed that these trends are not monotonic, as indicated by statistics at α = 0.8 where the performance outputs g₄-to-g₇ exhibit comparable magnitudes. This change in behavior, where g₇ is most influential for the baseline model (α = 0) and g₄ is most influential for the robust model (α = 1), affects the influence exercised by design parameters, and is a likely explanation for the change in parameter ranking observed in Table 14.1.

Table 14.2 Frequencies of occurrence of the maximum output for each horizon-of-uncertainty level

14.4.3 Development of Performance-Optimal and Robust-Optimal Designs

Contrasting, in the previous section, two strategies to rank the design parameters suggests that not accounting for the robustness to calibration variable uncertainty can provide an incorrect identification of the most influential parameters. Even though this is the main point of the publication, one may ask: “so what?” To illustrate the impact that it may have, we proceed to optimize the design parameters to search for a better-than-nominal performance.

First, the six design parameters identified as most influential in Table 14.1 are optimized to search for the best-possible performance while the calibration variables are kept constant, θ = θ₀. The non-influential design parameters are kept constant and also equal to their nominal values. Reducing the dimensionality from 14 potential parameters to only six greatly improves the efficiency of the genetic algorithm used for optimization. This first step yields two designs referred to as “performance-optimal” and “robust-optimal,” depending on which six parameters are optimized.

These two “optimal” designs can be compared to the baseline. The predictions of performance are −1.62 × 10⁻⁴ for the baseline, −1.90 × 10⁻⁴ for the performance-optimal design, and −1.36 × 10⁻⁴ for the robust-optimal design. While the performance-optimal design improves on the baseline by +17 %, the robust-optimal design performs −16 % worse. One cannot, however, confirm or disregard a particular design based on this observation alone because it does not quantify what happens to system performance when the calibration variable uncertainty is accounted for.

The next step is to examine the robustness to calibration variable uncertainty in an info-gap sense. The maximum level of uncertainty, with which the calibration variables can deviate from their nominal settings, is progressively increased, and the procedure outlined in Sect. 14.3.1 searches for the worst performance. Figure 14.14 compares the three robustness functions obtained. The curves indicate the level of calibration variable uncertainty (on the vertical axis) that can be tolerated until the design enters the failure domain, as indicated by the grey shaded areas.

Fig. 14.14

Comparison of info-gap robustness functions obtained with two Morris sensitivity strategies. (a) Left: Analysis of performance-optimal ranking. (b) Right: Analysis of robust-optimal ranking

Figure 14.14a (left side) shows the robustness of the performance-optimal design, indicated with a red dashed line. The fact that an improved design is found, at least initially at α = 0, demonstrates the ability of the Morris sensitivity analysis to identify a subset of design parameters that do exercise some influence on the performance. However, this design tolerates little calibration variable uncertainty. Only 8.7 % variability suffices to violate the requirement compliance, whereas the baseline design can tolerate up to 22.2 % variation. This rapid deterioration of performance is a direct consequence of the fact that the design analyzed is optimized for performance, not robustness.

Similar results are presented in Fig. 14.14b (right side) for the robust-optimal design. For conciseness, details of the numerical implementation are not discussed. Even for this “toy” problem, the computational cost of constructing the robustness function is large enough that 12 workstation processors are used for parallel processing, to speed-up the time-to-solution. Optimization using the robustness criteria takes 2 days of simulation time, approximately.

Figure 14.14b illustrates the robustness of the robust-optimal design (green dashed line). The robust-optimal design provides slightly less performance than the nominal design, but the gain in robustness is evident. In fact, the robust-optimal design never violates the requirement, even with up to 27 % calibration variable uncertainty (at α = 1). This ability to maintain performance, despite significant calibration variable uncertainty, is a consequence of selecting an appropriate subset of influential design parameters using our robust-optimal screening criterion in Sect. 14.4.2.

Table 14.3 summarizes the results by comparing the initial performances of the three design strategies (at α = 0), and their ability to sustain uncertainty until the requirement-compliant criterion is violated. The performance-optimal design improves on the baseline design is terms of performance. The price to pay, however, is a low tolerance to calibration variable uncertainty. We contend that a better design strategy is to seek robustness. Table 14.3 shows that the robust-optimal design tolerates more uncertainty at the cost of a slightly lower performance. It is a better strategy because the resulting design is guaranteed to be requirement-compliant, even when faced with uncertain calibration.

Table 14.3 Comparison of baseline, performance-optimal, and robust-optimal designs

14.5 Conclusion

This manuscript discusses the use of statistical effect screening to pursue robust design, as applied to multi-criteria simulations. We start by making a clear distinction between design parameters, that one wishes to optimize to reach the best-possible system performance, and calibration variables, that introduce uncertainty in the forecast of performance. While we wish to optimize performance of the design, we simultaneously strive to be as robust as possible to the calibration variable uncertainty.

A framework is proposed to screen the most influential design parameters based on a novel criterion of robustness to calibration variable uncertainty. Restricting the design optimization to the most influential parameters is essential for computational efficiency. The analysis of robustness is rooted in info-gap decision theory. Parameter screening uses the Morris one-at-a-time sensitivity method, which scales well to high-dimensional problems. The robust-optimal criterion proposed for sensitivity analysis attempts to identify the most influential design parameters, despite the fact that the calibration variables can deviate from their nominal settings. The framework is applied to the NASA Multi-disciplinary Uncertainty Quantification Challenge problem, which features 14 design parameters and 21 calibration variables. The main finding is that three of the four most influential design parameters are “missed” when sensitivity analysis does not account for robustness-to-uncertainty. We conclude that extra care should be applied to parameter screening, especially if a surrogate model is developed for optimization or calibration, to ensure that the influence of design parameters selected for analysis is not falsified due to the presence of calibration variable uncertainty.

To conclude our investigation, two competing designs are sought using subsets design parameters identified as most influential using either the performance-optimal or robust-optimal sensitivity analysis. When uncertainty introduced by the calibration variables is ignored, the performance-optimal design provides better system performance than the nominal design, while the robust-optimal design performs slightly worse. This result is expected, suggesting that the Morris sensitivity analysis efficiently identifies the most influential design parameters. The challenge, however, is to reach a parameter ranking that remains unchanged even as the calibration variables vary away from their nominal values. For the NASA flight control simulation, this challenge is met by our robust-optimal screening method. The robust-optimal design, while initially less accomplished than the nominal design, sustains its performance at large levels of calibration uncertainty. The robust-optimal design clearly outperforms the performance-optimal design.

Searching for a better-performing design can be computationally expensive. One strategy to mitigate this difficulty is to sample the design space more sparsely. Another strategy is to make simplifying assumptions about the structure of the numerical simulation, and replace it by a fast-running surrogate model. Neither approach is satisfactory in large-dimensional spaces, which motivates our desire to develop a parameter screening technique that scales well to several dozen of parameters. The framework proposed is particularly amenable to parallel computing, which reduces the time-to-completion of a Morris design and robustness analysis. Both algorithms are highly parallelizable. More extensive use of parallel computing resources will be investigated in the future, as it could prevent the formulation of simplifying assumptions about the simulation model, which introduces unwanted model-form sources of error.