Keywords

1 Verification in Modeling and Simulation

A very great deal more truth can become known than can be proven. – Richard Feynman, “The Development of the Space-Time View of Quantum Electrodynamics,” Nobel Lecture (11 December 1965, (Brown and Feynman 2000), p. 29)

Many models are expressed using the language of mathematics most often in the form of differential equations. For simple or idealized cases, the equations for the model can be solved though analytical means. For a variety of reasons, however, the analytical solutions are limited and inadequate for modeling most circumstances. For this reason models are solved via computers and via numerical approximations. In the process of numerical approximation the propriety of computer modeling to the intended model should not be assumed, but rather proven through careful checks and tests. How do we do this in practice? How can we provide proof of the correctness of the intended model as well as the accuracy of the model?

The answer to this question is the processes and procedures of verification. This chapter is devoted to describing the foundational aspects of verification.

Theoretically, the conditions for the numerical approximation to solve a model correctly are well known. The process of verification harnesses this theoretical knowledge to provide confidence in the numerical solution. In addition, the numerical solutions are approximate thus (almost) always deviating from the exact solution.Footnote 1 Verification also provides means to quantify this error.

In modeling and simulation verification is a set of activities broadly supporting the quality. Verification is a dominantly mathematical exercise that assures the quality of models. In this chapter the utility of verification is focused on models described by differential equations. The practice of verification for models based on other approaches is not discussed here. Verification is paired with validation as a means to assure modeling quality. Validation is the comparison of simulated results with physical reality either from experiments or observations. Validation is an exercise in physics and engineering.

Verification consists of two modes of practice: code verification where the mathematical correctness of the computer code is assessed, and solution (calculation) verification where the numerical error (uncertainty) is estimated (for verification, these errors include discretization error, linear, and nonlinear equation solution tolerance/discrepancy, round-off error, and parallel processing consistency; see Chap. 3 by Roy in this volume for a classification of errors). Solution verification is a core practice in estimating numerical error in simulations subjected to validation, or simply as part of a modeled prediction. Both activities are closely linked to each other and they are utterly complementary in nature. To a large extent the methodology used for both types of verification is similar. Much of the mathematics and flow of work are shared in all verification, but details, pitfalls and key tips differ (Rider et al. 2016), and the differences between the two are important to maintain (Roache 1998, 2009; Stern et al. 2001; Oberkampf and Roy 2010).

The remainder of the chapter will provide basic overviews of code verification (Sect. 11.2) and solution verification (Sect. 11.4) including simple examples of each of these key practices.

2 Code Verification

Science replaces private prejudice with public, verifiable evidence. – Richard Dawkins, The Enemies of Reason, The Irrational Health ServiceFootnote 2

Computer modeling and simulation is often an activity where continuous mathematics is converted to discrete computable quantities. This process involves approximation of the continuous mathematics and in almost every non-pathological circumstance is inexact. The core of modeling and simulation is the solution of (partial) differential equations using approximation methods. Code verification is a means of assuring that the approximations used to make the discrete solution of differential equations tractable on a computer are correct. A key aspect of code verification is determining that the discrete approximation of the differential equation is consistent with the continuous version of the differential equation.

Analytical consistency demands that the so-called order of accuracy of the differential equation be at least one. In other words, the discrete equations produce solutions that are the original continuous equations plus terms that are proportional to the size of the discretization or to a higher power of it. In verification practice, consistency is the observation of a positive convergence rate indicating convergence. Numerical methods can properly exhibit sublinear convergence rates while operating completely correctly (Banks et al. 2008). This character may be examined by solving problems with an exact analytical solution (or a problem with very well controlled and characterized errors) using several discretization sizes allowing the computation of errors, and determining the order of accuracy.

It has been shown that the combination of consistency with stability of the approximation means that the approximation converges to the correct solution of the continuous differential equation (Lax and Richtmyer 1956). A simple definition of stability is the nature of a solution where small changes in the conditions result in small changes in solution. If the solution changes greatly due to small changes, the solution is unstable. The stability is associated with the appropriate concept for the approximation in question, e.g., 0-stability for ODEs or Von Neumann stability for PDEs. For steady-state solutions require a certain attention to stability of the approximation when approached iteratively. In these cases the stability can mimic that associated with time-dependent solutions.

I will examine both the nature of different types of problems to determine code verification and the methods of determining the order of accuracy. One of the key aspects of code verification is the congruence of the theoretical order of accuracy for a method, and the observed order of accuracy. The latter is obtained from a comparison of simulation outputs with a reference solution. The former is usually determined via a numerical analysis exercise using a series expansion method such as Taylor series or Fourier series. It can be done via pen and paper, chalkboard or more recently via symbolic arithmetic. The discrete system of variables is approximated and the approximations are then replaced by one of these series expansions. Such methods are described in numerical analysis texts where solvers for ODEs are the simplest rudimentary examples.

It is important to note that the theoretical order of convergence also depends upon the problem being solved. The order of convergence is a product of both this order of accuracy, and the problem being solved and its degree of mathematical smoothness, the existence of derivatives of the function. The problem must possess enough regularity to support the convergence rate expected. For some problems the correct observed rate of convergence does not match the asymptotic rate of convergence because of the nature of the solution or the lack of an asymptotic range itself (i.e., problems with singularities) (Banks et al. 2008). At this point it is important to point out that code verification produces both an order of accuracy and an observed error in solution. Both of these quantities are important. For code verification, the order of accuracy is the primary quantity of interest. It depends on both the nature of the approximation method and the problem being solved. If the problem being solved is insufficiently regular and smooth (sufficient existence of derivatives of the functional solution), the order of accuracy will not match the theoretical expectations of the method.

As mentioned above, the key idea behind code verification is determining that the discrete approximation of the differential equation is consistent with the continuous version of the differential equation. To ensure this, the error is determined by comparing the output of the computer simulation with an otherwise available solution. Ideally the available solution is produced by a precise analytical method. One can then produce a sequence of numerical approximations and check to see if the theory available explains or predicts the dependence of the error on the parameter defining the sequence. If the theory and the observed behavior is consistent, the sequence provides evidence of correctness.

When one conducts a code verification study there is a basic flow of activities and practices to conduct. First, one looks at a code to target and a problem to solve. Several key bits of information should be immediately focused upon before the problem is solved. What is the order of accuracy for the method in the code being examined, and what is the order of accuracy that the problem being solved can expose? In addition the nature of the analytical solution to the problem should be carefully considered. For example what is the nature of the solution? Closed form? Series expansion? Numerical evaluation? Some of these forms analytical solution have errors that must be controlled and assessed before the codes method may be assessed. By the same token are there auxiliary aspects of the codes solution that might pollute results? Solution of linear systems of equations? Stability issues? Computer roundoff or parallel computing issues? In each case these details could pollute results if not carefully excluded from consideration.

Next, one needs to produce a solution on a sequence of meshes by running the simulation program. For simple verification using a single discretization parameter only two discretizations are needed for verification (two equations to solve for two unknowns). For code verification the standard model for error is simple, generally a power law, \( E_k = A h_k^a\) where the error \(E_k\) in the kth solution is proportional to the discretization parameter \(h_k\) to the power (order) a . The constant of proportionality is A. The order, a, is the target of the study and one looks at its congruence with the expected theoretical order for the method on the problem being solved. It is almost always advisable to use more than the minimum number of meshes to assure that one simply is not examining anomalous behavior from the code.

One of the problems with code verification is the rarity of the observed order of convergence to exactly match the expected order of convergence (for numerical work what does “exact” mean?). The question of how close is close enough haunts investigations. Invariably the observed order will deviate from the expected order by some amount. The question for the practitioner is how close is acceptable? Generally this question is given little attention. There are more advanced verification techniques that can put this issue to rest by producing uncertainties on the observed order, but the standard techniques simply produce a single result. Usually this results in rules of thumb that apply in broad brushes, but undermine the credibility of the whole enterprise. Often the criterion is that the observed order should be within a tenth of the theoretically expected result. More generally, common sense should be applied to the decision-making as long as it can be justified.

Another key caveat comes up when the problem is discontinuous, meaning the solution of the equations contains a jump in variables. In this case the observed order is either set to one for nonlinear solutions, or weakly tied to the theoretical order of convergence. For the wave equation this result was studied by Banks, Aslam and Rider and admits an analytical and firmly determined result. Banks et al. (2008) In this case the issue of inexact congruence with the expected rate of convergence remains. In addition for problems involving systems of equations will have multiple features each having a separate order of convergence, and the rates will combine within a solution. Ultimately in an asymptotic sense the lowest order of convergence will dominate as \( h \rightarrow 0\). This is quite difficult to achieve practically.

The last major issue that comes up in code verification (and solution verification too) is the nature of the discrete mesh and its connection to the asymptotic range of convergence. All of the theoretical results apply when the discretization parameter h is small in a broad mathematical sense. This is quite problem specific and generally ill defined. Examining the congruence of the numerical derivatives of the analytical solution with the analytical derivatives can generally assess this. When these quantities are in close agreement, the solution can be considered to be asymptotic. Again these definitions are loose and generally applied with a large degree of professional or expert judgment.

It is useful to examine these issues through a concrete problem in code verification. The example I will use is a simple ordinary differential equation integrator for a linear equation \( d u(t)/dt = - a u(t) \) coded up in Mathematica. We could solve this problem in a spreadsheet (like MS Excel), python, or a standard programming language. The example will look at two first-order methods, forwards \( u(t_{n+1}) + a h u(t_n) =u (t_n) \) and backwards \( u(t_{n+1}) + a h u(t_{n+1}) = u(t_n) \) Euler methods. Both of these methods produce leading first-order errors in an asymptotic sense, \( E_k = C h_k + O(h_k^2) \). If \( h_k\) is large enough, the high-order terms will pollute the error and produce deviations from the pure first-order error. Let us look at this example and the concrete analysis from verification. This will be instructive in getting to similar problems encountered in general code verification.

Here is the Mathematica code we are going to verify

figure a

This is coded in Mathematica for both a forward and backward Euler method for a linear ODE. The code is similar to what one might code in C or Pascal. It is a coded version of the difference equations expressed in the previous paragraph. This is probably about as simple as a numerical method can be.

Let us look at the forward Euler integrator for several different choices of h, different end times for the solution and a number of discrete solutions using the method. I will do the same thing for the backwards Euler method, which is different because it is unconditionally stable with respect to step size. Both the forward and backwards Euler methods are first-order accurate as a analyzed via Taylor series expansions. For this simple ODE, the method is stable to a stepsize of \( h = 2\) and I can solve the problem to three stopping times of \( T=1.0\), \( T=10.0\) and \( T=100.0\). The analytical solution is always, \( u(T) = \exp \left( - a T\right) \). I then solve this problem using a set of step sizes, \( h=1.0, h=0.5, h=0.25, h=0.125\) to demonstrate different convergence behaviors.

I can give results for various pairs of step sizes with both integrators, and see some common pathologies that we must deal with. Even solving such a simple problem with simple methods can prove difficult and prone to heavy interpretation (arguably the simplest problem with the simplest methods). Much different results are achieved when the problem is run until different stopping times. We see the impact of accumulated error (since I am using Mathematica so aspects of round-off error are pushed aside). In cases where round-off error is significant, it would be another complication. Furthermore the backward Euler method for multiple equations would involve a linear (or nonlinear) solution that itself has an error tolerance that may significantly impact verification results.

These results are compiled in Table 11.1 showing significant variation in the method’s performance. We see good results for \( T=1.0\) and a systematic deviation for longer ending times. The good results are associated with the observed convergence rate being close to the theoretically expected rate. This problem has been acknowledged in the ODE community for long integration times and is expected. At the core are differences between the local errors associated with truncation error and global error associated with full calculations (Ascher and Petzold 1998; Hairer et al. 1993). We would not expect this to occur for boundary value problems. To get acceptable verification results would require much smaller step sizes (for longer calculations!). This shows how easy it is to scratch the surface of really complex behavior in verification that might mask correctly implemented methods. What is not so well appreciated is that this behavior is expected and amenable to analysis through standard methods extended to look for it.

Table 11.1 Computed order of convergence for forward Euler (FE) and backward Euler (BE) methods for various stopping times and step sizes. The rates of convergence deviate from theory under many circumstances
Full size table

3 Types of Code Verification Problems and Associated Benchmarks

Dont give people what they want, give them what they need. – Joss Whedon, Conversations (University Press of Mississippi, 2011, (Lavery and Burkhead 2011), p. 31)

The problem types are categorized by the difficulty of providing a solution coupled with the quality of the solution that can be obtained. These two concepts go hand in hand. A simple closed form solution is easy to obtain and evaluate. Conversely, a numerical solution of partial differential equations is difficult and carries a number of serious issues regarding its quality and trustworthiness. These issues are addressed by an increased level of scrutiny on evidence provided by associated benchmarks. Each benchmark is not necessarily analytical in nature, and the solutions are each constructed in different means with different expected levels of quality and accompanying benchmarks. This necessitates the differences in level of required documentation and accompanying supporting material to assure the user of its quality. These recommendations are the direct result of executing code verification for high consequence programs for nearly twenty years. Invariably when code verification produces concerns for an important code, the credibility of the verification problem itself is questioned.

Next, I provide a list of types of benchmarks along with an archetypical example of each. This is intended to be instructive to the experienced reader, who may recognize the example. The list is roughly ordered in increasing level of difficulty and need for greater supporting material (Kamm et al. 2008).

  • Closed form analytical solution (usually algebraic in nature). Example: Incompressible, unsteady, 2-D, laminar flow given by the Taylor and Green known as the Taylor-Green vortices (note the 3-D version of this problem does not have an analytical solution, but is a common benchmark problem) Taylor, G. I. and Green, A. E., Mechanism of the Production of Small Eddies from Large Ones, Proc. R. Soc. Lond. A, 158, 499521 (1937) (Taylor and Green 1937; Kim and Moin 1985; Drikakis and Rider 2006).

  • Analytical solution with significantly complex numerical evaluation.

  • Series solution. Example: Numerous classical problems, in H. Lambs book, Hydrodynamics, Cambridge University Press (or Dover), 1932 (Lamb 1932). Classical separation of variables solution to heat conduction. Example: Incompressible, unsteady, axisymmetric 2-D, laminar flow in a circular tube impulsively started (Szymanski flow), given in White, F. M. (1991). Viscous Fluid Flow, New York, McGraw Hill, pp. 133–134 (White 1991).

  • Nonlinear algebraic solution. Example: The Riemann shock tube problem, J. Gottleib, C. Groth, Assessment of Riemann solvers for unsteady one-dimensional inviscid flows of perfect gases, Journal of Computational Physics, 78(2), pp. 437–458, 1988 (Gottlieb and Groth 1988).

  • A similarity solution requiring a numerical solution of nonlinear ordinary differential equations.

  • Manufactured Solution. Example: Incompressible, steady, 2-D, turbulent, wall-bounded flow with two turbulence models (makes no difference to me), given in Ea, L., M. Hoekstra, A. Hay and D. Pelletier (2007). “On the construction of manufactured solutions for one and two-equation eddy-viscosity models.” International Journal for Numerical Methods in Fluids. 54(2), 119–154 (Eça et al. 2007).

  • Highly accurate numerical solution (not analytical). Example: Incompressible, steady, 2-D, laminar stagnation flow on a flat plate (Hiemenz flow), given in White, F. M. (1991). Viscous Fluid Flow, New York, McGraw Hill. pp. 152–157 (White 1991).

  • Numerical benchmark with an accurate numerical solution. Example: Incompressible, steady, 2-D, laminar flow in a driven cavity (with the singularities removed), given in Prabhakar, V. and J. N. Reddy (2006). “Spectral/hp Penalty Least-Squares Finite Element Formulation for the Steady Incompressible Navier–Stokes Equations.” Journal of Computational Physics. 215(1), 274–297 (Prabhakar and Reddy 2006).

  • Code-to-code comparison data. Example: Incompressible, steady, 2-D, laminar flow over a back-step, given in Gartling, D. K. (1990). “A Test Problem for Outflow Boundary Conditions-Flow Over a Backward-Facing Step.” International Journal for Numerical Methods in Fluids. 11, 953–967 (Gartling 1990).

Below is a list of the different types of data associated with verification problems defined above. Here, “data” refers to information that needs to be given about the benchmarks, and not empirical data from the target system of the simulation.

Depending on the nature of the test problem only a subset of these data are advisable. This will be provided below in the following list of data types. As noted above, benchmarks with well-defined closed form analytical solutions require relatively less d than a benchmark associated with the approximate numerical solution of PDEs.

  • Detailed technical description of the problem (report or paper)

  • Analysis of the mathematics of the problem (report or paper)

  • Computer analysis of solution (input file)

  • Computer solution of the mathematical solution

  • Computer implementation of the numerical solution

  • Error analysis of the exact numerical solution

  • Derivation of the source term and software implementation or input

  • Computer implementation of the source term (manufactured solution)

  • Grids for numerical solution

  • Convergence and error estimation of approximate numerical solution

  • Uncertainty and sensitivity study of numerical solution

  • Description and analysis of computational methods

  • Numerical analysis theory associated with convergence

  • Code description/manuals

  • Input files for problems and auxiliary software

  • Patch test description, Derivation, input and analysis

  • Unusual boundary conditions (inflow, piston, etc.)

  • Physics restrictions (boundary layer theory, inviscid,)

  • Software quality documents

  • Scripts and auxiliary software for verification

  • Source code

  • Metric descriptions

  • Verification results including code version, date, etc.

  • Numerical sensitivity studies

  • Feature coverage in verification

The mapping of these forms of documentation to the different types of verification problems is given in the appendix at the end of the chapter. The wealth of potential documentation for verification work highlights the complexity of the professional practice of verification. The work is far more involved and technical than commonly appreciated.

The use of direct numerical simulation (DNS) (Moin and Mahesh 1998; Moser et al. 1999; Le et al. 1997) requires a similar or even higher level of documentation than analytical solutions. This topic is being addressed because of increasing interest in using comparison to DNS as a means of assessing the quality and correctness of simulation in many fields most prominently fluid dynamics. DNS is commonly used in fluid dynamics research and other fields also being called “first principles” or “fully resolved” and used in a similar vein. Here, I will adopt the term DNS to describe this broader class of simulations discussing their appropriate use in code verification (Galli and Pasquarello 1993; Kotschenreuther et al. 1995). This coincides with the discussion of the last type of verification benchmark where a complex numerical method with significant approximations is utilized to produce the solution. As a numerically computed benchmark, the burden of proof is much larger. Code verification is best served by exact analytical solutions because of the relative ease in assuring benchmark solution accuracy. Nonetheless, it remains a common practice due to its inherent simplicity. It also appeals to those who have a vested interest in the solutions produced by a certain computer code. The credibility of the comparison is predicated on the credibility of the code producing the benchmark used as the surrogate for truth. Therefore the documentation of the benchmark must provide the basis for the credibility.

The use of DNS as a surrogate for experimental data has received significant attention. This practice violates the fundamental definition of validation we have adopted because no observation of the physical world is used to define the data. This practice also raises other difficulties, which I will elaborate upon. First the DNS code itself requires that the verification basis be further augmented by a validation basis for its application. This includes all the activities that would define a validation study including experimental uncertainty analysis numerical and physical equation based error analysis. Most commonly, the DNS is promoted to provide validation, but the DNS contains approximation errors that must be estimated as part of the error bars for the data. For most DNS simulations the results are only significant statistically. As such the data must be processed to produce various statistical measures with necessary attention being paid to statistical convergence as well. Furthermore, the code must have documented credibility beyond the details of the calculation used as data. This level of documentation again takes the form of the last form of verification benchmark introduced above because of the nature of DNS codes. For this reason I include DNS as a member of this family of benchmarks.

4 Solution Verification

There are two ways to do great mathematics. The first is to be smarter than everybody else. The second way is to be stupider than everybody else – but persistent. – Raoul BottFootnote 3

The second form of verification is solution verification. This is quite similar to code verification, but its aim is the estimation of approximation errors in a calculation. When one runs into a problem without an analytical solution, the estimation of errors is more intricate because errors are not known and must be estimated from an approximate solution. One examines a series of solutions and estimates the solution that is indicated by the sequence. Essentially the question of what solution is the approximation appearing to converge toward is being asked. If the sequence of solutions converges, the error in the solution can be inferred. As with code verification the order of convergence and the error is a product of the analysis. Conversely to the code verification, the error estimate is the primary quantity of interest, and the order of convergence is secondary.

Most often, solution verification involves examining error and results without the knowledge the exact solution. This makes it a more difficult task than code verification where an exact solution is known removing a major uncertainty. A secondary issue associated with not knowing the exact solution is the implications on the nature of the solution itself. With an exact solution, a mathematical structure exists allowing the solution to be achievable analytically. Furthermore, exact solutions are limited to relatively simple models that often cannot model reality. Thus, the modeling approach to which solution verification is applied is necessarily more complex. All of these factors are confounding and produce a more perilous environment to conduct verification.

The way to cope with this generally more hostile analysis environment involves improved analysis methods. One of the key elements in the analysis is contending with the lack of certainty about the solution, its nature and character mathematically. For this reason the knowledge and guarantees about the results is missing. For instance we do not know what order of convergence to reasonably expect from the analysis and cannot use this to screen our results. Generally speaking, if the verification result shows convergence at the theoretical rate for the method we can be sure we are solving problems that have smooth regular solutions either by character or sufficient mesh resolution. Usually the applied problems that modeling and simulation are attacking are mathematically difficult and may not be practically amenable to computational resolution sufficient to assure regularity. Philosophically, the whole reason for modeling and simulation is solving problems that are beyond our analytical grasp. In a deep sense the complex and difficult character to problems is unavoidable for the practical a use of modeling with computers. When we have successfully attacked the problem of verification for a problem without an exact solution, the same analysis methodology can improve our code verification practice.

It is important to understand solution verification within the broader context of computational modeling. Solution verification contributes to the overall enterprise of analysis uncertainty quantification. The most classical investigation will involve comparing the modeled results with observations in the real World (ideally an experiment). There are many elements to the uncertainty in this case including the model parameters, the constitutive properties, the experimental measurements and the numerical solution. Solution verification is the process for examining and estimating the numerical error and specifying its uncertainty. Sometimes this is applied in the use of computational modeling for purposes of decision-making or scenario testing where no real World data exists. In this case the numerical error is an important element in the overall lack of certainty about the results. If the numerical error is well behaved it will be a bias from the exact continuum solution to the model. This bias is important to understand in how it might skew the results and any advise.

When one lays out the mathematical framework for solution verification, the immediate impression is an added difficulty compared to code verification. This is due to the lack of direct knowledge of the precise solution. The full solution to the problem is inferred from the inaccurate numerical solutions. The equation to solve is the following \( S_0 = S_k + C h_k^a\) where the new unknown is the ostensible estimate of the exact solution \( S_0\) that is the solution where \( h=0\). The solutions used to determine this estimate are \( S_k\) the solutions found with \( h_k\). We notice that we have three unknowns, \(S_0, C, a\) meaning that the well-determined solution requires three pieces of determined data, \( S_k\) . As we will discuss, this problem can be solved in a variety of ways including under-, fully and overdetermined forms. (Here the problem, is over- (or under-)determined if we have more (or less) \(S_k\) than for a well-determined problem.) We also note that another option when more data is available would be to expand the error ansatz to include more degrees of freedom.

One of the key issues to recognize with solving this problem is an aspect of complexity because of the general nonlinearity of the determination of the model. The solution to this coupled system of nonlinear equations is generally subtle, and necessarily solved numerically. As such, the solution can have its own errors requiring some care and verification. The system of equations admits a simple analytical solution in special cases where the discrete solutions use a sequence of meshes where \( r = h_k/h_{k-1}\) is constant. In this case we can write the solution in closed form \( \log (E_{1,2}/E_{2,3}) / \log (r) \), where \( E_{k,k-1} = S_k - S_{k-1}\). More generally, we need to attack this with a coupled nonlinear solve. If we deal with an overdetermined version of the problem we will use a nonlinear least squares solver (or this is the knee-jerk response at least). One could also seek to modify or enrich the error ansatz to utilize the extra degrees of freedom. As I discuss next, following the nonlinear least squares path opens the door to some more interesting and robust choices (Huber 1996).

The general overdetermined version of the solution verification equation (i.e., more than three grids) would be amenable to solution via nonlinear least-squares method. This is not the only choice, and consideration of this opens the door to other choices. The solution to the overdetermined problem is not unique, and the solution has the imprint of the method of solution. As such the choice of least squares implies a number of explicit assumptions that the typical practitioner does not even know they are making. For example, one may choose to solve the overdetermined problem in a different norm than the two norm (i.e., least squares) (Bjork 1996). One may choose to solve a constrained problem instead of an unconstrained problem. In addition, one could consider solving an under-determined problem adding either constraints or regularizing the solution. A classical example of regularization is the Tikhonov method where a penalty is added to make the problem well determined. A popular recent approach focuses on a similar regularization, but in the one norm (compressed sensing, LASSO, ...) (Rider et al. 2016; Tibshirani 1996).

There are several practical issues related to this whole thread of discussion. One often encountered and extremely problematic issue is insanely high convergence rates. After one has been doing verification or seeing others do verification for a while, the analysis will sometimes provide an extremely high convergence rate. For example a second-order method used to solve a problem will produce a sequence that produces a seeming 15th order solution (this example is given later). This is a ridiculous and results in woeful estimates of numerical error. A result like this usually indicates a solution on a tremendously unresolved mesh, and a generally unreliable simulation. This is one of those things that analysts should be mindful of. Constrained solution of the nonlinear equations can mitigate this possibility and exclude it a priori . This general approach including the solution with other norms, constraints and other aspects is explored in  (Rider et al. 2016). The key concept is that the solution to the error estimation problem is not unique and highly dependent upon many unwritten assumptions. Different assumptions lead to different results to the problem and can be harnessed to make the analysis more robust and impervious to issues that might derail it.

The techniques discussed in that paper were originally devised to deal with the all too common case where only one or two different grids are used and the error estimation problem is under-determined. The approach taken to solve this problem involves adding constraints to the solution based on expert knowledge and judgment. This problem was then approached when it was realized that the under- fully- and overdetermined cases should all be treated consistently. The verification problem is solved repeatedly using different assumptions resulting in a natural variation in the results providing uncertainty in the error estimation and the rate of convergence. If the data is self consistent with a well-defined solution the uncertainty in the error will itself be small and the convergence rate will also be certain. Conversely if the data is conflicting or opposes expert expectations, the uncertainty will be large. This entire methodology produces a more robust numerical uncertainty that adapts to the data, and avoids using fixed size safety factors. It turns out that this expert judgment is usually called into action with verification, but in an ad hoc manner and only when the issues are serious. So-called robust verification adds the expert judgment from the outset so that more subtle issues are subject to the same treatment.

Instead of solving the verification equation once using a nonlinear least-squares approach, robust verification solves the problem in a multitude of ways. This involves solving the verification problem using other error norms in a constrained minimization framework. The data from a verification study is also used over. One standard assumption is that the solutions on the finer grids (smaller h) are closer to the exact solution, and this data is more prominent in the solution. The end result of the analysis is a multitude of estimates of the numerical error and convergence. These results are then subjected to robust statistical examination using median statistics. We report the median of the estimates as the error and convergence rate. The median deviation is used to place an uncertainty on this estimate. One of the key benefits of this estimation is its lack of susceptibility to corruption by outliers in the analysis. Outliers are further suppressed in the analysis by the use of expert judgment as constraints. For example, the absurdly large convergence rates are removed by the constraints if the rate of convergence is constrained to be below a given value.

Before moving to examples of solution verification we will show how robust verification can be used for code verification work. Since the error is known, the only uncertainty in the analysis is the rate of convergence. As we can immediately notice, this technique will get rid of a crucial ambiguity in the analysis. In standard code verification analysis, the rate of convergence is never the exact formal order, and expert judgment is used to determine if the results is close enough. With robust verification, the convergence rate has an uncertainty and the question of whether the exact value is included in the uncertainty band can be asked. Before showing the results for this application of robust verification, we need to note that the exact rate of verification is only the asymptotic rate in the limit of \( h = 0\). For a finite step size the rate of convergence should deviate from this value and for simple cases the value can be derived using a modified version of classical numerical analysis.

Our first example of solution verification will repeat our examination of simple ODE integrators, but disregard our knowledge of the exact solution. The results of this study are given in Table 11.2 showing the solution and true error for different step sizes. It is a useful example because we can examine the efficacy of solution verification with a precise knowledge of the true errors. We can use the data from our code verification study to good effect here. Here is the raw data used for the forward Euler study.

Table 11.2 Computed solution verification error estimates for forward Euler (FE) for various step sizes. The same simple linear ODE is solved as in the code verification section
Full size table

For the code verification part of the example, the estimated truncation error is \( E=0.2030 h^{1.0245\pm 0.0124}\) (the constant also has uncertainty, but its value has to be matched to the solution and convergence rate). The error bars do not take us to the theoretical convergence rate of one. The data is consistent with the rate being above one (and this is analytically expected). Using this same data for solution verification yields the following model, \( S(h) = 0.3678 \pm 0.0002 - 0.2080 h^{1.0386 \pm 0.0207}\). Close examination shows that this solution is quite close to the exact solution 0.0001 and within the error bars. If we use the standard techniques of simply least square fitting the data we get the following model, \( S(h) = 0.3677 - 0.2239 h^{1.0717} \). The error estimate here is 0.0017, which ends up being rather over generous when the standard safety factor of 1.25 is applied. Using the robust verification technique we get a better estimate of the exact solution, the actual convergence rate and a tighter error bound.

5 Solution Verification for Complex Problems

Supposing is good, but finding out is better. – Mark Twain, Autobiography of Mark Twain, Vol. 3 (2015, (Twain 2015), p. 99)

It is also useful to look at a pathological case where the rate of convergence is absurd and standard analysis would be prone to missing it. The case we have at our fingertips involved very coarse grid solutions to large eddy simulation in a complex geometry relevant to heat transfer and fluid flow in nuclear reactors (Sagaut 2006). Early calculations were used to estimate the mesh required for well-resolved calculations. As we found out, this is a perilous enterprise. A couple of codes (one production and one research) we enlisted in this study used some initial grids that were known to be inadequate. One of the codes was relatively well trusted for this class of applications and produced three solutions that for all appearances appeared reasonable. An example solution is shown in Fig. 11.1 (Rider et al. 2010). One of the key parameters is the pressure drop through the test section. Using grids 664 K (664,000), 1224 K and 1934 K elements we got pressure drops of 31.8 kPa, 24.6 kPa and 24.4 kPa respectively. Using a standard curve fitting for the effective mesh resolution gave an estimate of \(24.3\,\text{ kPa } \pm 0.0080\,\text{ kPa }\) for the resolved pressure drop and a convergence rate of 15.84. This is an absurd result and needs to simply be rejected immediately. Using the robust verification methodology on the same data set, gives a pressure drop of \(16.1\,\text{ kPa } \pm 13.5\,\text{ kPa }\) with a convergence rate of 1.23, which is reasonable. Subsequent calculations on refined grids produced results that were remarkably close to this estimate confirming the power of the technique even when given data that was substantially corrupted.

Fig. 11.1
figure 1

A cross section of the flow in a unit cell for the large eddy simulation considered here

Full size image

Our final example is a simple case of validation using the classical phenomena of vortex shedding over a cylinder at a relatively small Reynolds number. Solution verification should be an integral part of any proper validation exercise as part of a systematic identification of modeling uncertainty. This is part of a reasonable effort to validate a research code before using in on more serious problems. The key experimental value to examine is the Stouhal number defined, \( St = f \ell /U \) the shedding frequency f normalized by the size \(\ell \) of the cylinder and the velocity U, which has the value experimentally of \( 0.164\pm 0.005\) for a flow of Reynolds number 100 (the Reynolds number is the non-dimensional ratio of inertial to viscous force in a flow). In Fig. 11.2 I show the calculations and the raw data for one of the calculations used (Pawlowski et al. 2006; Lin et al. 2006). Some of the raw data for shedding frequency is provided in Table 11.3.

Fig. 11.2
figure 2

A frame of the solution quantities for the vortex shedding example including the solution and shedding data generated

Full size image
Table 11.3 Computed Strouhal number estimates for vortex shedding for mesh sizes
Full size table

When we apply the robust verification methodology to this data I find that the code produces a Strouhal number that is slightly larger than the experimental value \( St(h) = 0.1657\pm 0.0013 + C h^{1.8486\pm 0.1476}\). Including error bars recovers the experimental value within the bounds. This can be regarded as a modest success for the codes ability to be considered for more complex flows.

6 Conclusion and Prospectus

The foundation of data gathering is built on asking questions. Never limit the number of hows, whats, wheres, whens, whys, and whos, as you are conducting an investigation. A good researcher knows that there will always be more questions than answers. – Karl Pippart III, Operation Mexico! (Pippart 2015)Footnote 4

In this chapter I have examined the foundational aspects of verification. I have provided the common core of mathematical expressions of numerical error used to define both the process of code and solution verification. In each case we have given examples of the practice. Code verification is reliant upon exact solutions whereas solution verification does not have this luxury. I have discussed the numerous ways code verification can be accomplished. Code verification acts to confirm that a numerical approximation is implemented correctly by compiling mathematical evidence of correctness. Solution verification produces evidence of the magnitude of the numerical error, and the well-behaved nature of the systematic approximations present in the modeling. Both practices are grounded on the same model for numerical error and involve solving problems on a sequence of numerical grids to determine the coefficients in the error model. Together these two practices provide a cornerstone for credibility of a modeling and simulation activity. Ultimately the foundation of verification is grounded on asking hard questions about numerical methods used in simulation and collecting evidence about the answers.