Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The basic idea of modeling and simulation (M&S) fidelity is “correspondence with reality.” There are a variety of concepts regarding what is involved in M&S fidelity and about how fidelity should be measured and described. Some of the concepts contradict other concepts; most fidelity concepts are incomplete (i.e., they do not cover all aspects of fidelity) and are not expressed within the context of an M&S theory. Since the level of fidelity required is one of the main M&S cost drivers, it is important to give careful attention to M&S fidelity. Higher fidelity increases the cost of M&S development and usually increases the cost of using the M&S. More extensive input data are required to run a high-fidelity M&S and it may require greater computational resources to run.

M&S fidelity is an important factor in M&S assessments, such as verification, validation, and accreditation (VV&A) . Because of increasing reliance upon M&S results in all areas of modern science and engineering, as well as in other aspects of modern life, capability to establish confidence in M&S correctness and in the appropriateness of M&S results for particular applications has increasing importance. Establishing such confidence in M&S results depends critically upon proper determination of M&S fidelity.

Some confuse M&S fidelity with validity. Fidelity is an absolute indication of M&S results’ correspondence with reality, while validity is a relative indication of appropriateness of M&S results for a specified purpose. Validity normally uses fidelity information about the M&S as a part of the basis for assessment of M&S appropriateness for a specified purpose.

This chapter presents a basic concept of M&S fidelity; one which is generally compatible with many contemporary approaches to M&S fidelity. Then, it discusses dimensions and attributes of M&S fidelity. Finally, the chapter addresses a number of issues often encountered with M&S fidelity. For convenience, any use of fidelity, validity, or related terms will be in the context of M&S unless specifically noted otherwise. This reduces repetition of the term “M&S.”

2 Basic Concept of Fidelity

The basic concept of fidelity is correspondence with reality. Some definitions of fidelity, such as in the Department of Defense (DoD) M&S Glossary,Footnote 1 use the term “real world” as the specification of reality. However, that kind of specification of reality does not easily accommodate simulation use with situations which have no exact correspondence within the “real world” as it is normally understood; usually the “real world” reference means the reality of interest is consistent with the laws of physics, chemistry, biology, etc. Examples of reality outside the real world include an artificial reality that might be found in cartoons or in speculations about the future in science fiction. It is better to specify reality as something, which is well defined in a comprehensive fashion. For many M&S applications, the reality and its characteristics are those of the real world.

Two extensive treatments of M&S fidelity are: (1) the 1999 report of the Fidelity Implementation Study Group presented at the Spring Simulation Interoperability Workshop, which is now the standard reference document from the Simulation Interoperability Standards OrganizationFootnote 2 (Gross 1999) and (2) the Ph.D. dissertation by Z. C.Footnote 3 Roza, Simulation Fidelity Theory and Practice (Roza 2005) . Also helpful is a far briefer treatment of the subject in the special topic on fidelity in the DoD M&S VV&A Recommended Practices Guide (Modeling & Simulation Coordination Office (MSCO) 2011).

Fidelity is multidimensional . Dimensions of fidelity are the essence of how fidelity is defined; they identify the portions of the particular application domain (i.e., the reality) which the simulation addresses. To illustrate how dimensions are critical to fidelity description, consider the spatial aspects of M&S representation. Most science and engineering M&S address spatial dimensions; some only use three degrees of freedom (DOF) for the spatial aspects of M&S representation and others need to consider object orientation and use six DOF. A simulation used in a high-fidelity simulator for training medical personnel may have to have many other dimensions as well as the six DOF for spatial representation, because it will need to respond physiologically to student interventions, such as medication administration, intravenous fluid infusions, and application of oxygen by blinking eyes with pupils that react to light, chests that rise and fall with respirations, palpable pulses, various heart and lung sounds, and the ability to cry, drool, and bleed.

Description of fidelity identifies aspects of the reality addressed by the M&S. For example, description of fidelity related to simulation of radar normally would not address acoustic factors in the environment, but would focus on factors specifically pertinent to the radio frequency (RF) environment such as atmospheric characteristics and structures that might impact RF propagation. Any M&S representation of the real world is incomplete; only the reality itself is a complete representation. This incompleteness characteristic has contributed to much of the variety in fidelity concepts in M&S literature (Roza 2005) .

Attributes of fidelity identify M&S characteristics that can be used in fidelity assessment (i.e., measurement). Sometimes a single measure is used to represent M&S fidelity. Assessment of the whole M&S fidelity is often expressed in qualitative terms such as low, medium, or high. Some use a dimensionless scale to indicate overall level of M&S fidelity. The Federal Aviation Administration (FAA) provides examples of this with its fidelity levels 4–7 for flight training devices and fidelity levels A–D for full flight simulators (Flight Simulation 2014). Other attributes address particular aspects of the M&S. Relative to such attributes; fidelity may be expressed in quantitative terms, such as specifying the resolution of description (e.g., distance measurement resolution is 0.1 km) or in the accuracy of measurements (e.g., frequency determination is 97 % accurate). For some M&S applications (especially in educational and training arenas), fidelity description or assessment may be quite subjective.

Fidelity dimensions are keys to fidelity description while fidelity attributes focus on fidelity measurements. Fidelity dimensions and attributes are discussed separately below.

3 Fidelity Dimensions

What are called fidelity dimensions in this chapter have been given a variety of labels in discussions of M&S fidelity; and the individual dimensions have been called by different names in different contexts. The point is, do not focus on the label. Consider the concept so it can be applied .

The essence of fidelity dimensions is the extent to which a simulation addresses the reality it represents. The approach to fidelity presented in this chapter uses three aspects to indicate the extent of M&S representation of the reality. The term “aspect” is used because an aspect can be a dimension or a collection of dimensions. The three aspects used here are:

  1. 1.

    Entities in the reality

  2. 2.

    Processes that affect the entities in some way (their interactions, their state changes, the environment in which the entities function, etc.)

  3. 3.

    Relationships among the entities and processes

Different communities give different connotation to these terms. Most recognize an entity as a distinguishable “something” which the simulation keeps information about as it executes; however, they may differ about the “something” considered an entity. The first aspect, a single dimension, of M&S fidelity is identification of entities that are in the reality and represented in the simulation. This requires knowledge of what entities are in the reality of interest. For example, as indicated in a previous example the real world includes both acoustic signals as well as RF ones. It also includes signals at other frequencies. In some realities (of a sort which some may consider artificial), the signal spectrum might include extrasensory perception even though such would not be part of the real world according to many.

Identification of an entity requires specification of the entity’s level of resolution or aggregation . For example, is a ball team treated only as a unit in the simulation? Or is it treated both as a unit and individuals on the team? Are individuals addressed only the players or do the individuals addressed also include coaches, managers, etc.?

These comments indicate that the level of information needed for meaningful fidelity description is far greater than many realize. At a minimum, a list of entities in the reality to be represented by the simulation is needed, along with an indication for each entity of its inclusion in or omission from the simulation. This would show clearly the extent of simulation coverage of the reality’s entities. The standard model of particle physics provides a convenient illustration of that process. The standard model is a theory about fundamental particles and how they interact, which is used to explain how things work in the real world. Although the standard model has worked very well, it has some notable omissions (such as gravity and dark matter). The limitations of its fidelity are well known; at least particle physicists know them.

The second aspect, which normally would be a dimension for each of the different kinds of processes considered, of M&S fidelity is identification of processes that affect entities in some way. Many of these processes are the laws of physics, chemistry, and biology. These normally are expressed in algorithms used in the simulation. These control the movement of entities and entity responses (as in the high-fidelity medical simulator response to student-administered medication). Processes are not restricted to physical science laws; they also include psychological processes.

The third aspect, its dimensions usually parallel those of entities and processes, of M&S fidelity is the identification of relationships among the entities and processes. For example, sensor tracking of an entity is dependent upon detection of that entity. Hence, there is a relationship of dependence between those two processes. Far too often, independence among entities and processes is erroneously assumed.

The simple discussion above makes one point very clear: careful description of M&S fidelity takes a good bit of effort. Failure to put forth such effort, because people do not understand what is needed or because they are trying to save time or money (the miser perspective), is why M&S fidelity is so tenebrous.

4 Fidelity Attributes

Attributes of simulation fidelity are concerned with the quality of parameter treatment within the dimensions of simulation fidelity, which is why fidelity attributes are key to fidelity measurement. Fidelity attributes address such characteristics as accuracy, precision, timeliness (especially in distributed simulation) , potential error sources and uncertainties, consistency , and repeatability. Each of these characteristics is discussed briefly.

Accuracy, precision (which is functionally equivalent to resolution and granularity), and timeliness are characteristics that describe how close the representation of an individual parameter is to reality. Accuracy is always limited by precision. Precision, the level of resolution or granularity with which a parameter can be determined, places fundamental limits on accuracy. Normally, accuracy is determined by how well M&S algorithms represent the reality simulated. Accuracy can be measured against the reality (when such real-world data exist) or against the articulation of the abstraction that defines the reality represented. Accuracy is determined both by correctness of reality abstraction in M&S algorithms and by the extent of reality representation. Accuracy has to relate to a single parameter or set of parameters (in which case, accuracy can be conceived as a multidimensional vector with a dimension for each parameter in the set). In many cases, simulation accuracy is difficult to determine when an abundance of real-world data does not exist .

Precision is much more amenable to quantification. Typically, precision (resolution, granularity) can be determined by examination of computational processes used (round-off procedures, interpolation intervals , minimum step sizes, update/refresh rates, number of pixels for displays, etc.). In hardware-in-the-loop (HWIL) simulation, precision may be limited by facility considerations such as how well alignment of physical elements of the simulation (antennas, receivers, etc.) can be maintained. Such alignment may also be dependent upon other physical aspects of the facility, such as control of temperature and humidity in the anechoic chamber used in the simulation .

Timeliness must be given special attention in distributed simulation , in unitary simulations employing distributed processing, in discrete event simulations , and in other kinds of simulations in which some parts of the simulation may advance time more rapidly or more slowly than other parts. Manifestation of timeliness’ impact on simulation fidelity depends upon how time is managed in the simulation: continuous, time step, discrete event with complex rollback capabilities, etc. In the most simple situations, a parameter update may be missed for a time cycle or two in a particular simulation implementation—because it took too long to compute the update, because there was a delay in communications between parts of a distributed simulation , etc. The maximum magnitude of error between the parameter value with and without the missed updates quantifies the impact of timeliness on that parameter. However time may be managed (simplistically or in a sophisticated manner in such a simulation), timeliness issues create fuzziness for parameter accuracy and precision, which must be addressed in considering simulation fidelity.

There are a variety of errors that must be considered in a comprehensive and workable treatment of fidelity. These errors include deviation from correct input data; less than perfect algorithms for description of entity state, behavior, and performance; and finite limitations in computational and logical processes. Often it is possible to define accuracy in terms of errors. Sometimes it is possible to identify and quantify some errors even when total accuracy cannot be determined because other errors have not proven amenable to quantification . For example, errors introduced by interpolation between values in a table lookup process can be quantified rather easily even if the errors in the values of the table itself cannot be quantified. Errors provide a partial way to address accuracy when accuracy of a parameter cannot be fully determined.

When such errors are independent of one another, standard statistical processes may be employed to estimate their combined impact. Similar processes exist for the way that error (inaccuracy) propagates through a distributed simulation. When errors are not independent, it becomes very difficult to estimate the impact of error combinations since the combined error may be greater or less than individual errors—and in many situations, simulation errors are not independent.

Consistency is also an attribute of M&S fidelity. Consistency addresses whether M&S results are biased (consistent in the direction of error) and stable in terms of the dispersion of results induced by M&S processes. Quantification of some consistency parameters can be estimated by test cases that use boundary condition values (such as values of 0 or 1 for probabilities within the M&S) .

Repeatability is a fidelity attribute that many assume, whether it has been demonstrated or not. Repeatability simply means that the simulation should produce the same results/responses given the same stimuli (inputs, decisions, operator actions, etc.). Testing of repeatability requires potential for complete control of stochastic processes within a simulation (such as pseudorandom seed draws) if repeatability is to be ensured. Major changes in simulation results caused by running the same simulation with the same inputs on different computers have been documented. Variation in communication delays among parts of a distributed simulation can make it impossible to replicate simulation results exactly at times. Quantification of potential variability in results is an important aspect of simulation fidelity.

Some modern computer programs pose a special repeatability challenge relative to M&S fidelity. They are the adaptive programs that employ techniques by which the program may modify itself as it runs. Techniques such as generic algorithms and other adaptive programming methods in the general field of artificial intelligence tend to preclude repeatability as an indication of M&S fidelity since the program may change as it runs. This creates uncertainty about the fidelity of the simulation based upon past performance, since its performance in the future might be quite different. Often the performance will be better, but not always .

There are special fidelity concerns when a simulation involves people. This is true whether the people are operators (who may emulate physical processes, such as when a member of the control team determines whether an intercept in a war game has resulted in a kill or not) or players. The key is that the people can impact simulation results. People impact accuracy, precision, consistency, and repeatability—both as stochastic processes of the reality represented in the simulation, and as aspects of simulation implementation. Often this human aspect of simulation fidelity is totally ignored.

A special aspect of human impact on M&S fidelity has been given a name: user effects. This term is used for the differences in simulation results when different users run the same problem using the same simulation code and get different results, results which may differ by factors of several and at times even differ by orders of magnitude. This has been a particular problem with the kinds of simulations used in the nuclear power plant industry. There are a number of papers on the subject from that part of the M&S world (Ashley et al. 1999 ; Petruzzi et al. 2008) .

Uncertainty impacts the fidelity attributes discussed above. M&S uncertainty originates from many sources. Stochastic processes bring uncertainty. Lack of accuracy and precision bring uncertainty. Errors bring uncertainty. Every kind of uncertainty impacts M&S fidelity. Unfortunately M&S uncertainty is seldom treated comprehensively (Pace 2013) . Any uncertainty not explicitly addressed creates uncertainty in M&S fidelity assessment .

Qualitative descriptions of M&S fidelity (such as low, medium, or high) should not be mixed with quantitative descriptions of fidelity, which usually relate to accuracy (or a related characteristic) of some subset of parameters. No suggestion is provided regarding how to combine or integrate individual aspects of quantitative fidelity into a singular fidelity metric. Often such a single fidelity metric is not very meaningful. By analogy, a person’s health may be described as good or poor. That kind of description may have some utility, but a set of descriptions such as blood pressure is high, heart beat is irregular, limited mobility for the left elbow, vision is 20/100 for each eye with limited color blindness, allergic to penicillin and vitamin E, etc. are much more useful in determining what medical procedures to apply or whether the person is suited for a particular activity or not .

5 Issues Often Encountered with M&S Fidelity

Five issues often encountered in regard to M&S fidelity are discussed briefly here. The issues are: (1) terminology confusion, (2) information limitations, (3) measurement limitations, (4) failure to address uncertainties comprehensively, and (5) misconceptions and varied concepts about M&S fidelity.

Previous comments in this chapter have indicated the kinds of terminology confusion that one might encounter in regard to fidelity. It can be very hard to get everyone concerned with the simulation and results from it to use fidelity terms in the same way. When some use a term one way and others use it differently, confusion can result. At times, consequences from such confusion are serious. A common problem of this sort is to confuse validity and fidelity. Even a low-fidelity simulation is valid for some applications, and even a high-fidelity simulation may not be valid for a particular application.

It is not unusual for information about the reality represented by the simulation and about the fidelity referent (i.e., what defines the reality represented by the M&S) to be limited, incomplete, and inconsistent. Many M&S endeavors fail to specify what sources of information are to be used as the fidelity referent. This can make any fidelity attribute assessment questionable. For example, accuracy of a parameter determined by comparison with results from a particular test could vary significantly (factors of several) depending upon which test is used for comparison. Without explicit specification of the source(s) for the fidelity referent, manipulation of fidelity attribute assessments by those with vested interests in the M&S is possible.

The laws of physics and mathematics impose fundamental limitations on resolution and accuracy measurements . In addition, information (data) limitations tend to impose even greater limitations on resolution and accuracy measurements. It is good practice to quantify such limitations whenever possible, even if the quantification is very general (such as “accuracy may be no closer than a factor of several”). It is easy to forget how much good engineering was done in the slide rule era when many calculations were made to three or four significant figures. Even with significant limitations in accuracy, many useful results can come from simulation runs.

M&S uncertainties often are ignored, whether they arise from input data limitations, algorithms that are not very good, uncertainties in referent data, user effects, or some other source. The practice of ignoring M&S uncertainties makes useful and meaningful assessment of M&S fidelity impossible. At least all uncertainties that are generally recognized by the community using the simulation should be addressed. That at least identifies some of the bounds on fidelity. Often such bounds on M&S fidelity are larger than desired, which may create a number of management and operational problems. However, if that is the situation, it should be addressed and not ignored. It would be better if M&S uncertainties were addressed comprehensively (as noted earlier), but partial treatment of uncertainties is better than ignoring uncertainty. However, unrecognized uncertainties still impact the real M&S fidelity.

Misperceptions about fidelity can create many problems for M&S use. A simulation might be applied to a problem for which it is unsuited. A poorer simulation might be used instead of a better one because of misperceptions about fidelities of the simulation.

In addition to misperceptions about fidelity, problems can also arise simply from different concepts about fidelity. Some M&S communities (such as the FAA for flight simulators) have well-defined approaches to M&S fidelity. Unfortunately, such procedures and approaches do not always work well for other communities, especially those M&S communities, which may not have such well-established processes as the FAA. The important challenge for those involved with the M&S is to encourage all involved to have a common approach and perception of fidelity.

Those M&S communities that do not have solid, established practices of meaningful descriptions and measurement of fidelity, addressing M&S uncertainties explicitly, etc. can benefit greatly from considering the practices of those who deal carefully with such things. This would include the nuclear power plant industry for user effects, the FAA and medical device community for formal fidelity description and assessment, and the DoD for M&S validation, especially for distributed simulation.

6 Conclusions

This chapter has provided a very brief and general introduction to the subject of M&S fidelity. Its objective has been to provide general perspective on fidelity and appreciation for the importance of dealing with M&S fidelity appropriately. Some guidance has been provided about how to describe M&S fidelity by its dimensions and how to measure M&S fidelity by its attributes. A few comments about fidelity issues that are often encountered were provided .

The three fidelity sources identified in Sect. 3.2 (Gross 1999), (MSCO 2011), and (Roza 2005) provide a foundation upon which one can develop a meaningful approach to M&S fidelity within one’s M&S arena. Guidance from this chapter will help to shape such a fidelity concept.

While this chapter has focused upon the kind of real-world engineering applications that would characterize most M&S of interest to those involved with system engineering, its comments are equally applicable to M&S in other application domains (such as the social sciences or video gaming).