Abstract
Life cycle validation, verification, and testing (VV&T) is extremely important for the success of a simulation study. This paper surveys current software VV&T techniques and current simulation model VV&T techniques and describes how they can all be applied throughout the life cycle of a simulation study. The processes and credibility assessment stages of the life cycle are described and the applicability of the VV&T techniques for each stage is stated. A glossary is provided to explicitly define important terms and VV&T techniques.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
A.F. Ackerman, P.J. Fowler and R.G. Ebenau, Software inspections and the industrial production of software, in:Software Validation: Inspection, Testing, Alternatives, Alternatives, Proc. Symp. on Software Validation, Darmstadt, Germany, ed. H.-L. Hausen (1983) pp. 13–40.
W.R. Adrion, M.A. Branstad and J.C. Cherniavsky, Validation, verification, and testing of computer software, Comp. Surveys 14(1982)159–192.
D.J. Aigner, A note on verification of computer simulation models, Manag. Sci. 18(1972)615–619.
F.E. Allen and J. Cocke, A program data flow analysis procedure, Commun. ACM 19(1976)137–147.
R.C. Backhouse,Program Construction and Verification (Prentice-Hall, London, 1986).
O. Balci, Requirements for model development environments, Comp. Oper. Res. 13(1986)53–67.
O. Balci, The implementation of four conceptual frameworks for simulation modeling in high-level languages, in:Proc. 1988 Winter Simulation Conf., ed. M.A. Abrams, P.L. Haigh, and J.C. Comfort (IEEE, Piscataway, NJ, 1988) pp. 287–295.
O. Balci, Guidelines for successful simulation studies, in:Proc. 1990 Winter Simulation Conf., ed. O. Balci, R.P. Sadowski, and R.E. Nance (IEEE, Piscataway, NJ, 1990) pp. 25–32.
O. Balci, Principles of simulation model validation, verification, and testing, Technical Report TR-94-24 Department of Computer Science. Virginia Tech, Blacksburg, VA (1994).
O. Balci and R.E. Nance, Formulated problem verification as an explicit requirement of model credibility, Simulation 45(1985)76–86.
O. Balci and R.E. Nance, Simulation model development environments: A research prototype, J. Oper. Res. Soc. 38(1987)753–763.
O. Balci and R.G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Commun. ACM 24(1981)190–197.
O. Balci and R.G. Sargent, Some examples of simulation model validation using hypothesis testing, in:Proc. 1982 Winter Simulation Conf., ed. H.J. Highland, Y.W. Chao and O.S. Madrigal (IEEE, Piscataway, NJ, 1982) pp. 620–629.
O. Balci and R.G. Sargent, Validation of multivariate response models using Hotelling's two-sampleT 2 test, Simulation 39(1982)185–192.
O. Balci and R.G. Sargent, Validation of multivariate response trace-driven simulation models, in:Performance '83, ed. A. K. Agrawala and S.K. Tripathi (North-Holland, Amsterdam, 1983) 309–323.
O. Balci and R.G. Sargent, Validation of simulation models via simultaneous confidence intervals, Amer. J. Math. Manag. Sci. 4(1984)375–406.
J. Banks and J.S. Carson,Discrete-Event System Simulation (Prentice-Hall, Englewood Cliffs, NJ, 1984).
J. Banks, D. Gerstein and S.P. Searles, Modeling processes, validation, and verification of complex simulations: A survey, in:Methodology and Validation, ed. O. Balci (SCS, San Diego, CA, 1987) pp. 13–18.
H.P. Barendregt,The Lambda Calculus: Its Syntax and Semantics (North-Holland, New York, 1981).
T. Chusho, Test data selection and quality estimation based on the concept of essential branches for path testing, IEEE Trans. Software Eng. SE-13(1987)509–517.
K.J. Cohen and R.M. Cyert, Computer models in dynamic economics, Quarterly J. Econ. 75(1961)112–127.
M.J. Damborg and L.F. Fuller, Model validation using time and frequency domain error measures, ERDA Report 76-152, NTIS, Springfield, VA (1976).
M.S. Deutsch,Software Verification and Validation: Realistic Project Approaches (Prentice-Hall, Englewood Cliffs, NJ, 1982).
E.W. Dijkstra, Guarded commands, non-determinacy and a calculus for the derivation of programs, Commun. ACM 18(1975)453–457.
L.K. Dillon, Using symbolic execution for verification of Ada tasking programs, ACM Trans. Progr. Languages Syst. 12(1990)643–669.
J.H. Dobbins, Inspections as an up-front quality technique, in:Handbook of Software Quality Assurance, ed. G.G. Schulmeyer and J.I. McManus (Van Nostrand-Reinhold, New York, NY, 1987) pp. 137–177.
R.H. Dunn, The quest for software reliability, in:Handbook of Software Quality Assurance, ed. G.G. Schulmeyer and J.I. McManus (Van Nostrand-Reinhold, New York, NY, 1987) pp. 342–384.
S.E. Elmaghraby, The role of modeling in IE design, Ind. Eng. 19(1968)292–305.
J.R. Emshoff and R.L. Sisson,Design and Use of Computer Simulation Models (MacMillan, New York, NY, 1970).
R.E. Fairley, An experimental program-testing facility, IEEE Trans. Software Eng. SE-1(1975)350–357.
R.E. Fairley, Dynamic testing of simulation software, in:Proc. 1976 Summer Computer Simulation Conf., Washington, DC (Simulation Councils, La Jolla, CA, 1976) pp. 708–710.
G.S. Fishman,Principles of Discrete Event Simulation (Wiley-Interscience, New York, NY, 1978).
G.S. Fishman and P.J. Kiviat, The analysis of simulation generated time series, Manag. Sci. 13(1967)525–557.
J.W. Forrester,Industrial Dynamics (MIT Press, Cambridge, MA, 1961).
A.V. Gafarian and J.E. Walsh, Statistical approach for validating simulation models by comparison with operational systems, in:Proc. 4th Int. Conf. on Operations Research (Wiley, New York, NY, 1969) pp. 702–705.
A.R. Gallant, T.M. Gerig and J.W. Evans, Time series realizations obtained according to an experimental design, J. Amer. Statist. Assoc. 69(1974)639–645.
M. Garratt, Statistical validation of simulation models, in:Proc. 1974 Summer Computer Simulation Conf, Houston, TX (Simulation Councils, La Jolla, CA, 1974) pp. 915–926.
S.I. Gass, Decision-aiding models: Validation, assessment, and related issues for policy analysis, Oper. Res. 31(1983)603–631.
C.F. Hermann, Validation problems in games and simulations with special reference to models of international politics, Behav. Sci. 12(1967)216–231.
W. Hetzel,The Complete Guide to Software Testing (QED Information Sciences, Wellesley, MA, 1984).
C.P. Hollocker, The standardization of software reviews and audits, in:Handbook of Software Quality Assurance, ed. G.G. Schulmeyer and J.I. McManus (Van Nostrand-Reinhold, New York, NY, 1987) pp. 211–266.
W.E. Howden, Reliability of the path analysis testing strategy, IEEE Trans. Software Eng. SE-2(1976)208–214.
W.E. Howden, Functional program testing, IEEE Trans. Software Eng. SE-6(1980)162–169.
P. Howrey and H.H. Kelejian, Simulation versus analytical solutions, in:The Design of Computer Simulation Experiments, ed. T.H. Naylor (Duke University Press, Durham, NC, 1969) pp. 207–231.
A.W. Hunt, Statistical evaluation and verification of digital simulation models through spectral analysis, Ph.D. Dissertation, University of Texas at Austin, Austin, TX (1970).
S.H. Jacobson and E. Yücesan, On the NP-completeness of verifying structural properties of discrete event simulation models, Technical Report, Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA (1993).
S. Khanna, Logic programming for software verification and testing, Comp. J. 34(1991)350–357.
N.A. Kheir and W.M. Holmes, On validating simulation models of missile systems, Simulation 30(1978)117–128.
J.C. King, Symbolic execution and program testing, Commun. ACM 19(1976)385–394.
J.P.C. Kleijnen,Statistical Techniques in Simulation, Vol. 2 (Marcel Dekker, New York, NY, 1975).
P.L. Knepell and D.C. Arangno,Simulation Validation: A Confidence Assessment Methodology, Monograph 3512-04 (IEEE Computer Society Press, Los Alamitos, CA, 1993).
J.C. Knight and E.A. Myers, An improved inspection technique, Commun. ACM 36(1993)51–61.
A.M. Law, Statistical analysis of simulation output data, Oper. Res. 31(1983)983–1029.
A.M. Law and W.D. Kelton,Simulation Modeling and Analysis, 2nd ed. (McGraw-Hill, New York, NY, 1991).
Z. Manna, S. Ness and J. Vuillemin, Inductive methods for proving properties of programs, Commun. ACM 16(1973)491–502.
J. Martin and C. McClure,Diagramming Techniques for Analysts and Programmers (Prentice-Hall, Englewood Cliffs, NJ, 1985).
D.K. Miller, Validation of computer simulations in the social sciences, in:Proc. 6th Annual Conf. on Modeling and Simulation (Pittsburg, PA, 1975) pp. 743–746.
D.R. Miller, Model validation through sensitivity analysis, in:Proc. 1974 Summer Computer Simulation Conf., Houston, TX (Simulation Councils, La Jolla, CA, 1974) pp. 911–914.
D.R. Miller, Sensitivity analysis and validation of simulation models, J. Theor. Biol. 48(1974)345–360.
R.L. Moose and R.E. Nance, The design and development of an analyzer for discrete event model specifications, in:Impacts of Recent Computer Advances on Operations Research, ed. R. Sharda, B.L. Golden, E. Wasil, O. Balci and W. Stewart (Elsevier, New York, NY, 1989) pp. 407–421.
G.J. Myers, A controlled experiment in program testing and code walkthroughs/inspections, Commun. ACM 21(1978)760–768.
G.J. Myers,The Art of Software Testing (Wiley, New York, NY, 1979).
R.E. Nance, The feasibility of and methodology for developing federal documentation standards for simulation models: Final report to the National Bureau of Standards, Department of Computer Science, VPI&SU, Blacksburg, VA (1977).
R.E. Nance, Model representation in discrete event simulation: The conical methodology, Technical Report CS81003-R, Department of Computer Science, VPI&SU, Blacksburg, VA (1981).
R.E. Nance, The conical methodology: A framework for simulation model development, in:Methodology and Validation, ed. O. Balci, (SCS, San Diego, CA, 1987) pp. 38–43.
R.E. Nance, Conical methodology: An evolutionary convergence of systems and software engineering, Ann. Oper. Res. 53(1994), this volume.
R.E. Nance and C.M. Overstreet, Diagnostic assistance using digraph representations of discrete event simulation model specifications. Trans. SCS 4(1987)33–57.
T.H. Naylor and J.M. Finger, Verification of computer simulation models, Manag. Sci. 14(1967)B92-B101.
M.A. Ould and C. Unwin,Testing in Software Development (Cambridge University Press, Cambridge, 1986).
C.M. Overstreet and R.E. Nance, A specification language to assist in analysis of discrete event simulation models, Commun. ACM 28(1985)190–201.
T.I. Ören, Concepts and criteria to assess acceptability of simulation studies: A frame of reference, Commun. ACM 24(1981)180–189.
T.I. Ören, Artificial intelligence in quality assurance of simulation studies, in:Modelling and Simulation Methodology in the Artificial Intelligence Era, ed. M.S. Elzas, T.I. Ören and B.P. Zeigler (North Holland, Amsterdam, 1986) pp. 267–278.
T.I. Ören, Quality assurance paradigms for artificial intelligence in modelling and simulation, Simulation 48(1987)149–151.
R.J. Paul, Visual simulation: Seeing is believing?, in:Impaçts of Recent ComputerAdvances on Operations Research, ed. R. Sharda, B.L. Golden, E. Wasil, O. Balci, and W. Stewart (Elsevier, New York, NY, 1989) pp. 422–432.
R.E. Prather and J.P. Myers, Jr., The path prefix software testing strategy, IEEE Trans. Software Eng. SE-13(1987)761–766.
C.V. Ramamoorthy, S.F. Ho and W.T. Chen, On the automated generation of program test data, IEEE Trans. Software Eng. SE-2(1976)293–300.
C. Reynolds and R.T. Yeh, Induction as the basis for program verification, IEEE Trans. Software Eng. SE-2(1976)244–252.
D.J. Richardson and L.A. Clarke, Partition analysis: A method combining testing and verification, IEEE Trans. Software Eng. SE-11(1985)1477–1490.
J.R. Rowland and W.M. Holmes, Simulation validation with sparse random data, Comp. Elect. Eng. 5(1978)37–49.
R.G. Sargent, Validation and verification of simulation models, in:Proc. 1992 Winter Simulation Conf., ed. J.J. Swain, D. Goldsman, R.C. Crain, and J.R. Wilson (IEEE, Piscataway, NJ, 1992) pp. 104–114.
E. Satterthwaite, Debugging tools for high level languages, Software — Practice and Experience 2(1972)197–217.
S.R. Schach,Software Engineering, 2nd ed. (Irwin, Homewood, IL 1993).
S. Schlesinger et al., Terminology for model credibility, Simulation 32(1979)103–104.
B. Schmeiser, Random variate generation, in:Proc. 1981 Winter Simulation Conf., ed. T.I. Ören, C.M. Delfosse and C.M. Shub (IEEE, Piscataway, NJ, 1981) pp. 227–242.
T.J. Schriber,Simulation Using GPSS (Wiley, New York, NY, 1974).
L.W. Schruben, Establishing the credibility of simulations, Simulation 34(1980)101–105.
R.E. Shannon,Systems Simulation: The Art and Science (Prentice-Hall, Englewood Cliffs, NJ, 1975).
L.G. Stucki, New directions in automated tools for improving software quality, in:Current Trends in Programming Methodology, Vol. 2, ed. R. Yeh (Prentice-Hall, Englewood Cliffs, NJ, 1977) pp. 80–111.
T.J. Teorey, Validation criteria for computer system simulations, Simuletter 6(1975)9–20.
H. Theil,Economic Forecasts and Policy (North-Holland, Amsterdam, 1961).
A.M. Turing, Computing machinery and intelligence, in:Computers and Thought, ed. E.A. Feigenbaum and J. Feldman (McGraw-Hill, New York, NY, 1963) pp. 11–15.
T.P. Tytula, A method for validating missile system simulation models, Technical Report E-78-11, U.S. Army Missile R&D Command, Redstone Arsenal, AL (1978).
U.S. GAO,DOD Simulations: Improved Assessment Procedures Would Increase the Credibility of Results, U.S. General Accounting Office GAO/PEMD-88-3, Washington, DC (1987).
R.L. Van Horn, Validation of simulation results, Manag. Sci. 17(1971)247–258.
D. Watts, Time series analysis, in:The Design of Computer Simulation Experiments, ed. T.H. Naylor (Duke University Press, Durham, NC, 1969) pp. 165–179.
R.B. Whitner and O. Balci, Guidelines for selecting and using simulation model verification techniques, in:Proc. 1989 Winter Simulation Conf., ed. E.A. MacNair, K.J. Musselman and P. Heidelberger (IEEE, Piscataway, NJ, 1989) pp. 559–568.
J.R. Wilson and A.A.B. Pritsker, A survey of research on the simulation startup problem, Simulation 31(1978)55–58.
R.N. Woolley and M. Pidd, Problem structuring — A literature review, J. Oper. Res. Soc. 32(1981)197–206.
R.D. Wright, Validating dynamic models: An evaluation of tests of predictive power, in:Proc. 1972 Summer Computer Simulation Conf., San Diego, CA, (Simulation Councils, La Jolla, CA, 1972) pp. 1286–1296.
R.T. Yeh, Verification of programs by predicate transformation, in:Current Trends in Programming Methodology, Vol. 2, ed. R. Yeh (Prentice-Hall, Englewood Cliffs, NJ, 1977) pp. 228–247.
E. Yourdon,Structured Walkthroughs, 3rd ed. (Yourdon Press, New York, NY, 1985).
E. Yücesan and S.H. Jacobson, Building correct simulation models is difficult, in:Proc. 1992 Winter Simulation Conf., ed. J.J. Swain, D. Goldsman, R.C. Crain, and J.R. Wilson (IEEE, Piscataway, NJ, 1992) pp. 783–790.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Balci, O. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Ann Oper Res 53, 121–173 (1994). https://doi.org/10.1007/BF02136828
Issue Date:
DOI: https://doi.org/10.1007/BF02136828