Abstract
Since Mutation Testing was proposed in the 1970s, it has been considered as an effective technique of software testing process for evaluating the quality of the test data. In other words, Mutation Testing is used to evaluate the fault detection capability of the test data by inserting errors into the original program to generate mutations, and after then check whether tests are good enough to detect them. However, the problems of mutation testing such as a large number of generated mutants or the existence of equivalent mutants, are really big barriers for applying mutation testing. A lot of solutions have been proposed to solve that problems. A new form of Mutation Testing, Higher Order Mutation Testing, was first proposed by Harman and Jia in 2009 and is one of the most promising solutions. In this paper, we consider the main limitations of Mutation Testing and previous proposed solutions to solve that problems. This paper also refers to the development of Higher Order Mutation Testing and reviews the methods for finding the good Higher Order Mutants.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Computer 11(4), 34–41 (1978)
Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering SE-3(4), 279–290 (1977)
Jia, Y., Harman, M.: Higher order mutation testing. Information and Software Technology 51, 1379–1393 (2009)
Harman, M., Jia, Y., Langdon, W.B.: A Manifesto for Higher Order Mutation Testing. In: Third International Conf. on Software Testing, Verification, and Validation Workshops (2010)
Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Information and Software Technology 51(6), 957–976 (2009)
Harman, M.: The current state and future of search based software engineering. In: Briand, L., Wolf, A. (eds.) Future of Software Engineering 2007, pp. 342–357. IEEE Computer Society Press, Los Alamitos (2007)
Harman, M., Mansouri, A., Zhang, Y.: Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Technical Report TR-09-03, Department of Computer Science, King’s College London (2009)
Raiha, O.: A survey on search based software design. Technical Report Technical Report D-2009-1, Department of Computer Sciences, University of Tamper (2009)
Langdon, W.B., Harman, M., Jia, Y.: Efficient multi-objective higher order mutation testing with genetic programming. The Journal of Systems and Software 83 (2010)
Jia, Y., Harman, M.: Constructing Subtle Faults Using Higher Order Mutation Testing. In: Proc. Eighth Int’l Working Conf. Source Code Analysis and Manipulation (2008)
Jia, Y., Harman, M.: MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language. In: Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART 2008), pp. 94–98. IEEE Computer Society, Windsor (2008)
Offutt, A.J.: Investigations of the software testing coupling effect. ACM Transactions on Software Engineering and Methodology 1(1), 5–20 (1992)
Adamopoulos, K., Harman, M., Hierons, R.M.: How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. In: Deb, K., Tari, Z. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 1338–1349. Springer, Heidelberg (2004)
Acree, A.T.: On Mutation. PhD thesis, Georgia Inst. of Technology (1980)
Budd, T.A.: Mutation Analysis of Program Test Data. PhD thesis, Yale Univ. (1980)
DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, W.M., Offutt, A.J.: An Extended Overview of the Mothra Software Testing Environment. In: Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, pp. 142–151 (1988)
Sahinoglu, M., Spafford, E.H.: A Bayes Sequential Statistical Procedure for Approving Software Products. In: Proc. IFIP Conf. Approving Software Products, pp. 43–56 (1990)
King, K.N., Offutt, A.J.: A Fortran Language System for Mutation-Based Software Testing. Software: Practice and Experience 21(7), 685–718 (1991)
Mathur, A.P., Wong, W.E.: An Empirical Comparison of Mutation and Data Flow Based Test Adequacy Criteria. Technical Report, Purdue Univ. (1993)
Wong, W.E.: On Mutation and Data Flow. PhD thesis, Purdue Univ. (1993)
Hussain, S.: Mutation Clustering. Master’s thesis, King’s College London (2008)
Ji, C., Chen, Z., Xu, B., Zhao, Z.: A Novel Method of Mutation Clustering Based on Domain Analysis. In: Proc. 21st Int’l Conf. Software Eng. and Knowledge Eng. (2009)
Agrawal, H., DeMillo, R.A., Hathaway, B., Hsu, W., Krauser, E.W., Martin, R.J., Mathur, A.P., Spafford, E.: Design of Mutant Operators for the C Programming Language. Technical Report SERC-TR-41-P, Purdue Univ. (1989)
Mathur, A.P.: Performance, Effectiveness, and Reliability Issues in Software Testing. In: Proc. Fifth Int’l Computer Software and Applications Conf., pp. 604–605 (1991)
Offutt, A.J., Rothermel, G., Zapf, C.: An Experimental Evaluation of Selective Mutation. In: Proc. 15th Int’l Conf. Software Eng., pp. 100–107 (1993)
Wong, W.E., Mathur, A.P.: Reducing the Cost of Mutation Testing: An Empirical Study. J. Systems and Software 31(3), 185–196 (1995)
Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An Experimental Determination of Sufficient Mutant Operators. ACM Trans. Soft. Eng. and Methodology (1996)
Mresa, E.S., Bottaci, L.: Efficiency of Mutation Operators and Selective Mutation Strategies: An Empirical Study. Software Testing, Verification, and Reliability 9(4), 205–232 (1999)
Barbosa, E.F., Maldonado, J.C., Vincenzi, A.M.R.: Toward the Determination of Sufficient Mutant Operators for C. Software Testing, Verification, and Reliability 11(2), 113–136 (2001)
Namin, A.S., Andrews, J.H.: Finding Sufficient Mutation Operators via Variable Reduction. In: Proc. Second Workshop Mutation Analysis, p. 5 (2006)
Namin, A.S., Andrews, J.H.: On Sufficiency of Mutants. In: Proc. 29th Int’l Conf. Software Eng., pp. 73–74 (2007)
Namin, A.S., Andrews, J.H., Murdoch, D.J.: Sufficient Mutation Operators for Measuring Test Effectiveness. In: Proc. 30th Int’l Conf. Software Eng., pp. 351–360 (2008)
Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the Cost of Mutation Testing with Second-Order Mutants. Software Testing, Verification, and Reliability 19(2), 111–131 (2008)
Purushothaman, R., Perry, D.E.: Toward Understanding the Rhetoric of small source code changes. IEEE Transactions on Software Engineering 31(6) (2005)
Langdon, W.B., Harman, M., Jia, Y.: Multi Objective Higher Order Mutation Testing with Genetic Programming. In: Proc. Fourth Testing: Academic and Industrial Conf. Practice and Research (2009)
Howden, W.E.: Weak Mutation Testing and Completeness of Test Sets. IEEE Trans. Soft. Eng. 8(4), 371–379 (1982)
Girgis, M.R., Woodward, M.R.: An Integrated System for Program Testing Using WeakMutation and Data Flow Analysis. In: Proc. Eighth Int’l Conf. Software Eng. (1985)
Horgan, J.R., Mathur, A.P.: Weak Mutation is Probably Strong Mutation. Technical Report SERC-TR-83-P, Purdue Univ. (1990)
Woodward, M.R.: Mutation Testing-An Evolving Technique. In: Proc. IEE Colloquium on Software Testing for Critical Systems, pp. 3/1–3/6 (1990)
Marick, B.: The Weak Mutation Hypothesis. In: Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 190–199 (1991)
Offutt, A.J., Lee, S.D.: How Strong is Weak Mutation. In: Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 200–213 (1991)
Offutt, A.J., Lee, S.D.: An Empirical Evaluation of Weak Mutation. IEEE Trans. Software Eng. 20(5), 337–344 (1994)
Woodward, M.R., Halewood, K.: From Weak to Strong Dead or Alive? An Analysis of Some Mutationtesting Issues. In: Proc. Second Workshop on Software Testing, Verification, and Analysis, pp. 152–158 (1988)
Jackson, D., Woodward, M.R.: Parallel Firm Mutation of Java Programs. In: Proc. First Workshop on Mutation Analysis, pp. 55–61 (2000)
Offutt, A.J., King, K.N.: A Fortran 77 Interpreter for Mutation Analysis. ACM SIGPLAN Notices 22(7), 177–188 (1987)
Choi, B., Mathur, A.P.: High-Performance Mutation Testing. J. Systems and Software 20(2), 135–152 (1993)
Delamaro, M.E.: Proteum-A Mutation Analysis Based Testing Environment. Master’s thesis, Univ. of Sao Paulo (1993)
Delamaro, M.E., Maldonado, J.C.: Proteum: A Tool for the Assessment of Test Adequacy for C Programs. In: Proc. Conf. Performability in Computing Systems (1996)
DeMillo, R.A., Krauser, E.W., Mathur, A.P.: Compiler-Integrated Program Mutation. In: Proc. Fifth Ann. Computer Software and Applications Conf., pp. 351–356 (1991)
Krauser, E.W.: Compiler-Integrated Software Testing. PhD thesis, Purdue Univ. (1991)
Untch, R.H.: Mutation-Based Software Testing Using Program Schemata. In: Proc. 30th Ann. Southeast Regional Conf., pp. 285–291 (1992)
Mathur, A.P.: CS 406 Software Engineering I. Course Project Handout (1992)
Untch, R.H., Offutt, A.J., Harrold, M.J.: Mutation Analysis Using Mutant Schemata. In: Proc. Int’l Symp. Software Testing and Analysis, pp. 139–148 (1993)
Untch, R.H.: Schema-Based Mutation Analysis: A New Test Data Adequacy Assessment Method. PhD thesis, Clemson Univ. (1995)
Offutt, A.J., Ma, Y.S., Kwon, Y.R.: An Experimental Mutation System for Java. ACM SIGSOFT Software Eng. Notes 29(5), 1–4 (2004)
Ma, Y.S., Offutt, A.J., Kwon, Y.R.: MuJava: An Automated Class Mutation System. Software Testing, Verification, and Reliability 15(2), 97–133 (2005)
Ma, Y.S., Offutt, A.J., Kwon, Y.R.: MuJava: A Mutation System for Java. In: Proc. 28th Int’l Conf. Software Eng., pp. 827–830 (2006)
Schuler, D., Dallmeier, V., Zeller, A.: Efficient Mutation Testing by Checking Invariant Violations. In: Proc. Int’l Symp. Software Testing and Analysis (2009)
Bogacki, B., Walter, B.: Evaluation of Test Code Quality with Aspect-Oriented Mutations. In: Abrahamsson, P., Marchesi, M., Succi, G. (eds.) XP 2006. LNCS, vol. 4044, pp. 202–204. Springer, Heidelberg (2006)
Bogacki, B., Walter, B.: Aspect-Oriented Response Injection: An Alternative to Classical Mutation Testing. In: Sacha, K. (ed.) Soft. Eng. Techniques: Design for Quality. IFIP, pp. 273–282. Springer, Boston (2007)
Mathur, A.P., Krauser, E.W.: Mutant Unification for Improved Vectorization. Technical Report SERC-TR-14-P, Purdue Univ. (1988)
Krauser, E.W., Mathur, A.P., Rego, V.J.: High Performance Software Testing on SIMD Machines. In: Proc. Second Workshop on Software Testing, Verification and Analysis (1988)
Krauser, E.W., Mathur, A.P., Rego, V.J.: High Performance Software Testing on SIMD Machines. IEEE Trans. Software Eng. 17(5), 403–423 (1991)
Offutt, A.J., Pargas, R.P., Fichter, S.V., Khambekar, P.K.: Mutation Testing of Software Using a MIMD Computer. In: Proc. Int’l Conf. Parallel Processing, pp. 255–266 (1992)
Zapf, C.N.: A Distributed Interpreter for the Mothra Mutation Testing System. Master’s thesis, Clemson Univ. (1993)
Weiss, S.N., Fleyshgakker, V.N.: Improved Serial Algorithms for Mutation Analysis. ACM SIGSOFT Software Eng. Notes 18(3), 149–158 (1993)
Fleyshgakker, V.N., Weiss, S.N.: Efficient Mutation Analysis: A New Approach. In: Proc. Int’l Symp. Software Testing and Analysis, pp. 185–195 (1994)
Madeyski, L., Orzeszyna, W., Torkar, R., Józala, M.: Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering (2013), http://dx.doi.org/10.1109/TSE.2013.44 (accepted)
Baldwin, D., Sayward, F.G.: Heuristics for determin-ing equivalence of program mutations. Yale University, New Haven, Connecticut, Tech. Report 276 (1979)
Offutt, A.J., Craft, W.M.: Using compiler optimization techniques to detect equivalent mutants. Software Testing, Verification and Reliability 4(3), 131–154 (1994)
Offutt, A.J., Pan, J.: Detecting equivalent mutants and the feasible path problem. In: Proc. Eleventh Annual Conf. ‘Systems Integrity Computer Assurance COMPASS 1996 Software Safety. Process Security’, pp. 224–236 (1996)
Hierons, R., Harman, M., Danicic, S.: Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability (1999)
Ellims, M., Ince, D., Petre, M.: The Csaw C mutation tool: Initial results. In: Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques- MUTATION, pp. 185–192. IEEE Computer Society Press, Washington, DC (2007)
Martin, E., Xie, T.: A fault model and mutation testing of access control policies. In: Proceedings of the 16th International Conference on World Wide Web. WWW 2007, pp. 667–676. ACM Press, New York (2007)
DuBousquet, L., Delaunay, M.: Towards mutation analysis for Lustre programs. Electronic Notes in Theoretical Computer Science 203(4), 35–48 (2008)
Offutt, A.J.: Investigations of the software testing coupling effect. ACM Transactions on Software Engineering Methodology 1, 5–20 (1992)
Mresa, E.S., Bottaci, L.: Efficiency of mutation operators and selective mutation strategies: An empirical study. Software Testing, Verification and Reliability (1999)
Harman, M., Hierons, R., Danicic, S.: The relationship between program dependence and mutation analysis. In: Wong, W.E. (ed.) Mutation Testing for the New Century, pp. 5–13. Kluwer Academic Publishers, Norwell (2001)
Adamopoulos, K., Harman, M., Hierons, R.M.: How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In: Deb, K., Tari, Z. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 1338–1349. Springer, Heidelberg (2004)
Offutt, A.J., Ma, Y.S., Kwon, Y.R.: The class-level mutants of MuJava. In: Proceedings of the 2006 International Workshop on Automation of Software Test - AST 2006. AST 2006, pp. 78–84. ACM Press, New York (2006)
Kaminski, G., Ammann, P.: Using a fault hierarchy to improve the efficiency of DNF logic mutation testing. In: Proc. Int. Conf. Software Testing Verification and Validation, ICST 2009, pp. 386–395 (2009)
Ji, C., Chen, Z., Xu, B., Wang, Z.: A new mutation analysis method for testing Java exception handling. In: Proc. 33rd Annual IEEE Int. Computer Software and Applications Conf., COMPSAC 2009, vol. 2, pp. 556–561 (2009)
Kintis, M., Papadakis, M., Malevris, N.: Evaluating mutation testing alternatives: A collateral experiment. In: Proc. 17th Asia Pacific Soft. Eng. Conf. (APSEC) (2010)
Papadakis, M., Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. ICSTW 2010, pp. 90–99. IEEE Computer Society (2010)
Vincenzi, A.M.R., Nakagawa, E.Y., Maldonado, J.C., Delamaro, M.E., Romero, R.A.F.: Bayesian-learning based guide-lines to determine equivalent mutants. International Journal of Soft. Eng. and Knowledge Engineering 12(6), 675–690 (2002)
Grün, B.J.M., Schuler, D., Zeller, A.: The impact of equivalent mutants. In: Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops, pp. 192–199. IEEE Computer Society, Denver (2009)
Schuler, D., Dallmeier, V., Zeller, A.: Efficient mutation testing by checking invariant violations. In: Proceedings of the Eighteenth International Symposium on Software Testing and Analysis. ISSTA 2009. ACM Press, New York (2009)
Schuler, D., Zeller, A.: (Un-)covering equivalent mutants. In: Proceedings of the 3rd International Conference on Software Testing Verification and Validation (ICST 2010), Paris, France, pp. 45–54 (2010)
Schuler, D., Zeller, A.: Covering and uncovering equivalent mutants. Software Testing, Verification and Reliability 23(5), 353–374 (2012)
Offutt, A.J., Pan, J.: Automatically Detecting Equivalent Mutants and Infeasible Paths. Software Testing, Verification and Reliability 7(3), 165–192 (1997)
Clark, J.A., Dan, H., Hierons, R.M.: Semantic Mutation Testing. In: Third International Conf. on Software Testing, Verification and Validation Workshops (2010)
Dan, H., Hierons, R.M.: Semantic Mutation Analysis of Floating-point Comparison. In: IEEE Fifth International Conference on Software Testing, Verification and Validation (2012)
Boonyakulsrirung, P., Suwannasare, T.: A Weak Mutation Testing Framework for WS-BPEL. In: Eighth International Joint Conf. on Computer Science and Soft. Engineering (2011)
Durelli, V.H.S., Offutt, A.J., Delamaro, M.E.: Toward Harnessing High-level Language Virtual Machines for Further Speeding up Weak Mutation Testing. In: IEEE Fifth International Conference on Software Testing, Verification and Validation (2012)
Mateo, P.R., Usaola, M.P.: Mutant Execution Cost Reduction Through MUSIC (MUtant Schema Improved with extra Code). In: IEEE Fifth International Conference on Software Testing, Verification and Validation (2012)
Lisherness, P., Lesperance, N., Cheng, K.T.: Mutation Analysis with Coverage Discounting. In: Design, Automation and Test in Europe Conference and Exhibition (2013)
Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the Cost of Mutation Testing with Second-Order Mutants. Software Testing, Verification, and Reliability 19(2), 111–131 (2008)
Madeyski, L., Radyk, N.: Judy - a mutation testing tool for Java. IET Software 4(1), 32–42 (2010), http://madeyski.e-informatyka.pl/download/Madeyski10b.pdf
Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007), http://madeyski.e-informatyka.pl/download/Madeyski07.pdf
Madeyski, L.: Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites. Software Process: Improvement and Practice 13(3), 281–295 (2008), http://madeyski.e-informatyka.pl/download/Madeyski08.pdf
Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Information and Software Technology 52, 169–184 (2010), http://madeyski.e-informatyka.pl/download/Madeyski10c.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Nguyen, Q.V., Madeyski, L. (2014). Problems of Mutation Testing and Higher Order Mutation Testing. In: van Do, T., Thi, H., Nguyen, N. (eds) Advanced Computational Methods for Knowledge Engineering. Advances in Intelligent Systems and Computing, vol 282. Springer, Cham. https://doi.org/10.1007/978-3-319-06569-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-06569-4_12
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06568-7
Online ISBN: 978-3-319-06569-4
eBook Packages: EngineeringEngineering (R0)