Skip to main content
SpringerLink
Log in

Software Regression Testing in Industrial Settings: Preliminary Findings from a Literature Review

  • Conference paper
  • First Online:
Trends in Artificial Intelligence and Computer Engineering (ICAETT 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 407))

Abstract

In the professional field of software development, unit tests are often used as a verification mechanism of the code produced. With the design and execution of test cases, it is possible to verify new or modified code, as well as to verify existing versions of it. In order to prevent new defects from spreading over existing versions of the code, it is necessary to strategically re-run different test cases. This activity is known as software regression testing and can constitute a high percentage of the total cost of the verification process. Researchers continuously are dealing with making software regression testing efficient. In order to shed light on the application of software regression testing in industry, this paper presents a literature review on different aspects of regression testing applied in industrial settings. As a result, 40 primary studies that report the use of regression testing in the industry were identified. We observe that the main regression testing technique used is the selection of test cases followed by the prioritization of them. The use of combinations of metrics based on coverage, requirements, risk, defects, cost-efficiency searches is also observed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Rothermel, G., Harrold, M.J.: A safe, efficient algorithm for regression test selection. In: Proceedings of the Conference on Software Maintenance, 1993, CSM-1993, pp. 358–367 (1993)

    Google Scholar 

  2. Leung, H.K.N., White, L.: Insights into regression testing [software testing]. In: Proceedings Conference on Software Maintenance - 1989. pp. 60–69 (1989)

    Google Scholar 

  3. Beizer, B.: Software Testing Techniques, 2Nd edn. Van Nostrand Reinhold Co., New York (1990)

    MATH  Google Scholar 

  4. Humble, J., Farley, D.: Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Pearson Education, Boston (2010)

    Google Scholar 

  5. Rosero, R.H., Gómez, O.S., Rodríguez, G.: 15 years of software regression testing techniques — a survey. Int. J. Softw. Eng. Knowl. Eng.. 26, 675–689 (2016)

    Article  Google Scholar 

  6. Rosero, R.H., Gómez, O.S., Rodríguez, G.: Regression testing of database applications under an incremental software development setting. IEEE Access. 5, 18419–18428 (2017)

    Article  Google Scholar 

  7. Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering – a systematic literature review. Inf. Softw. Technol. 51, 7–15 (2009)

    Article  Google Scholar 

  8. Rothermel, G., Harrold, M.J.: Empirical studies of a safe regression test selection technique. IEEE Trans. Softw. Eng. 24, 401–419 (1998)

    Article  Google Scholar 

  9. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22, 67–120 (2012)

    Article  Google Scholar 

  10. Rothermel, G., Harrold, M.J.: Analyzing regression test selection techniques. IEEE Trans. Softw. Eng. 22, 529–551 (1996)

    Article  Google Scholar 

  11. Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng.. 27, 929–948 (2001)

    Article  Google Scholar 

  12. Mohanty, R., Ravi, V., Patra, M.R.: The application of intelligent and soft-computing techniques to software engineering problems: a review. Int. J. Inf. Decis. Sci. 2, 233–272 (2010)

    Google Scholar 

  13. Santos, A., Gómez, O., Juristo, N.: Analyzing families of experiments in SE: a systematic mapping study. IEEE Trans. Softw. Eng. 46, 566–583 (2020)

    Article  Google Scholar 

  14. Dieste, O., Grimán, A., Juristo, N.: Developing search strategies for detecting relevant experiments. Empir. Softw. Eng. 14, 513–539 (2009)

    Article  Google Scholar 

  15. Lübke, D.: Selecting and prioritizing regression test suites by production usage risk in time-constrained environments. In: Winkler, D., Biffl, S., Mendez, D., Bergsmann, J. (eds.) SWQD 2020. LNBIP, vol. 371, pp. 31–50. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-35510-4_3

    Chapter  Google Scholar 

  16. Magalhães, C., Andrade, J., Perrusi, L., Mota, A., Barros, F., Maia, E.: HSP: a hybrid selection and prioritisation of regression test cases based on information retrieval and code coverage applied on an industrial case study. J. Syst. Softw. 159, 110430 (2020)

    Google Scholar 

  17. Wu, Z., Yang, Y., Li, Z., Zhao, R.: A time window based reinforcement learning reward for test case prioritization in continuous integration. In: Proceedings of the 11th Asia-Pacific Symposium on Internetware. pp. 1–6. Association for Computing Machinery, New York (2019)

    Google Scholar 

  18. Pal, D., Vain, J.: A systematic approach on modeling refinement and regression testing of real-time distributed systems. IFAC-PapersOnLine. 52, 1091–1096 (2019)

    Article  Google Scholar 

  19. Correia, D.: An industrial application of test selection using test suite diagnosability. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1214–1216. Association for Computing Machinery, New York (2019)

    Google Scholar 

  20. Marijan, D., Gotlieb, A., Liaaen, M.: A learning algorithm for optimizing continuous integration development and testing practice. Softw. Pract. Exp. 49, 192–213 (2019)

    Google Scholar 

  21. Ali, S., Hafeez, Y., Hussain, S., Yang, S.: Enhanced regression testing technique for agile software development and continuous integration strategies. Softw. Qual. J. 28(2), 397–423 (2019). https://doi.org/10.1007/s11219-019-09463-4

    Article  Google Scholar 

  22. Wang, X., Zeng, H., Gao, H., Miao, H., Lin, W.: Location-based test case prioritization for software embedded in mobile devices using the law of gravitation. Mobile Inf. Syst. 2019, e9083956 (2019)

    Google Scholar 

  23. Tulasiraman, M., Vivekanandan, N., Kalimuthu, V.: Multi-objective test case prioritization using improved Pareto-optimal clonal selection algorithm. 3D Res. 9(3), 1–13 (2018). https://doi.org/10.1007/s13319-018-0182-y

    Article  Google Scholar 

  24. Aman, H., Nakano, T., Ogasawara, H., Kawahara, M.: A topic model and test history-based test case recommendation method for regression testing. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). pp. 392–397 (2018)

    Google Scholar 

  25. Akin, A., Sentürk, S., Garousi, V.: Transitioning from manual to automated software regression testing: experience from the banking domain. In: 2018 25th Asia-Pacific Software Engineering Conference (APSEC), pp. 591–597 (2018)

    Google Scholar 

  26. Marijan, D., Liaaen, M.: Practical selective regression testing with effective redundancy in interleaved tests. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, pp. 153–162. Association for Computing Machinery, New York (2018)

    Google Scholar 

  27. Araújo, J., Araújo, J., Magalhães, C., Andrade, J., Mota, A.: Feasibility of using source code changes on the selection of text-based regression test cases. In: Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing, pp. 1–6. Association for Computing Machinery, New York (2017)

    Google Scholar 

  28. Aman, H., Nakano, T., Ogasawara, H., Kawahara, M.: A test case recommendation method based on morphological analysis, clustering and the Mahalanobis-Taguchi method. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 29–35 (2017)

    Google Scholar 

  29. Ramler, R., Salomon, C., Buchgeher, G., Lusser, M.: Tool support for change-based regression testing: an industry experience report. In: Winkler, D., Biffl, S., Bergsmann, J. (eds.) SWQD 2017. LNBIP, vol. 269, pp. 133–152. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-49421-0_10

    Chapter  Google Scholar 

  30. EmaShankari, K.H., ThirumalaiSelvi, R., Balasubramanian, N.V.: Industry based regression testing using IIGRTCP algorithm and RFT tool. In: Lecture Notes in Engineering and Computer Science, pp. 473–478. Newswood Limited (2016)

    Google Scholar 

  31. de Oliveira Neto, F.G., Torkar, R., Machado, P.D.L.: Full modification coverage through automatic similarity-based test case selection. Inf. Softw. Techn.. 80, 124–137 (2016)

    Google Scholar 

  32. Anderson, J., Salem, S., Do, H.: Striving for failure: an industrial case study about test failure prediction. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, pp. 49–58 (2015)

    Google Scholar 

  33. Horváth, F., et al.: Test suite evaluation using code coverage based metrics. Presented at the Proceedings of the 14th Symposium on Programming Languages and Software Tools (SPLST’15) October (2015)

    Google Scholar 

  34. Marijan, D.: Multi-perspective regression test prioritization for time-constrained environments. In: 2015 IEEE International Conference on Software Quality, Reliability and Security, pp. 157–162 (2015)

    Google Scholar 

  35. Arora, A., Chauhan, N.: A regression test selection technique by optimizing user stories in an agile environment. In: 2014 IEEE International Advance Computing Conference (IACC), pp. 1454–1458 (2014)

    Google Scholar 

  36. Fourneret, E., Cantenot, J., Bouquet, F., Legeard, B., Botella, J.: SeTGaM: Generalized technique for regression testing based on UML/OCL Models. In: 2014 Eighth International Conference on Software Security and Reliability (SERE), pp. 147–156 (2014)

    Google Scholar 

  37. Gligoric, M., Negara, S., Legunsen, O., Marinov, D.: An empirical evaluation and comparison of manual and automated test selection. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 361–372. Association for Computing Machinery, New York (2014)

    Google Scholar 

  38. Anderson, J., Salem, S., Do, H.: Improving the effectiveness of test suite through mining historical data. In: Proceedings of the 11th Working Conference on Mining Software Repositories, pp. 142–151. ACM, New York (2014)

    Google Scholar 

  39. Marijan, D., Gotlieb, A., Sen, S.: Test case prioritization for continuous regression testing: an industrial case study. In: 2013 IEEE International Conference on Software Maintenance, pp. 540–543 (2013)

    Google Scholar 

  40. Xu, Z., Liu, Y., Gao, K.: A novel fuzzy classification to enhance software regression testing. In: 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pp. 53–58 (2013)

    Google Scholar 

  41. Rogstad, E., Briand, L., Torkar, R.: Test case selection for black-box regression testing of database applications. Inf. Softw. Technol. 55, 1781–1795 (2013)

    Article  Google Scholar 

  42. Buchgeher, G., Ernstbrunner, C., Ramler, R., Lusser, M.: Towards tool-support for test case selection in manual regression testing. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, pp. 74–79 (2013)

    Google Scholar 

  43. Prakash, N., Rangaswamy, T.R.: Modular based multiple test case prioritization. In: 2012 IEEE International Conference on Computational Intelligence and Computing Research, pp. 1–7 (2012)

    Google Scholar 

  44. Chen, W., et al.: Change impact analysis for large-scale enterprise systems. In: ICEIS (2012)

    Google Scholar 

  45. Srikanth, H., Cohen, M.B.: Regression testing in software as a service: an industrial case study. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM), pp. 372–381 (2011)

    Google Scholar 

  46. Rogstad, E., Briand, L., Dalberg, R., Rynning, M., Arisholm, E.: Industrial experiences with automated regression testing of a legacy database application. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM), pp. 362–371 (2011)

    Google Scholar 

  47. Engström, E., Runeson, P., Ljung, A.: Improving regression testing transparency and efficiency with history-based prioritization – an industrial case study. In: Verification and Validation 2011 Fourth IEEE International Conference on Software Testing, pp. 367–376 (2011)

    Google Scholar 

  48. Juergens, E., Hummel, B., Deissenboeck, F., Feilkas, M., Schlögel, C., Wübbeke, A.: Regression test selection of manual system tests in practice. In: 2011 15th European Conference on Software Maintenance and Reengineering, pp. 309–312 (2011)

    Google Scholar 

  49. Wikstrand, G., Feldt, R., Gorantla, J.K., Zhe, W., White, C.: Dynamic regression test selection based on a file cache an industrial evaluation. In: 2009 International Conference on Software Testing Verification and Validation, pp. 299–302 (2009)

    Google Scholar 

  50. Ramasamy, K., Arul Mary, S.: Incorporating varying requirement priorities and costs in test case prioritization for new and regression testing. In: 2008 International Conference on Computing, Communication and Networking, pp. 1–9 (2008)

    Google Scholar 

  51. Sneed, H.M.: Selective Regression testing of a host to DotNet migration. In: 2006 22nd IEEE International Conference on Software Maintenance, pp. 104–112 (2006)

    Google Scholar 

  52. Skoglund, M., Runeson, P.: A case study of the class firewall regression test selection technique on a large scale distributed software system. In: 2005 International Symposium on Empirical Software Engineering, 2005, p. 10 (2005)

    Google Scholar 

  53. White, L., Robinson, B.: Industrial real-time regression testing and analysis using firewalls. In: 20th IEEE International Conference on Software Maintenance, 2004. Proceedings, pp. 18–27 (2004)

    Google Scholar 

  54. Briand, L.C., Labiche, Y., Soccar, G.: Automating impact analysis and regression test selection based on UML designs. In: International Conference on Software Maintenance, 2002. Proceedings, pp. 252–261 (2002)

    Google Scholar 

Download references

Acknowledgments

The authors thank the Escuela Superior Politécnica de Chimborazo, in particular to the Faculty of Informatics and Electronics for their support in carrying out this work. Also thank anonymous reviewers. César Pardo acknowledges the contribution of the Universidad del Cauca, where he works as an associate professor.

Author information

Authors and Affiliations

  1. GrIISoft Research Group, Escuela Superior Politécnica de Chimborazo, Riobamba, 060155, Ecuador

    Raúl H. Rosero, Omar S. Gómez & Eduardo R. Villa

  2. Facultad de Ingeniería Electrónica Y Telecomunicaciones, Programa de Ingeniería de Sistemas, Grupo de Investigación Y Desarrollo de Tecnologías de La Información (GTI), Universidad del Cauca, Popayán, Colombia

    César J. Pardo

  3. Facultad de Matemáticas, Universidad Autónoma de Yucatán, Mérida, Yucatán, México

    Raúl A. Aguilar

Authors
  1. Raúl H. Rosero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Omar S. Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eduardo R. Villa
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raúl A. Aguilar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. César J. Pardo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raúl H. Rosero .

Editor information

Editors and Affiliations

  1. Eindhoven University of Technology, Guayaquil, Ecuador

    Miguel Botto-Tobar

  2. Escuela Superior Politécnica del Litoral, Riobamba, Chimborazo, Ecuador

    Omar S. Gómez

  3. Escuela Superior Politécnica del Litoral, Riobamba, Chimborazo, Ecuador

    Raul Rosero Miranda

  4. Universitat de Valencia, Valencia, Valencia, Spain

    Angela Díaz Cadena

  5. Universidad de las Fuerzas Armadas (ESPE), Quito, Ecuador

    Sergio Montes León

  6. Escuela Superior Politécnica del Litoral, Riobamba, Chimborazo, Ecuador

    Washington Luna-Encalada

Annex A

Annex A

Table 1. Characterization of the software regression testing approaches used in industry.
Full size table

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rosero, R.H., Gómez, O.S., Villa, E.R., Aguilar, R.A., Pardo, C.J. (2022). Software Regression Testing in Industrial Settings: Preliminary Findings from a Literature Review. In: Botto-Tobar, M., S. Gómez, O., Rosero Miranda, R., Díaz Cadena, A., Montes León, S., Luna-Encalada, W. (eds) Trends in Artificial Intelligence and Computer Engineering. ICAETT 2021. Lecture Notes in Networks and Systems, vol 407. Springer, Cham. https://doi.org/10.1007/978-3-030-96147-3_18

Download citation

Publish with us

Policies and ethics

Search

Navigation