Abstract
Covering arrays have been widely used to detect the presence of faults in large software and hardware systems. Indeed, finding failures that result from faulty interactions requires that all interactions that may cause faults be covered by a test case. However, finding the actual faults requires more, because the failures resulting from two potential sets of faults must not be the same. The combinatorial requirements on test suites to enable a tester to locate the faults are developed, and set in the context of similar combinatorial search questions. Test suites known as locating and detecting arrays to locate faults both in principle and in practice generalize covering arrays, thereby addressing combinatorial fault characterization. In common with covering arrays, these locating and detecting arrays scale logarithmically in size with the number of factors, but unlike covering arrays they support complete characterization of the interactions that underlie faults.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aldaco, A.N., Colbourn, C.J., Syrotiuk, V.R.: Locating arrays: a new experimental design for screening complex engineered systems. SIGOPS Oper. Syst. Rev. 49(1), 31–40 (2015)
Alon, N., Spencer, J.H.: The Probabilistic Method. Wiley-Interscience Series in Discrete Mathematics and Optimization, 3rd edn. Wiley, Hoboken (2008)
Bach, J., Schroeder, P.J.: Pairwise testing: a best practice that isn’t. In: 22nd Annual Pacific Northwest Software Quality Conference, pp. 180–196 (2004)
van den Berg, E., Candès, E., Chinn, G., Levin, C., Olcott, P.D., Sing-Long, C.: Single-photon sampling architecture for solid-state imaging sensors. Proc. Natl. Acad. Sci. 110(30), E2752–E2761 (2013)
Bryce, R.C., Colbourn, C.J.: A density-based greedy algorithm for higher strength covering arrays. Softw. Test. Verif. Reliab. 19, 37–53 (2009)
Bryce, R.C., Colbourn, C.J.: Expected time to detection of interaction faults. J. Comb. Math. Comb. Comput. 86, 87–110 (2013)
Bryce, R.C., Colbourn, C.J., Kuhn, D.R.: Finding interaction faults adaptively using distance-based strategies. In: 2011 18th IEEE International Conference and Workshops on Engineering of Computer Based Systems (ECBS), pp. 4–13 (2011)
Bshouty, N.H., Costa, A.: Exact learning of juntas from membership queries. Lect. Notes Artif. Intell. 9925, 115–129 (2016)
Chandrasekaran, J., Ghandehari, L.S., Lei, Y., Kacker, R., Kuhn, D.R.: Evaluating the effectiveness of BEN in localizing different types of software fault. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 26–34 (2016)
Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)
Chen, T.Y., Kuo, F.C., Merkel, R.G., Tse, T.: Adaptive random testing: the art of test case diversity. J. Syst. Softw. 83(1), 60–66 (2010)
Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: 9th Asian Computing Science Conference Advances in Computer Science—ASIAN 2004, Higher-Level Decision Making, pp. 320–329 (2004)
Cheng, C.S., Tang, B.: Upper bounds on the number of columns in supersaturated designs. Biometrika 88(4), 1169–1174 (2001)
Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23, 437–44 (1997)
Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34, 633–650 (2008)
Colbourn, C.J., Fan, B.: Locating one pairwise interaction: three recursive constructions. J. Algebra Comb. Discrete Struct. Appl. 3, 125–134 (2016)
Colbourn, C.J., Fan, B., Horsley, D.: Disjoint spread systems and fault location. SIAM J. Discrete Math. 30, 2011–2016 (2016)
Colbourn, C.J., Lanus, E., Sarkar, K.: Asymptotic and constructive methods for covering perfect hash families and covering arrays. Designs Codes Cryptogr. 86, 907–937 (2018)
Colbourn, C.J., McClary, D.W.: Locating and detecting arrays for interaction faults. J. Comb. Optim. 15, 17–48 (2008)
Colbourn, C.J., Syrotiuk, V.R.: Coverage, location, detection, and measurement. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 19–25. IEEE Press (2016)
Compton, R., Mehari, M.T., Colbourn, C.J., De Poorter, E., Syrotiuk, V.R.: Screening interacting factors in a wireless network testbed using locating arrays. In: IEEE INFOCOM International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT) (2016)
Dalal, S.R., Mallows, C.L.: Factor-covering designs for testing software. Technometrics 40, 234–243 (1998)
Damaschke, P.: Adaptive versus nonadaptive attribute-efficient learning. Mach. Learn. 41, 197–215 (2000)
Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47, 2845–2862 (2001)
Du, D.Z., Hwang, F.K.: Combinatorial Group Testing and Its Applications, 2nd edn. World Scientific Publishing Co., Inc., River Edge (2000)
Erdős, P., Lovász, L.: Problems and results on 3-chromatic hypergraphs and some related questions. In: Infinite and finite sets (Colloq., Keszthely, 1973 Vol. II, pp. 609–627. Colloq. Math. Soc. János Bolyai, Vol. 10. North-Holland, Amsterdam (1975)
Francetić, N., Stevens, B.: Asymptotic size of covering arrays: an application of entropy compression. J. Comb. Des 25, 243–257 (2017)
Frankl, P.G., Hamlet, R.G., Littlewood, B., Strigini, L.: Evaluating testing methods by delivered reliability. IEEE Trans. Softw. Eng. 24(8), 586–601 (1998)
Ghandehari, L.S., Chandrasekaran, J., Lei, Y., Kacker, R., Kuhn, D.R.: BEN: a combinatorial testing-based fault localization tool. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–4 (2015)
Ghosh, S., Shirakura, T., Srivastava, J.N.: Model identification using search linear models and search designs. In: Entropy, search, complexity, Bolyai Soc. Math. Stud., vol. 16, pp. 85–112. Springer, Berlin (2007)
Godbole, A.P., Skipper, D.E., Sunley, R.A.: \(t\)-covering arrays: upper bounds and Poisson approximations. Comb. Probab. Comput. 5, 105–118 (1996)
Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies—a survey. Softw. Test. Verif. Reliab. 5, 167–199 (2005)
Hamlet, D., Taylor, R.: Partition testing does not inspire confidence (program testing). IEEE Trans. Softw. Eng. 16(12), 1402–1411 (1990)
Hartman, A.: Software and hardware testing using combinatorial covering suites. In: Golumbic, M.C., Hartman, I.B.A. (eds.) Interdisciplinary Applications of Graph Theory, Combinatorics, and Algorithms, pp. 237–266. Springer, Norwell (2005)
Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discrete Math. 284, 149–156 (2004)
Jin, H., Tsuchiya, T.: Constrained locating arrays for combinatorial interaction testing. CoRR arXiv:1801.06041 (2018)
Johnson, D.S.: Approximation algorithms for combinatorial problems. J. Comput. Syst. Sci. 9, 256–278 (1974)
Jones, B., Majumdar, D.: Optimal supersaturated designs. J. Am. Stat. Assoc. 109(508), 1592–1600 (2014)
Konishi, T., Kojima, H., Nakagawa, H., Tsuchiya, T.: Finding minimum locating arrays using a SAT solver. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2017, Tokyo, Japan, March 13–17, 2017, pp. 276–277 (2017)
Kuhn, D.R., Bryce, R.C., Duan, F., Ghandehari, L.S.G., Lei, Y., Kacker, R.N.: Combinatorial testing: theory and practice. Adv. Comput. 99, 1–66 (2015)
Kuhn, D.R., Kacker, R., Lei, Y.: Introduction to Combinatorial Testing. CRC Press, Boca Raton (2013)
Kuhn, D.R., Kacker, R., Lei, Y.: Combinatorial coverage as an aspect of test quality. Crosstalk pp. 19–23 (2015)
Kuhn, D.R., Mendoza, I.D., Kacker, R., Lei, Y.: Combinatorial coverage measurement concepts and applications. In: Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013 Workshops Proceedings, Luxembourg, Luxembourg, March 18–22, 2013, pp. 352–361 (2013)
Kuhn, D.R., Reilly, M.: An investigation of the applicability of design of experiments to software testing. In: Proceedings of 27th Annual NASA Goddard/IEEE Software Engineering Workshop, pp. 91–95. IEEE, Los Alamitos, CA (2002)
Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30, 418–421 (2004)
Li, P.C., Meagher, K.: Sperner partition systems. J. Comb. Des. 21(7), 267–279 (2013)
Lovász, L.: On the ratio of optimal integral and fractional covers. Discrete Math. 13(4), 383–390 (1975)
Martínez, C., Moura, L., Panario, D., Stevens, B.: Locating errors using ELAs, covering arrays, and adaptive testing algorithms. SIAM J. Discrete Math. 23, 1776–1799 (2009-2010)
Meagher, K., Moura, L., Stevens, B.: A Sperner-type theorem for set-partition systems. Electron. J. Combin. 12, Note 20 (electronic) (2005)
Montgomery, D.C.: Design and Analysis of Experiments, 8th edn. Wiley, New York (2012)
Moser, R.A., Tardos, G.: A constructive proof of the general Lovász local lemma. J. ACM 57(2), Art. 11, 15 (2010)
Nagamoto, T., Kojima, H., Nakagawa, H., Tsuchiya, T.: Locating a faulty interaction in pair-wise testing. In: 20th IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2014, Singapore, November 18–21, 2014, pp. 155–156 (2014)
Nayeri, P., Colbourn, C.J., Konjevod, G.: Randomized postoptimization of covering arrays. Eur. J. Comb. 34, 91–103 (2013)
Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), #11 (2011)
Nie, C., Wu, H., Niu, X., Kuo, F., Leung, H.K.N., Colbourn, C.J.: Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures. Inf. Softw. Technol. 62, 198–213 (2015)
Petke, J.: Constraints: The future of combinatorial interaction testing. In: 2015 IEEE/ACM 8th International Workshop on Search-Based Software Testing (SBST), pp. 17–18 (2015)
Petke, J., Cohen, M.B., Harman, M., Yoo, S.: Practical combinatorial interaction testing: empirical findings on efficiency and early fault detection. IEEE Trans. Softw. Eng. 41(9), 901–924 (2015)
Sarkar, K., Colbourn, C.J.: Upper bounds on the size of covering arrays. SIAM J. Discrete Math. 31, 1277–1293 (2017)
Schroeder, P.J., Bolaki, P., Gopu, V.: Comparing the fault detection effectiveness of n-way and random test suites. In: Proceedings of International Symposium on Empirical Software Engineering (ISESE04), pp. 49–59 (2004)
Seidel, S.A., Mehari, M.T., Colbourn, C.J., De Poorter, E., Moerman, I., Syrotiuk, V.R.: Analysis of large-scale experimental data from wireless networks. In: IEEE INFOCOM International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT) (2018)
Seidel, S.A., Sarkar, K., Colbourn, C.J., Syrotiuk, V.R.: Separating interaction effects using locating and detecting arrays. In: International Workshop on Combinatorial Algorithms (2018)
Shi, C., Tang, Y., Yin, J.: Optimal locating arrays for at most two faults. Sci. China Math. 55(1), 197–206 (2012)
Stein, S.K.: Two combinatorial covering theorems. J. Comb. Theory Ser. A 16, 391–397 (1974)
Tai, K.C., Yu, L.: A test generation strategy for pairwise testing. IEEE Trans. Softw. Eng. 28, 109–111 (2002)
Tang, Y., Colbourn, C.J., Yin, J.: Optimality and constructions of locating arrays. J. Stat. Theory Pract. 6(1), 20–29 (2012)
Torres-Jimenez, J., Rodriguez-Tello, E.: New upper bounds for binary covering arrays using simulated annealing. Inf. Sci. 185(1), 137–152 (2012)
Yilmaz, C., Cohen, M.B., Porter, A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Softw. Eng. 31, 20–34 (2006)
Yilmaz, C., Dumlu, E., Cohen, M.B., Porter, A.: Reducing masking effects in combinatorial interaction testing: A feedback driven adaptive approach. IEEE Trans. Softw. Eng. 40(1), 43–66 (2014)
Zhang, Y.: On theory of compressive sensing via \(\ell _1\)-minimization: Simple derivations and extensions. Tech. Rep. CAAM TR08-11, Rice University (2008)
Zhou, W., Zhang, D.: Sole error locating array and approximate error locating array. In: 2012 IEEE International Conference on Computer Science and Automation Engineering, pp. 480–483 (2012)
Author information
Authors and Affiliations
Corresponding author
Additional information
Research of CJC was supported in part by the Software Test & Analysis Techniques for Automated Software Test program supported by OPNAV N-84, United States Navy. Research of CJC and VRS was supported in part by the National Science Foundation under Grant No. 1421058.
Rights and permissions
About this article
Cite this article
Colbourn, C.J., Syrotiuk, V.R. On a Combinatorial Framework for Fault Characterization. Math.Comput.Sci. 12, 429–451 (2018). https://doi.org/10.1007/s11786-018-0385-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11786-018-0385-x
Keywords
- Covering array
- Combinatorial testing
- Locating array
- Detecting array
- Compressive sensing
- Experimental design