Abstract
In this paper, we describe a testing technique, called structural specification-based testing (SST), which utilizes the formal specification of a program unit as the basis for test selection and test coverage measurement. We also describe an automated testing tool, called ADLscope, which supports SST for program units specified in Sun Microsystems’ Assertion Definition Language (ADL). ADLscope automatically generates coverage conditions from a program’s ADL specification. While the program is tested, ADLscope determines which of these conditions are covered by the tests. An uncovered condition exhibits aspects of the specification inadequately exercised during testing. The tester uses this information to develop new test data to exercise the uncovered conditions.
We provide an overview of SST’s specification-based test criteria and describe the design and implementation of ADLscope. Specification-based testing is guided by a specification, whereby the testing activity is directly related to what a component under test is supposed to do, rather than what it actually does. Specification-based testing is a significant advance in testing, because it is often more straightforward to accomplish and it can reveal failures that are often missed by traditional code-based testing techniques. As an initial evaluation of the capabilities of specification-based testing, we conducted an experiment to measure defect detection capabilities, code coverage and usability of SST/ADLscope; we report here on the results.
This research was sponsored in part by the Air Force Material Command, Rome Laboratory, and the Defense Advanced Research Projects Agency under agreement number #F30602-97-2-0033; and Sun Microsystems Inc. and the University of California under MICRO grants #95-124 and #96-002. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official position or policy, either expressed or implied, of the U.S. Government, AFMC, Rome Laboratory, DARPA, or the University of California, and no official endorsement should be inferred.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Mutation analysis. Technical Report TR GIT-ICS-79/08, Georgia Institute of Technology, September 1979.
Bender and Associates. http://www.softtest.com.
T. A. Budd, R. J. Lipton, R. A. DeMillo, and F.G. Sayward. Mutation analysis. Department Comput. Sci. Res. Rep. 155, Yale University, New Haven, Connecticut, April 1979.
J. Chang. Automating specification-based test coverage measurement and test selection. PhD thesis, University of California, Irvine, March 1999.
J. Chang, D. J. Richardson. ADLscope: an automated specification-based unit testing tool. ICS Technical Report 98-26, Department of Information and Computer Science, University of California, Irvine, California, 1998.
J. Chang, D. J. Richardson, and S. Sankar. Structural specification-based testing with ADL. In Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA’96), pp. 62–70, San Diego, California, January 1996.
R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: help for the practicing programmer. Computer, 4(11), April 1978.
R. K. Doong and P. G. Frankl. The ASTOOT approach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodology, April 1994, pp. 101–130.
K. A. Foster. Error sensitive test case analysis (ESTCA). IEEE Transactions on Software Engineering, 6(3):258–264, May 1980.
K. A. Foster. Sensitive test data for logic expressions. ACM SIGSOFT Software Engineering Notes, vol. 9, no. 2, pp. 120–126, April 1984.
J. B. Goodenough and S. L. Gerhart. Toward a theory of test data selection. IEEE Transactions on Software Engineering, SE-1(2):156–173, June 1975.
James Gosling, Bill Joy, and Guy Steele. The Java Language Specification, Addison Wesley, 1996.
R. G. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering, SE-3(4):279–290, July 1977.
W. E. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering, SE-8(4):371–379, July 1982.
M. Hughes and D. Stotts. Daistish: systematic algebraic testing for OO programs in the presence of side-effects. In Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA), pp.53–61, San Diego, California, January 1996.
P. Jalote. Specification and testing of abstract data types. Comput. Lang. Vol. 17, pp. 75–82, 1992.
David C. Luckham and Friedrich W. von Henke. An overview of ANNA, a specification language for Ada. IEEE Software, 2(2):9–22, March 1985.
Brian Marick. Generic Coverage Tool (GCT) User’s Guide, Testing Foundations, Champaign, Illinois, 1992.
G. J. Myers. The art of software testing. New York: John Wiley and Sons, 1979.
A. Jefferson Offutt. The coupling effect: fact or fiction. In Proceedings of the Third Workshop on Software Testing, Verification, and Analysis, December 1989.
T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6):676–686, June 1988.
Robert M. Poston. Automating Specification-based Software Testing, IEEE, 1996.
D. J. Richardson, S. L. Aha, and T. O. O’Malley. Specification-based test oracles for reactive systems. In Proceedings of the Fourteenth International Conference on Software Engineering, pages 105–118, Melbourne, Australia, May 1992.
D. J. Richardson and L. A. Clarke. Partition analysis: a method combining testing and verification. IEEE Transactions on Software Engineering, SE-11(12):1477–1490, December 1985.
S. Sankar and R. Hayes. Specifying and testing software components using ADL. Technical Report SMLI TR-94-23, Sun Microsystems Laboratories, Inc., Mountain View, California, April 1994.
J. M. Spivey. The Z Notation: a Reference Manual. Prentice Hall, New York, 1989.
P. Stocks and D. Carrington. Test template framework: a specification-based testing case study. In Proceedings of the 1993 International Symposium on Software Testing and Analysis (ISSTA’93), pp. 11–18, Cambridge, Massachusetts, June 1993.
Sun Microsystems Inc., ADL Language Reference Manual, Release 1.1, Mountain View, California, December 1996.
Sun Microsystems Inc. ADLT FTP Download Site. http://ftp.uu.net/vendor/adl.
K. C. Tai. Condition-based software testing strategies. In Proceedings of the 14th Annual International Computer Software and Applications Conference, 1990.
E. Weyuker, T. Goradia, and A. Singh. Automatically generating test data from a boolean specification. IEEE Transactions on Software Engineering, SE-20(5):353–363, May 1994.
E. J. Weyuker and T. J. Ostrand. Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering, SE-6(3):236–246, May 1980.
C. J. Van Wyk. Data structures and C programs. Addison-Wesley, 1988.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1999 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chang, J., Richardson, D.J. (1999). Structural Specification-Based Testing: Automated Support and Experimental Evaluation. In: Nierstrasz, O., Lemoine, M. (eds) Software Engineering — ESEC/FSE ’99. ESEC SIGSOFT FSE 1999 1999. Lecture Notes in Computer Science, vol 1687. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-48166-4_18
Download citation
DOI: https://doi.org/10.1007/3-540-48166-4_18
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-66538-0
Online ISBN: 978-3-540-48166-9
eBook Packages: Springer Book Archive