Abstract
This paper presents results of a literature analysis on Empirical Research Approaches in Software Engineering (SE). The analysis explores reasons why traditional methods, such as statistical hypothesis testing and experiment replication are weakly utilized in the field of SE. It appears that basic assumptions and preconditions of the traditional methods are contradicting the actual situation in the SE. Furthermore, we have identified main issues that should be considered by the researcher when selecting the research approach. In virtue of reasons for weak utilization of traditional methods we propose stronger use of Multi-Method approach with Pragmatism as the philosophical standpoint.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
References
Shaw, M.: What makes good research in software engineering? International Journal of Software Tools for Technology Transfer 4(1), 1–7 (2002)
Kitchenham, B.: Guidelines for performing systematic literature reviews in software engineering. Technical report, TR-EBSE-2007-01, UK (2007)
Juristo, N., Moreno, A.: Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Dordrecht (2003)
Wohlin, C., Runeson, P., Horst, M., Ohlsson, M., Regnell, B., Wesslen, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, Dordrecht (2000)
Turabian, K.: A Manual for Writers of Research Papers, Theses, and Dissertations. The University of Chicago Press, Chicago (2007)
Fenton, N., Pfleeger, S., Glass, R.: Science and substance: A challenge to software engineers. IEEE Software 4(11), 86–95 (1994)
Basili, V., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25(4), 456–473 (1999)
Oivo, M.: New opportunities for empirical research. In: Basili, V.R., Rombach, H.D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Issues. LNCS, vol. 4336, p. 22. Springer, Heidelberg (2007)
Oivo, M., Kuvaja, P., Pulli, P., Similä, J.: Software engineering research strategy: Combining experimental and explorative research (eer). In: Bomarius, F., Iida, H. (eds.) PROFES 2004. LNCS, vol. 3009, pp. 302–317. Springer, Heidelberg (2004)
Basili, V.: The role of experimentation in software engineering: Past, current, and future. In: 18th International Conference on Software Engineering, pp. 442–449. IEEE, Berlin (1996)
Basili, V.: Measurement and model building, introduction. In: Basili, V.R., Rombach, H.D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Issues. LNCS, vol. 4336, pp. 68–69. Springer, Heidelberg (2007)
Vegas, S., Basili, V.: Measurement and model building, discussion and summary. In: Basili, V.R., Rombach, H.D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Issues. LNCS, vol. 4336, pp. 115–120. Springer, Heidelberg (2007)
Curtis, B.: Measurement and experimentation in software engineering. In: Proceedings of IEEE, pp. 1144–1157. IEEE, Los Alamitos (1980)
Miller, J.: Statistical significance testing – a panacea for software technology experiments? Journal of Systems and Software 2(73), 183–192 (2004)
Zelkowitz, M., Wallance, D.: Experimental validation in software engineering. Information and Software Technology 11(39), 735–743 (1997)
Basili, V., Selby, R., Hutchens, D.: Experimentation in software engineering. IEEE Transactions on Software Engineering 12(7), 733–743 (1986)
Endres, A., Rombach, D.: A Handbook of Software and Systems Engineering: Empirical Observations, Laws and Theories. Pearson Education, Harlow (2003)
Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., et al.: Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 8(28), 721–734 (2002)
Cook, T., Campbell, D.: Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin Company, USA (1979)
Miller, J., Daly, J., Wood, M., Roper, M., Brooks, A.: Statistical power and its subcomponents - missing and misunderstood concepts in empirical software engineering research. Information and Software Technology 4(39), 285–295 (1997)
Dybå, T., Kampenes, V., Sjøberg, D.: A systematic review of statistical power in software engineering experiments. Information and Software Technology 8(48), 745–755 (2006)
Miller, J.: Replicating software engineering experiments: a poisoned chalice or the holy grail. Information and Software Technology 4(47), 233–244 (2005)
Wood, M., Daly, J., Miller, J., Roper, M.: Multi-method research: An empirical investigation of object-oriented technology. Journal of Systems and Software 1(48), 13–26 (1999)
Popper, K.: The Logic of Scientific Discovery. Routledge Classics, New York (1959)
Ramesh, V., Glass, R., Vessey, I.: Research in computer science: an empirical study. Journal of systems and Software 2(70), 165–176 (2004)
Sjøberg, D., Hannay, J., Hansen, O., By Kampenes, V., Karahasanovic, A., Liborg, N.K., et al.: A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31(9), 733–753 (2005)
Brooks, A., Roper, M., Wood, M., Daly, J., Miller, J.: Replication’s role in software engineering. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering, pp. 365–379. Springer, London (2008)
Basili, V., Selby, R.: Paradigms for experimentation and empirical studies in software engineering. Reliability Engineering and System Safety 1(32), 171–191 (1991)
Lott, C., Rombach, D.: Repeatable software engineering experiments for comparing defect-detection techniques. Empirical Software Engineering 1(3), 241–277 (1996)
Jedlitschka, A., Ciolkowski, M.: Reporting experiments in software engineering. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering, pp. 201–228. Springer, London (2007)
Easterbrook, S., Singer, J., Storey, M.A., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering, pp. 285–311. Springer, London (2008)
Creswell, J.: Research Design: Qualitative, Quantitative, and Mixed Method Approaches. Sage Publications, Inc., London (2008)
Petter, S., Gallivan, M.: Toward a framework for classifying and guiding mixed method research in information systems. In: The 37th Hawaii International Conference on System Sciences, Big Island, HI, USA, pp. 1–10 (2004)
Nunamaker, J., Chen, M., Purdin, T.: Systems development in information systems research. Journal of Management Information Systems 7(3), 89–106 (1991)
Galliers, R.: Research issues in information systems. Journal of Information Technology 2(8), 92–98 (1993)
Sawyer, S.: Studying organizational computing infrastructures: Multi-method approaches. In: Baskerville, R., et al. (eds.) Organizational and Social Perspectives on Information Technology, IFIP TC8 WG8.2 International Working Conference on the Social and Organizational Perspective on Research and Practice in Information Technology, pp. 213–232. Kluwer, Aalborg (2000)
Mingers, J.: Combining is research methods: Towards a pluralist methodology. Information Systems Research 12(3), 240–259 (2001)
Steinfield, C., Huysman, M., David, K., Yang Jang, C., Poot, J., Huis in ’t Veld, M., et al.: New methods for studying global virtual teams: Towards a multi-faceted approach, Wailea Maui, Hawaii, USA. In: The 34th Hawaii International Conference on System Sciences 2001, pp. 1–10 (2001)
Howe, K.: Against the quantitative-qualitative incompatibility thesis. Educational Researcher 17(8), 10–16 (1998)
Bratthall, L., Jørgensen, M.: Can you trust a single data source exploratory software engineering case study? Empirical Software Engineering 7(1), 9–26 (2002)
Daly, J., El Emam, K., Miller, J.: An empirical research methodology for software process improvement. In: El Emam, K., et al. (eds.) Elements of Software Process Assessment and Improvement. Wiley-IEEE Computer Society Press, London (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mandić, V., Markkula, J., Oivo, M. (2009). Towards Multi-Method Research Approach in Empirical Software Engineering. In: Bomarius, F., Oivo, M., Jaring, P., Abrahamsson, P. (eds) Product-Focused Software Process Improvement. PROFES 2009. Lecture Notes in Business Information Processing, vol 32. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02152-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-02152-7_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02151-0
Online ISBN: 978-3-642-02152-7
eBook Packages: Computer ScienceComputer Science (R0)