Abstract
An experiment-driven approach to software product and service development is gaining increasing attention as a way to channel limited resources to the efficient creation of customer value. In this approach, software functionalities are developed incrementally and validated in continuous experiments with stakeholders such as customers and users. The experiments provide factual feedback for guiding subsequent development. Although case studies on experimentation in industry exist, the understanding of the state of the practice and the encountered obstacles is incomplete. This paper presents an interview-based qualitative survey exploring the experimentation experiences of ten software development companies. The study found that although the principles of continuous experimentation resonated with industry practitioners, the state of the practice is not yet mature. In particular, experimentation is rarely systematic and continuous. Key challenges relate to changing organizational culture, accelerating development cycle speed, and measuring customer value and product success.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
References
Bosch, J.: Building products as innovation experiment systems. In: Cusumano, M.A., Iyer, B., Venkatraman, N. (eds.) ICSOB 2012. LNBIP, vol. 114, pp. 27–39. Springer, Heidelberg (2012)
Version One: The 8th Annual “State of Agile” Survey. http://www.versionone.com/pdf/2013-state-of-agile-survey.pdf
Holmström Olsson, H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”: a multiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: 38th EUROMICRO Conference on Software Engineering and Advanced Applications SEAA, pp. 392–399. IEEE Press (2012)
Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: 1st International Workshop on Rapid Continuous Software Engineering, pp. 26–35. ACM, New York (2014)
Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York (2011)
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. In: 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1168–1176. ACM, New York (2013)
Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure: more, better, faster experimentation. In: 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 17–26. ACM, New York (2010)
Steiber, A., Alänge, S.: A Corporate System for Continuous Innovation: The Case of Google Inc. European Journal of Innovation Management 16(2), 243–264 (2013)
Amatriain, X.: beyond data: from user information to business value through personalized recommendations and consumer science. In: 22nd ACM International Conference on Information and Knowledge Management, pp. 2201–2208. ACM, New York (2013)
Adams, R.J., Evans, B., Brandt, J.: Creating small products at a big company: adobe’s pipeline innovation process. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 2331–2332. ACM, New York (2013)
Davenport, T.H.: How to Design Smart Business Experiments. Harvard Business Review 87(2), 68–77 (2009)
Holmström Olsson, H., Bosch, J.: Post-deployment data collection in software-intensive embedded products. In: Herzwurm, G., Margaria, T. (eds.) ICSOB 2013. LNBIP, vol. 150, pp. 79–89. Springer, Heidelberg (2013)
Holmström Olsson, H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013)
Münch, J., Fagerholm, F., Johnson, P., Pirttilahti, J., Torkkel, J., Jäarvinen, J.: Creating minimum viable products in industry-academia collaborations. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 137–151. Springer, Heidelberg (2013)
Fink, A.: Analysis of qualitative surveys. In: Fink, A. (ed.) The survey handbook, pp. 61–78. SAGE Publications, California (2003)
Jansen, H.: The Logic of Qualitative Survey Research and its Position in the Field of Social Research Methods. Forum: Qualitative Social Research 11(2), 1–21 (2010)
Runeson, P., Höst, M.: Guidelines for Conducting and Reporting Case Study Research. Empirical Software Engineering 14(2), 131–164 (2009)
Robson, C.: Real World Research: A Resource for Users of Social Research Methods in Applied Settings. Wiley, Chichester (2011)
Lindgren, E., Münch, J.: Interview guide and codebook for the paper “Software Development as an Experiment System”. http://dx.doi.org/10.6084/m9.figshare.1254619
Need for Speed research program (N4S). http://www.n4s.fi
Vinson, N., Singer, J.: A practical guide to ethical research involving humans. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds.) Guide to Advanced Empirical Software Engineering, pp. 229–256. Springer, London (2008)
ATLAS.ti Scientific Software Development GmbH, http://www.atlasti.com
Ståhl, D., Bosch, J.: Modeling Continuous Integration Practice Differences in Industry Software Development. Journal of Systems and Software 87, 48–59 (2014)
Basili, V., Heidrich, J., Lindvall, M., Münch, J., Regardie, M., Rombach, D., Seaman, C., Trendowicz, A.: GQM+Strategies: a comprehensive methodology for aligning business strategies with software measurement. In: DASMA Software Metric Congress (MetriKon 2007): Magdeburger Schriften Zum Empirischen Software Engineering, Kaiserslautern, Germany, pp. 253–266 (2007)
Easterbrook, S., Singer, J., Storey, M., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds.) Guide to Advanced Empirical Software Engineering, pp. 285–311. Springer, London (2008)
Lindgren, E., Münch, J., Männistö, T.: Exploring Software Development as an Experiment System. Master’s Thesis. University of Helsinki (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Lindgren, E., Münch, J. (2015). Software Development as an Experiment System: A Qualitative Survey on the State of the Practice. In: Lassenius, C., Dingsøyr, T., Paasivaara, M. (eds) Agile Processes in Software Engineering and Extreme Programming. XP 2015. Lecture Notes in Business Information Processing, vol 212. Springer, Cham. https://doi.org/10.1007/978-3-319-18612-2_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-18612-2_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-18611-5
Online ISBN: 978-3-319-18612-2
eBook Packages: Computer ScienceComputer Science (R0)