Abstract
In an era of digital transformation, where in the last decades has increased the need of embrace change rapidly, it’s important that organizations make their transformation in the path of continue being competitive. To this, organizations have started adopting agile methodologies so that they can respond faster to the daily change. DevOps allows companies to gain a leverage in reach their goals and increasing their velocity in react to change. This way, measure DevOps teams’ performance is a bigger and bigger necessity of the organizations so that they can identify improvement points. A Systematic Literature Review was performed and were identified 13 Key Performance Indicators were identified. Among them, those referred to the quality and results of testing are the most referred and implemented.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
MarketLine Company Profile: Tecnotree Oyj. [Internet]. Tecnotree Oyj MarketLine Company Profile. MarketLine, a Progressive Digital Media business, February 2021
Michel, M.: Managing technical debt in the data warehouse. Bus. Intell. J. [Internet]. 22(2), 24–32 (2017)
Gabriel Fernández, J., et al.: SOCIB data infrastructure. Boll di Geofis Teor ed Appl. [Internet]. 2(62), 317–318 (2021)
Zaydi, M., Nassereddine, B.: DevSecOps practices for an agile and secure IT service management. J. Manag. Inf. Decis. Sci. [Internet] 22(4), 527–540 (2019)
Förd\Hos, V., Cesarini, F.: CRDTs for the configuration of distributed erlang systems. In: Proceedings of the 15th International Workshop on Erlang [Internet]. Erlang 2016, pp. 42–53. Association for Computing Machinery, New York (2016)
Yelland, P.: A modern retail forecasting system in production. Foresight Int. J. Appl. Forecast [Internet] 59, 5–15 (2020)
Bulgarelli, A.: The AGILE gamma-ray observatory: software and pipelines: software management approach and lessons learned for the next generation of high-energy observatories. Exp. Astron. [Internet] 48(2/3), 199–231 (2019)
Sharpening the Board’s Cybersecurity Acumen. NACD Dir [Internet], pp. 6–10, 2 July 2019
Kulkarni, G.: Transitioning an enterprise from COBIT 5 to COBIT 2019. COBIT Focus [Internet] 11, 1–9 (2019)
Harris, C.C.: Agile software development: DHS has made significant progress in implementing leading practices, but needs to take additional actions. GAO Reports [Internet], i–107, June 2020
Rosenberg, D., Boehm, B., Wang, B., Qi, K.: Rapid, evolutionary, reliable, scalable system and software development: the resilient agile process. In: Proceedings of the 2017 International Conference on Software and System Process [Internet]. ICSSP 2017, pp. 60–69. Association for Computing Machinery, New York (2017)
Woods, D.D., Allspaw, J.: Revealing the critical role of human performance in software. Commun. ACM [Internet] 63(5), 64–67 (2020)
Storey, M.-A., Zagalsky, A.: Disrupting developer productivity one bot at a time. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering [Internet]. FSE 2016, pp. 928–931. Association for Computing Machinery, New York (2016)
Harty, J.: Designing engineering onboarding for 60+ nationalities. In: Proceedings of the 15th International Conference on Global Software Engineering [Internet]. ICGSE 2020, pp. 76–80. Association for Computing Machinery, New York (2020)
Laanti, M.: Agile transformation model for large software development organizations. In: Proceedings of the XP2017 Scientific Workshops [Internet]. XP 2017. Association for Computing Machinery, New York (2017)
Sánchez-Gordón, M., Colomo-Palacios, R.: Security as culture: a systematic literature review of DevSecOps. In: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops [Internet]. ICSEW 2020, pp. 266–269. Association for Computing Machinery, New York (2020)
Wang, Y., et al.: A self-assessment instrument for assessing test automation maturity. In: Proceedings of the Evaluation and Assessment on Software Engineering [Internet]. EASE 2019, pp. 145–154. Association for Computing Machinery, New York (2019)
Kirikova, M.: Continuous requirements engineering. In: Proceedings of the 18th International Conference on Computer Systems and Technologies [Internet]. CompSysTech 2017, pp. 1–10. Association for Computing Machinery, New York (2017)
de Lacerda, A.R.T., Aguiar, C.S.R.: FLOSS FAQ chatbot project reuse: how to allow nonexperts to develop a chatbot. In: Proceedings of the 15th International Symposium on Open Collaboration [Internet]. OpenSym 2019. Association for Computing Machinery, New York (2019)
Yi, F., Ye, Z., Wei, F.: Data visualization of FADEC software engineering information technology. In: Proceedings of the 2018 International Conference on Computing and Data Engineering [Internet]. ICCDE 2018, pp. 44–48. Association for Computing Machinery, New York (2018)
Bailis, P., Venkataraman, S., Franklin, M.J., Hellerstein, J.M., Stoica, I.: PBS at work: advancing data management with consistency metrics. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data [Internet]. SIGMOD 2013, pp. 1113–1116. Association for Computing Machinery, New York (2013)
Limoncelli, T.A.: The small batches principle. Commun. ACM [Internet] 59(7), 52–57 (2016)
Werner, C., Li, Z.S., Lowlind, D., Elazhary, O., Ernst, N.A., Damian, D.: Continuously managing NFRs: opportunities and challenges in practice. IEEE Trans. Softw. Eng. 1 (2021)
Vassallo, C., Proksch, S., Gall, H.C., Di Penta, M.: Automated reporting of anti-patterns and decay in continuous integration. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 105–115 (2019)
Tutorial and Workshop Summaries. In: 2016 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA), pp. xviii–xxiv (2016)
Margineanu, C., Grigoras, C., Carabas, M., Weisz, S., Mihai, D., Mihailescu, M.-E., et al.: Client request analysis tool for CERN ALICE grid services. In: 2020 International Conference on Computing and Data Science (CDS), pp. 462–467 (2020)
Kerzazi, N., Adams, B.: Botched releases: do we need to roll back? Empirical study on a commercial web app. In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 574–583 (2016)
Rosser, L.A., Norton, J.H.: A systems perspective on technical debt. In: 2021 IEEE Aerospace Conference (50100), pp. 1–10 (2021)
Lopez, L., Bagnato, A., Ahberve, A., Franch, X.: QFL: data-driven feedback loop to manage quality in agile development. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS), pp. 58–66 (2021)
Liechti, O., Pasquier, J., Reis, R.: Supporting agile teams with a test analytics platform: a case study. In: 2017 IEEE/ACM 12th International Workshop on Automation of Software Testing (AST), pp. 9–15 (2017)
Pailwan, A., Abraham, J., Saraf, M.: Landscape of monitoring and visualization of technologies in DevOps for classification and prediction. In: 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 581–588 (2020)
Marnewick, C., Langerman, J.: DevOps and organizational performance: the fallacy of chasing maturity. IEEE Softw. 38(5), 48–55 (2021)
Telemaco, U., Oliveira, T., Alencar, P., Cowan, D.: A catalogue of agile smells for agility assessment. IEEE Access. 8, 79239–79259 (2020)
Zamfir, V., Carabas, M., Carabas, C., Tapus, N.: Systems monitoring and big data analysis using the elasticsearch system. In: 2019 22nd International Conference on Control Systems and Computer Science (CSCS), pp. 188–193 (2019)
Lundqvist, K.Ø., Warburton, S.: Visualising learning pathways in MOOCs. In: 2019 IEEE Learning with MOOCS (LWMOOCS), pp. 185–190 (2019)
Raygan, R.E., Henry, S.: Manifesto for enterprise agility. In: 2019 International Symposium on Systems Engineering (ISSE), pp. 1–6 (2019)
Syed-Mohamad, S.M., Akhir, N.S.M.: SoReady: an extension of the test and defect coverage-based analytics model for pull-based software development. In: 2019 26th Asia-Pacific Software Engineering Conference (APSEC), pp. 9–14 (2019)
Chen, H., Kazman, R., Haziyev, S.: Agile big data analytics for web-based systems: an architecture-centric approach. IEEE Trans. Big Data 2(3), 234–248 (2016)
Hussaini, S.W.: Strengthening harmonization of development (Dev) and operations (Ops) silos in IT environment through systems approach. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 178–183 (2014)
Sangiampak, J., et al.: LockerSwarm: an IoT-based smart locker system with access sharing. In: 2019 IEEE International Smart Cities Conference (ISC2), pp. 587–592 (2019)
Amershi, S., et al.: Software engineering for machine learning: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019)
Balalaie, A., Heydarnoori, A., Jamshidi, P.: Microservices architecture enables DevOps: migration to a cloud-native architecture. IEEE Softw. 33(3), 42–52 (2016)
López-Peña, M.A., Díaz, J., Pérez, J.E., Humanes, H.: DevOps for IoT systems: fast and continuous monitoring feedback of system availability. IEEE Internet Things J. 7(10), 10695–10707 (2020)
Ludwig, J., Cline, D., Novstrup, A.: A case study using CBR-insight to visualize source code quality. In: 2020 IEEE Aerospace Conference, pp. 1–12 (2020)
Cito, J., Oliveira, F., Leitner, P., Nagpurkar, P., Gall, H.C.: Context-based analytics - establishing explicit links between runtime traces and source code. In: 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp. 193–202 (2017)
Snyder, B., Curtis, B.: Using analytics to guide improvement during an Agile–DevOps transformation. IEEE Softw. 35(1), 78–83 (2018)
Jana, D., Pal, P.: ESSENCE kernel in overcoming challenges of agile software development. In: 2020 IEEE 17th India Council International Conference (INDICON), pp. 1–8 (2020)
Kalinowski, M., et al.: Towards lean R&D: an agile research and development approach for digital transformation. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 132–136 (2020)
Díaz, J., Pérez, J.E., Lopez-Peña, M.A., Mena, G.A., Yagüe, A.: Self-service cybersecurity monitoring as enabler for DevSecOps. IEEE Access 7, 100283–100295 (2019)
Rafi, S., Yu, W., Akbar, M.A., Alsanad, A., Gumaei, A.: Multicriteria based decision making of DevOps data quality assessment challenges using fuzzy TOPSIS. IEEE Access 8, 46958–46980 (2020)
McCarthy, M.A., Herger, L.M., Khan, S.M., Belgodere, B.M.: Composable DevOps: automated ontology based DevOps maturity analysis. In: 2015 IEEE International Conference on Services Computing, pp. 600–607 (2015)
Bernsmed, K., Jaatun, M.G.: Threat modelling and agile software development: identified practice in four Norwegian organisations. In: 2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security), pp. 1–8 (2019)
Bansal, C., Renganathan, S., Asudani, A., Midy, O., Janakiraman, M.: DeCaf: diagnosing and triaging performance issues in large-scale cloud services. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 201–210 (2020)
Wang, Y., Pyhäjärvi, M., Mäntylä, M.V.: Test automation process improvement in a DevOps team: experience report. In: 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 314–321 (2020)
Pina, F., Correia, J., Filipe, R., Araujo, F., Cardroom, J.: Nonintrusive monitoring of microservice-based systems. In: 2018 IEEE 17th International Symposium on Network Computing and Applications (NCA), pp. 1–8 (2018)
Martínez-Fernández, S., et al.: Continuously assessing and improving software quality with software analytics tools: a case study. IEEE Access. 7, 68219–68239 (2019)
Ahlgren, J., et al.: Testing web enabled simulation at scale using metamorphic testing. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 140–149 (2021)
Beck, F., Lahmadi, A., François, J.: HSL: a cyber security research facility for sensitive data experiments. In: 2021 IFIP/IEEE International Symposium on Integrated Network Management (IM), pp. 956–961 (2021)
Renzel, D., Koren, I., Klamma, R., Jarke, M.: Preparing research projects for sustainable software engineering in society. In: 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Society Track (ICSE-SEIS), pp. 23–32 (2017)
Tomas, N., Li, J., Huang, H.: An empirical study on culture, automation, measurement, and sharing of DevSecOps. In: 2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security), pp. 1–8 (2019)
Farroha, B.S., Farroha, D.L.: A framework for managing mission needs, compliance, and trust in the DevOps environment. In: 2014 IEEE Military Communications Conference, pp. 288–293 (2014)
Saltz, J.S., Shamshurin, I.: Achieving agile big data science: the evolution of a team’s agile process methodology. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 3477–3485 (2019)
Szabó, D.M., Steghöfer, J.-P.: Coping strategies for temporal, geographical and sociocultural distances in agile GSD: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 161–170 (2019)
Huijgens, H., van Deursen, A., van Solingen, R.: Success factors in managing legacy system evolution: a case study. In: 2016 IEEE/ACM International Conference on Software and System Processes (ICSSP), pp. 96–105 (2016)
Mayer, B., Weinreich, R.: A dashboard for microservice monitoring and management. In: 2017 IEEE International Conference on Software Architecture Workshops (ICSAW), pp. 66–69 (2017)
Sukhwani, H., Matias, R., Trivedi, K.S., Rindos, A.: Monitoring and mitigating software aging on IBM cloud controller system. In: 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 266–272 (2017)
Rathod, N., Surve, A.: Test orchestration a framework for continuous integration and continuous deployment. In: 2015 International Conference on Pervasive Computing (ICPC), pp. 1–5 (2015)
Nogueira, A.F., Ribeiro, J.C.B., Zenha-Rela, M.A., Craske, A.: Improving La Redoute’s CI/CD pipeline and DevOps processes by applying machine learning techniques. In: 2018 11th International Conference on the Quality of Information and Communications Technology (QUATIC), pp. 282–286 (2018)
Baxley, R.J., Thompson, R.S.: Team Zylinium DARPA spectrum collaboration challenge radio design and implementation. In: 2019 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), pp. 1–6 (2019)
Gupta, R.K., Venkatachalapathy, M., Jeberla, F.K.: Challenges in adopting continuous delivery and DevOps in a globally distributed product team: a case study of a healthcare organization. In: 2019 ACM/IEEE 14th International Conference on Global Software Engineering (ICGSE), pp. 30–4 (2019)
Neely, S., Stolt, S.: Continuous delivery? Easy! just change everything (well, maybe it is not that easy). In: 2013 Agile Conference, pp. 121–128 (2013)
Pedretti, K., et al.: Chronicles of astra: challenges and lessons from the first petascale arm supercomputer. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–14 (2020)
Atwood, C.A., et al.: Secure web-based access for productive supercomputing. Comput. Sci. Eng. 18(1), 63–72 (2016)
Ostakhov, V., Artykulna, N., Morozov, V.: Models of IT projects KPIs and metrics. In: 2018 IEEE Second International Conference on Data Stream Mining and Processing (DSMP), pp. 50–55 (2018)
Angara, J., Prasad, S.: Continuous testing real-time health analytics dashboard. Int. J. Adv. Trends Comput. Sci. Eng. [Internet] 9(2), 1713–1719 (2020)
Lopez, L., et al.: QaSD: a quality-aware strategic dashboard for supporting decision makers in agile software development. Sci. Comput. Program, 202 (2021)
Janes, A., Succi, G.: Lean software development in action [Internet]. In: Lean Software Development in Action, vol. 9783642005, pp. 1–393 (2014)
De Gyves, A.S., Ortegon, C.P., Mondragon, M.A., Solis, M.I, Navarro, L.A., Zagal, D.G.E.: A data analysis platform to evaluate performance during software development process. RISTI - Rev Iber Sist e Tecnol. Inf. [Internet] 2020(36) (2020)
Patwardhan, A.: Sentiment identification for collaborative, geographically dispersed, cross-functional software development teams. In: Proceedings - 2017 IEEE 3rd International Conference on Collaboration and Internet Computing, CIC 2017 [Internet], pp. 20–26 (2017)
Rocha, T., Borba, P., Santos, J.P.: Using acceptance tests to predict files changed by programming tasks. J. Syst. Softw. [Internet] 154, 176–195 (2019)
Mahdavi-Hezaveh, R., Dremann, J., Williams, L.: Software development with feature toggles: practices used by practitioners. Empir. Softw. Eng. 26(1), 1–33 (2021). https://doi.org/10.1007/s10664-020-09901-z
Adalı, O.E., Özcan-Top, Ö., Demirörs, O.: Evaluation of agility assessment tools: a multiple case study. In: Clarke, P.M., O’Connor, R.V., Rout, T., Dorling, A. (eds.) SPICE 2016. CCIS, vol. 609, pp. 135–149. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-38980-6_11
Bobbert, Y., Scheerder, J.: On the design and engineering of a zero trust security artefact. In: Arai, K. (ed.) FICC 2021. AISC, vol. 1363, pp. 830–848. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73100-7_58
Conde, M.Á., Sarasa-Cabezuelo, A., Sierra, J.-L.: 9th International Workshop on Software Engineering for ELearning (ISELEAR 2018). In: Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality [Internet]. TEEM 2018, pp. 879–882. Association for Computing Machinery, New York (2018)
Kanoulas, E., Karlgren, J.: Practical issues in information access system evaluation. SIGIR Forum [Internet] 51(1), 67–72 (2017)
Van Heesch, U., Theunissen, T., Zimmermann, O., Zdun, U.: Software specification and documentation in continuous software development: a focus group report. In: Proceedings of the 22nd European Conference on Pattern Languages of Programs [Internet]. EuroPLoP 2017. Association for Computing Machinery, New York (2017)
Barcellos, M.P.: Towards a framework for continuous software engineering. In: Proceedings of the 34th Brazilian Symposium on Software Engineering [Internet]. SBES 2020, pp. 626–631. Association for Computing Machinery, New York (2020)
Janes, A., Lenarduzzi, V., Stan, A.C.: A continuous software quality monitoring approach for small and medium enterprises. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion [Internet]. ICPE 2017 Companion, pp. 97–100. Association for Computing Machinery, New York (2017)
Yan, S., Ramachandran, P.G.: The current status of accessibility in mobile apps. ACM Trans. Access Comput. [Internet] 12(1) (2019)
Munappy, A.R., Mattos, D.I., Bosch, J., Olsson, H.H., Dakkak, A.: From ad-hoc data analytics to DataOps. In: Proceedings of the International Conference on Software and System Processes [Internet]. ICSSP 2020, pp. 165–174. Association for Computing Machinery, New York (2020)
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., Butler, J.: The SPACE of developer productivity: there’s more to it than you think. Queue [Internet] 19(1), 20–48 (2021)
Huijgens, H., Lamping, R., Stevens, D., Rothengatter, H., Gousios, G., Romano, D.: Strong agile metrics: mining log data to determine predictive power of software metrics for continuous delivery teams. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering [Internet]. ESEC/FSE 2017, pp. 866–871. Association for Computing Machinery, New York (2017)
Mohsen, W., Aref, M., El Bahnasy, K.: Scaled scrum framework for cooperative domain ontology evolution. In: Proceedings of the 2020 6th International Conference on Frontiers of Educational Technologies [Internet]. ICFET 2020, pp. 135–143. Association for Computing Machinery, New York (2020)
Abdelkebir, S., Maleh, Y., Belaissaoui, M.: An agile framework for ITS management in organizations: a case study based on DevOps. In: Proceedings of the 2nd International Conference on Computing and Wireless Communication Systems [Internet]. ICCWCS 2017. Association for Computing Machinery, New York (2017)
Sürücü, C., Song, B., Krüger, J., Saake, G., Leich, T.: Establishing key performance indicators for measuring software-development processes at a large organization. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering [Internet]. ESEC/FSE 2020, pp. 1331–1341. Association for Computing Machinery, New York (2020)
Huijgens, H., Spadini, D., Stevens, D., Visser, N., van Deursen, A.: Software analytics in continuous delivery: a case study on success factors. In: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement [Internet]. ESEM 2018. Association for Computing Machinery, New York (2018)
Than, P.P., Phyu, M.P.: Continuous integration for Laravel applications with GitLab. In: Proceedings of the International Conference on Advanced Information Science and System [Internet]. AISS 2019. Association for Computing Machinery, New York (2019)
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering – a systematic literature review. Inf. Softw. Technol. 51(1), 7–15 (2009)
Prates, L., Faustino, J., Silva, M., Pereira, R.: DevSecOps metrics. In: Information Systems: Research, Development, Applications, Education, pp. 77–90 (2019)
Silva, M.A., Faustino, J.P., Pereira, R., da Silva, M.M.: Productivity gains of DevOps adoption in an IT team: a case study (2018)
Teixeira, D., Pereira, R., Henriques, T., da Silva, M.M., Faustino, J., Silva, M.: A maturity model for DevOps. Int. J. Agil. Syst. Manag. 13(4), 464–511 (2020)
Teixeira, D., Pereira, R., Henriques, T.A., Silva, M., Faustino, J.: A systematic literature review on DevOps capabilities and areas. Int. J. Hum. Cap. Inf. Technol. Prof. 11(3), 1–22 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gomes, M., Pereira, R., Silva, M., de Vasconcelos, J.B., Rocha, Á. (2022). KPI’s for Evaluation of DevOps Teams. In: Rocha, A., Adeli, H., Dzemyda, G., Moreira, F. (eds) Information Systems and Technologies. WorldCIST 2022. Lecture Notes in Networks and Systems, vol 470. Springer, Cham. https://doi.org/10.1007/978-3-031-04829-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-04829-6_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04828-9
Online ISBN: 978-3-031-04829-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)