Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Developing software systems or services within budget and schedule requires an elaborated decision-making process supported by risk information [1, 2]. For instance, competent decisions on what to release [3, 4] or on how much to test [5, 6] are supported by risk management activities. Given the importance of risk management during software development, it is important to investigate its state-of-practice and to provide respective guidelines. This holds especially for software houses, i.e., companies whose primary products are software, for which risk management during development is essential to guarantee quality and to deliver within time and budget. The goal of this paper is to investigate the state-of-practice of risk management during development in software houses. For this purpose, we present results of a survey conducted by the University of Innsbruck together with the Austrian consultancy company Software Quality Lab in software houses from Germany (D), Austria (A) and Switzerland (CH), the so called “DACH region”. Overall 57 software houses from the DACH region responded to questions on risk management during software development. The results were complemented by interviews and results from related surveys. The results presented in this paper provide information on the state-of-practice and are equally relevant for research (by guiding it to relevant topics) and practice (by serving as a baseline for comparison). The presented survey on risk management was part of a comprehensive survey with additional questions in the areas of process models, release planning, requirements engineering, implementation, and testing [7].

This paper is structured as follows. Section 2 presents related work. Section 3 discusses the survey goal, design and execution. Section 4 presents results and discusses them. Finally, Sect. 5 concludes the paper.

2 Background and Related Work

In this section, we provide background on risk management and summarize related results on risk management during development from other surveys reporting respective results. Relevant related results reported in these studies address the project effort spent on implementation, tool support during implementation as well as the usage of agile practices during implementation. In the following paragraphs we summarize these quantitative and qualitative results and later in Sect. 4 we relate them to our findings.

A risk is the chance of injury, damage or loss and is typically determined by the combination of the probability of an event and its consequence [8]. Amongst others, risks can refer to the software project or product level. For instance, with regards to project risks, the Project Management Body of Knowledge (PMBOK) [9] defines risk as an uncertain event or condition that, if it occurs, has a positive or negative effect on a project’s objectives. With regards to product risks, the IEEE 829:2008 standard on software and system test documentation [10] defines risk as the combination of the probability of an abnormal event or failure and the consequence(s) of that event or failure to a system’s components, operators, users, or environment.

Risk management is according to the standard ISO/IEC/IEEE 24765:2010 on systems and software engineering [11] an organized process for identifying and handling risk factors to identify what might cause harm or loss (identify risks); to assess and quantify the identified risks; and to develop and, if needed, implement an appropriate approach to prevent or handle causes of risk that could result in significant harm or loss. Reasons to apply risk management are to increase the likelihood and impact of positive events, and decrease the likelihood and impact of negative events in the project [9]. According to Sommerville [12] risk management is one of the most important jobs for a project manager.

A risk management process contains the core activities risk identification, risk analysis, risk treatment and risk monitoring [13]. In the risk identification phase the risk items are identified. In the risk analysis phase the probability and impact of risk items and, hence, the risk exposure is estimated. On the basis of risk exposure values, risk items may be prioritized and assigned to risk levels. This results in a risk classification. In the risk treatment phase the actions for obtaining a satisfactory situation are determined and implemented. In the risk monitoring phase the risks are tracked over time and their status is reported. In addition, the effect of the implemented actions is determined. The activities risk identification and risk analysis are often collectively referred to as risk assessment, while the activities risk treatment and risk monitoring are referred to as risk control.

The factors that influence how a risk is assessed are called risk criteria and can be quite diverse including associated cost and benefits, legal and statutory requirements, socio-economic and environmental aspects, the concerns of stakeholders, priorities and other inputs to the assessment [8].

Several surveys address risk management during development [14,15,16]. Haberl et al. [15] asked in 2011 for the reasons to not applying risk analysis. 32.3% of all respondents answered to not apply it because of missing methodological skills, 24.5% mentioned insufficient resources as a reason, 17.4% did not see the necessity and finally 10.8% argued to not have the time for it. The remaining 15% mentioned other reasons that were not further discussed in the report. Similar issues for risk management integration were also observed by Kajko-Mattsson et al. [14] in 18 of the 37 surveyed companies (48%). Amongst others, the following three reasons (similar to [15]) were found. First, that employees do not see necessity for risk management, which may make it more difficult to motivate necessary work for risk assessment. Second, another reason are inexperienced risk managers that do not have enough knowledge and experience in risk management. Third, risk management requires to allocate resources exclusively to the task. This may be a problem for organizations. In the case of insufficient resources for risk management, risks cannot adequately be monitored and controlled. As a consequence, risk management possible fails. Overall, one can state that according to available studies, missing methodological knowledge, insufficient resources and a lack of time are very common reasons to not apply risk management.

Kajko-Mattsson et al. [14] found in 2008 that risk management in the context of software development is essential and that its integration into the process of development is important. Furthermore, the authors concluded that the implementation of risk management is experienced to be very difficult in the studied organizations [14]. Nevertheless, Haberl et al. [15] noted that almost 70% of participants stated that they perform risk management in their projects. A similar percentage was found in 2014 by Arnuphaptrairong [16] for the investigated software companies from Thailand. 77.5% of them responded to have risk management integrated in their software development process. In addition, Arnuphaptrairong asked the participants which techniques for risk identification are practiced. The most common answers were brain storming (73.3% of all respondents), check lists (70%) and interview (23.3%). Other techniques like top ten lists, risk dimensions or the delphi method, were only mentioned by at most 16% of the participants to be used in practice. Concerning risk roles, 80.6% of all participants identified the project manager as the person responsible for software project risk management.

3 Survey Goal, Design, Distribution, and Analysis

This section provides the survey goal and research questions (Sect. 3.1), the survey design (Sect. 3.2), the survey distribution (Sect. 3.3), as well as the survey analysis (Sect. 3.4). Finally, Sect. 3.5 provides a summarizing timeline of the performed survey design, distribution and analysis activities.

3.1 Goal and Research Questions

The goal of this survey is to investigate the role of risk management during development in software houses from Germany, Austria and Switzerland. The target audience of the survey are therefore software houses that are located in Germany, Austria or Switzerland and do not operate in a domain that may impose restrictions on their software development, e.g., medical or automotive. Based on the goal and taking industrial relevance from experiences of the involved company Software Quality Lab into account, we investigate the following three research questions (RQs):

 

RQ 1. :

How common is risk management during development and what are the reasons for not performing risk management?

RQ 2. :

For what purposes are risk assessment results applied for?

RQ 3. :

Which risk assessment criteria are considered for risk management during development?

 

3.2 Survey Design

In this section, we present the sampling plan, the questionnaire design and the performed pilot test. As the presented survey on risk management was performed as part of a more comprehensive survey, the presentation of the survey design is based on a previous publication [7]. The survey design is based on lessons learned and guidelines reported by other researchers in software engineering [17, 18].

Sampling Plan. The sampling plan describes how the participants are representatively selected from the target population. The first decision, whether a probabilistic, non-probabilistic or census sample should be considered, was already made by selecting the target audience. Given that no list of all companies exists that have the characteristics of to target audience, a truly probabilistic or census sample is not feasible. The first (probabilistic) would require an enumeration of all members of the target audience to select randomly participants and the later (census) can as well only be conducted if all individuals of the target audience are known. As a result, non-probabilistic sampling was chosen.

As a method to draw the sample from the population quota sampling with the two strata geographical location of the software house (Germany, Austria or Switzerland) and number of employees (less or equal 10, 11 to 100 and more than 100) was applied.

Overall 57 software houses, 19 from each of the three countries, evenly distributed over the three company sizes were selected and could be consulted within the given time and resources. Based on the activities relevant for software houses from the OECD [19] industry categories, i.e., 62 – Computer programming, consultancy and related activities, as well as 631 – Data processing, hosting and related activities; web portals, the overall number of software houses in Germany, Austria and Switzerland could be estimated based on data from governmental statistical offices. For Germany, the “IKT-BRANCHE IN DEUTSCHLAND” [20] report identified 61,029 companies in 2013 that are classified with one of the two categoriesFootnote 1. For Austria, the governmental statistical office reported 13,281 companies in the respective categoriesFootnote 2 in 2012. Finally for Switzerland, the federal statistical office measured 2008 in the census of companiesFootnote 3 15,466 companies that have amongst their main activities programming, information technology consulting and data processing. As a result, the total number of software houses in the DACH region can be estimated with 90,000 (61,029 + 15,466 + 13,281 = 89,776). Taking the population size of 90,000 into account, with the 57 participating companies a precision [17], which measures how close an estimate (resulting from the survey data) is to the actual characteristic in the population, of 87% is achieved.

Questionnaire Design. The questionnaire was designed based on the experiences of Software Quality Lab and the involved researchers in conducting surveys as well as academic findings of related surveys (see Sect. 2) and the software engineering body of knowledge (SWEBOK) [21]. The knowledge and practical consultancy experience of Software Quality Lab was a valuable input to design the questionnaire. Furthermore, a technical report of a survey on software quality conducted by Winter et al. [22] in 2011 provided many useful insights for the questionnaire design. The questions included in the questionnaire were transformed into closed-ended questions and ordered by topic. The questionnaire was implemented and performed online with the survey tool LimeSurveyFootnote 4. Each research question was addressed by one question in the questionnaire, which was embedded into a larger questionnaire on software quality processes. The answer options for each questions are shown in Figs. 2, 3, and 4. The complete questionnaire is available via the first author upon request.

Pilot Test. The questionnaire was validated internally, i.e., by the involved researchers and Software Quality Lab, as well as externally by six employees of software houses. Internally, there were several iterations and the involvement of researchers and industrialists guaranteed a high quality review from different perspectives. Externally, the reviewers provided valuable, written feedback to further improve the questionnaire.

3.3 Survey Distribution

The distribution of the questionnaires among the potential participants included a pre-notification, the invitation with the questionnaire, reminders and a thank-you letter. The survey distribution started on April 1, 2015. The participants were selected by using Google Maps and searching for ‘software company’. Searching for this term reveals all software companies at the related location. Furthermore, information about the number of employees for each identified software house were determined. This made it possible to come up with 450 participants – 50 small, 50 medium and 50 large software houses per country. Two weeks after the pre-notification emails were sent, the invitation emails with a link to the online survey were distributed. As a result, 13 participants responded to not wish to participate and 20 software houses participated. One reminder was sent in the middle of the survey (end of April 2015) to remember possible participants about the survey. Due to the low number of responses, additional 500 software companies were contacted via email. In addition, new participants were searched and contacted exclusively by phone to invite them to the survey. During three days within the last week, 200 potential software houses in Germany, Austria and Switzerland were called and asked for participation. In this three days 18 software houses could be convinced to participate. Thus, the response rate for the phone calls was 9%, which is double the response rate of the email invitations (4% for the first half of the survey and 3% for the second). During the phone calls, also some of the reasons against the participation were mentioned. Amongst others, no time, no interest, already having participated in similar surveys as well as the absence of the respective decision maker were mentioned. The survey distribution ended on May 22, 2015.

3.4 Survey Analysis

The data was first analyzed quantitatively and then qualitatively by interviews with survey participants and evidence extracted from related work.

As the responses for each question were nominally scaled, the votes for each question were counted and then visualized in bar charts. In addition, we performed 12 interviews with survey participants (i.e., 21% of all participants) to triangulate the quantitative analysis and to identify the reasons behind some survey answers. The semi-structured interview type was chosen, because the structured interview limits the discussion freedom to enter unforeseen subtopics or ask questions that may arise during the interview. Another alternative would have been the unstructured interview. However, this form would have not allowed to ask prepared questions of interest that emerged during the analysis of the empirical survey. Telephone calls were used contact each participant in an economic and for the interviewee time- and place-flexible way. In the short interview one question on implementation was asked. In addition, the non-structured part of the interview followed subtopics of interest that were raised by the interviewee or that turned out as a result of previously conducted interviews to be of interest.

3.5 Survey Timeline

This section summarizes the survey design, distribution and analysis by providing the concrete timeline in which the respective activities were performed in 2015. Figure 1 shows the timeline for survey design, distribution and analysis activities. Activities with concrete dates in parentheses were performed in the given date range, the other activities were performed during the whole month.

Fig. 1.
figure 1

Timeline of the survey

Full size image

4 Results and Discussion

In this section, we first present the demographics of our survey, then we present and discuss main findings for each of the three research questions, and finally we discuss threats to validity.

4.1 Demographics

Overall 57 software houses, 19 from Germany, 19 from Austria and 19 from Switzerland, participated in the survey. Most of the software houses (84%) stated that they perform more than one type of software project. On average three types were stated. The three most common project types are development of web-applications (71%), individual software (61%), and standard software (56%).

In the sample of 57 software houses small, medium and large companies are present with a similar frequency: 38% of the companies are small-sized (up to 10 employees), 35% medium-sized (11 to 100) and 26% large-sized (more than 100 employees).

4.2 Main Findings

In this section we present the main findings for each of the three research questions.

RQ 1: Frequency of Risk Management During Development and Reasons for not Performing It. We asked whether risk management is applied or not during software development. 58% of the participants (33 of the responding software houses) answered to not perform risk management, 26% (15 respondents) stated to perform risk management, and 16% (9 respondents) did not know or provided no answer. As a follow-up question we asked the 33 respondents not performing risk management for their reasons. The results are shown in Fig. 2.

Fig. 2.
figure 2

Reasons for not performing risk management (N = 33)

Full size image

The main reasons for most respondents are lack of resources (48%) or no necessity (42%). Furthermore, lack of knowledge (33%) and no time (24%) are common reasons for not performing risk management. Finally, 6% provide no reason why they do not perform risk management.

In two surveys from 2014 [16] and 2011 [15] it was found that at least two of three projects apply risk management. However, according to the collected survey data of this study only about every third company performs risk management. Thus, it may seem that the number of risk management practitioners decreased. However, it should be considered that the survey presented in this paper had a well-defined target audience consisting of software houses in the DACH region, whereas Haberl et al. did not restrict their target audience. Thus, it may be that because of the inclusion of other industry branches (more mission or safety-critical domains like finance or production) the ratio of companies that perform risk management that also influences software development is higher. The higher rate reported in Arnuphaptrairong [16] could be due to regional differences.

Haberl et al. [15] and Kajko et al. [14] identified missing methodological knowledge, insufficient resources and a lack of time as the most common reasons to not apply risk analysis. The reasons for not performing risk management reported in our survey, i.e., lack of resources, lack of knowledge as well as no time are similar.

Also a subset of 12 respondents was interviewed whether they applied risk management or not and how this influenced their software projects. Reasons mentioned during the interviews for not applying risk management are that the interviewees were not convinced by the idea of it or that they had the feeling that it does not fit for agile development. Interviewees that declared to use risk management highlighted to use it because it helps them to assess the priorities of the requirements, it stresses the importance of specific tests and helps to decide which requirements should be included in the next release.

RQ 2: Application of Risk Assessment Results. Figure 3 shows the application areas of risk assessment results as provided by the 15 respondents who perform risk management. 73% of them apply risk assessment results to prioritize test cases. 53% apply them to target test selection methods, to allocate resources, and to define test criteria, respectively. Furthermore, 33% apply them to evaluate remaining risks in test reports. Finally, 13% of the respondents who perform risk management do not name an application area of risk assessment results.

Fig. 3.
figure 3

Application areas of risk assessment results (N = 15)

Full size image

In terms of the usage of the risk management and its affect on the software project, the survey found that the main application of risk management is to prioritize test cases. Similar, the participants mentioned during the interviews that again the main usage is to prioritize the requirements. Furthermore, they claimed that it is used to stress certain tests or to decide which requirements should be part of the next release. The main application of risk management is to prioritize activities, especially during software testing. Prioritization of test cases is an important application of risk assessments in software testing [23,24,25].

Interviewees that declared to use risk management pointed out during the interviews to use risk management because it helps them to assess the priorities of the requirements and to use this information for test and release management. Furthermore, the importance of risk management for specific tests was highlighted.

RQ 3: Risk Assessment Criteria Considered for Risk Management. Figure 4 shows the risk assessment criteria considered during risk management as reported by the 15 respondents who perform risk management. 93% of them consider technical product risk (i.e., a technical risk directly related to a system, service or test object), and project risk (i.e., risks related to management and control of a (test) project), respectively. 67% consider business product risk (i.e., a business risk directly related to a system, service or test object), 60% criticality (which informally integrates other risk assessment criteria), and 27% frequency of execution. Finally, 7% of the respondents who perform risk management do not which risk assessment criteria are considered during risk management.

Fig. 4.
figure 4

Risk assessment criteria considered during risk management (N = 15)

Full size image

In the survey of Arnuphaptrairong [16] from 2014 the participants stated that brain storming, check lists and interview are common methods for risk identification. Furthermore, most participants identified the project manager as the person responsible for software project risk management. Although neither this result nor results from other surveys provide results on the actual usage of risk assessment criteria as our survey does, we can assume from Arnuphaptrairong [16] that assessment criteria are often not specified explicitly (as brain storming plays an important role) or that project risks play an important role (as project managers are often responsible for risk management).

4.3 Threats to Validity

In this section we discuss the main threats to validity of the presented study and measures performed to mitigate them. First, the survey was answered by representatives from 57 software houses (from overall about 90,000 software houses in the whole DACH region). Although the number of participating software houses is relatively small, the precision of the conducted survey is already 87%. Furthermore, the results were triangulated by interviews and a comparison to results of related studies. The partial deviation of the study results from those of similar studies, especially with regards to the frequency of risk management during software development in software houses, suggest that the presented results are specific for software houses and not generalizable to arbitrary domains. The questionnaire itself was reviewed internally by the involved researchers and the involved company Software Quality Lab as well as externally by six employees of software houses. Furthermore, other already validated and successfully implemented related questionnaires like [22] were taken into account when constructing the performed questionnaire. Finally, the questions asked on risk management were part of a more comprehensive survey on software quality and processes in software houses. This might reduce the threat that the survey participants have in general a positively biased attitude towards risk management in software development.

5 Conclusion

This paper presented a survey on risk management during development in software houses from Germany, Austria and Switzerland. Overall 57 software houses participated. Results from the survey show that risk management is often not performed during software development. The most important reasons for not performing risk management are lack of resources or missing necessity for it. The most important application area for risk assessment results is the prioritization of tasks like testing. The most important risk assessment criteria are technical product risk and project risk. The survey was restricted to the DACH region including the countries Germany, Austria and Switzerland, and based on a limited amount of participants. However, the results are triangulated by results from literature and interviews with a subset of the survey participants.

In future, we plan to replicate the survey in other regions and to perform case studies to investigate in which context (for instance, with respect to the process model applied or the development domain) specific risk management approaches are effective and efficient. Based on the results of these empirical studies we plan to derive practical guidelines to improve risk management during development.