Keywords

1 Introduction

Requirements Engineering (RE) is one of the key processes in software development. It has been observed that when the RE process is orchestrated properly it can favorably influence the whole software development process [3, 6]. Conversely, problems related to requirements were often identified as main causes of the failures of IT projects, e.g., [4, 8, 9, 18].

With the advent of agile software development methods, such as eXtreme Programming (XP) [1], or Scrum [14], new challenges have emerged for RE. The approach to planning projects has changed from traditional—predictive to adaptive [7]. The need for adaptivity is deeply-rooted in the Agile Manifesto [2] that advise teams to regularly reflect on how to become more effective and make use of the lessons they learned. Although this principle makes the team responsible for improving the software development process, many agile software development methods introduce additional roles responsible for driving the improvement process, e.g., Coach in XP [1], or Scrum Master [14] (we will refer to them as agile coaches). The agile coaches are supposed to have decent knowledge of agile methods to help teams find the best solutions to their problems, as well as be able to convince management to allocate necessary resources to support the improvement process.

Among the different tools that agile coaches can use to support the improvement process, they can employ one of the existing agile maturity models (e.g., AMM [11], SAMI [15]). These tools might be useful for discovering problems in their projects, as they prescribe sets of guidelines related to the proper usage of practices in agile projects.

One of such tools is the approach to assessing the maturity of Requirements Engineering practices in agile projects (REMMA). It is a unique approach. It does not only support the improvement of RE by indicating some practices that need to be tweaked but also it allows incorporating information about how the specific context of a project affects the usage of agile RE practices. Thus, REMMA is a hybrid method in the sense that it combines elements characteristic to both prescriptive and inductive approaches to process improvement. REMMA has been proposed by Ochodek and Kopczyńska in [10]. Although it has been developed using Design Science Research, so its components resulted from several empirical studies, it lacks empirical validation. Therefore, in the paper we present a study with which we would like to fill this gap.

We carried out an exploratory case study in a software house context to understand to what extent the method satisfies its requirements: is perceived as useful (i.e., provides results that correctly reflect the current state of implementation of RE practices in a project), easy to use (can be used by project team), and cost-effective (project team can afford using the method).

2 Case Study

We conducted an exploratory case study to characterize and understand an application of REMMA to assess the alignment of practices within the context of an organization simultaneously running several agile software development projects using the guidelines by Runeson et al. [13].

To formulate the research questions we referred to the theoretical framework of the Technology Acceptance Model (TAM) [17], which was used while creating REMMA to define quality attributes that the method should exhibit. The following determinants of Perceived usefulness (PU) and perceived ease of use (PEOU) were selected—job relevance, output quality, and result demonstrability, and self-efficacy (an individual’s belief that he or she can perform the maturity assessment in his or her project), and perception of external control, which is the degree to which an individual believes that organizational and technical resources exist to support the use of the artifact (e.g., management support). Finally, the authors wanted the method to be cost-effective concerning the effort required to perform the assessment. Thus, the following research questions were formulated:

  • RQ1 What is the cost of performing the assessment?

  • RQ2 How easy to use is the method?

  • RQ3 How useful are the results of the assessment?

Case and Subject Selection. In our study, we decided to look for an organization that could provide us with some ‘typical cases’ [19]. We decided to conduct the study in a software house (we refer to it as Company) that at the moment of conducting the study employed nearly 100 Python programmers, which made it one of the largest software development houses in Central Europe. The company ran approximately seven concurrent projects. It had three locations in Poland. One of their main goals is continuously improving and working towards becoming a truly agile organization.

We were allowed to carry out the case study in three of the company’s project teams. We were also able to talk to a person that had worked at the company for 3 months as a Scrum Master (Coach) with the goal to improve their processes (we will refer to this person as Agile Coach). This gave us the chance to investigate the RE practices from multiple perspectives (Agile Coach, project team members) and triangulate the observations made during the case study. The Agile Coach had cross-sectional knowledge of all the projects and was no longer employed at the time. This was important for us and helped reduce potential biases. Prior to the case study, we had asked the Agile Coach to select the projects (cases) to analyze. He chose three projects, A, B, and C that he thought corresponded to outstanding, good and poor projects with respect to how agile practices were implemented at the time he was working for the company.

All projects developed web-based applications using Scrum for small- or medium-sized organizations that were located in different countries. We describe the projects in more detail in Table 1, to the extent that we are allowed.

Data Collection and Analysis Procedure. We decided to use semi-structured interviews to collect data during the study. The interview guide for the study was prepared by one of the researchers (the primary interviewer), based on research questions. As a following step, the guide was reviewed by the secondary interviewer. We additionally decided to extend the model by adding more context factors than those presented in the detailed paper on REMMA [10]. Based on our experience we defined the following factors: adherence to Scrum, use of XP, team location, team communication type, type of budget, and staff retention. Finally, we decided to test the prepared instrumentation during a pilot study. The primary interviewer conducted an interview session with the secondary interviewer as an interviewee. The pilot interview pertained to the three software projects that the secondary interviewer had participated in during the previous year.

After incorporating all the suggestions and remarks from the pilot study, we proceeded to the actual interview sessions. We began by interviewing the Agile Coach in two sessions. Afterward, we met with the representatives of each project team. We organized separate sessions for each project team. Each session was conducted in Polish, and was constituted by five phases:

  1. 1.

    General Questions—the goal was to better understand the company and project context and to inquire about the interviewees’ experience;

  2. 2.

    Assessment—the primary interviewer facilitated the process of interviewees collectively filling in the assessment form about their project using a prototype version of a software tool supporting the REMMA assessment;

  3. 3.

    Presentation of the method—both interviewers explained the mechanism of practice alignment assessment with REMMA by giving a small presentation;

  4. 4.

    Presentation of results—both interviewers discussed the assessment results with the interviewees. Then the interviewers presented results of the assessment performed by the Agile Coach and asked interviewees to discuss the commonalities and differences between these two assessments;

  5. 5.

    Questions about REMMA and Closing up—the interviewers asked about the interviewees’ impressions of using REMMA and closed up the interview.

Before the sessions, we obtained support for the study from the CEO. During the sessions, we observed that the participants were committed, sometimes even enthusiastic. Both interviewers were present during all sessions. The primary interviewer focused on asking questions while the secondary interviewer tried to reflect and ask follow-up questions. All recordings from the sessions were transcribed by the primary interviewer (7.5 h of the audio recordings) and reviewed by the secondary interviewer. Both interviewers analyzed the collected data according to the guidelines of Charmaz [5]. First, initial coding was carried out incident-by-incident, individually by each researcher. Then in vivo codes underwent constant comparison, and finally axial coding was performed by both reviewers working together. At the end, we discussed the results. The part of our coding scheme that we perceive as relevant to answering the research questions is presented in Fig. 1. To make the codes more understandable to the reader, we substituted their names with exemplary quotations they tagged. We also changed the names of categories to questions. Finally, we used font weight to reflect the number of projects in which the underlying code existed.

3 Results

Cost of Performing Assessment (RQ1). The Agile Coach needed 40 min to assess all three projects while projects’ teams assessed their projects in 20 min (C), 28 min (B), and 38 min (A). All interviewees stated that from their perspective using REMMA did not require much effort. Moreover, they regarded it as an acceptable investment for their current projects. They also claimed that they could afford to perform this assessment regularly, e.g., every sprint (Fig. 1).

Table 1. Description of the projects and interviewees presented according to the guidelines of Petersen and Wohlin [12].
Full size table
Fig. 1.
figure 1

Interviewees opinions on REMMA. The size and emphasis reflects the number of projects in which the statement was made.

Full size image

REMMA was also indicated as well suited for retrospectives. Conversely, one person asserted that he would not use REMMA when there are multiple problems in the project as“investing in anything other than source code development activities would be treated as a waste of resources”. Taking into consideration the quantitative data related to the effort required to perform assessments, and the positive feedback of team members, we might come to the conclusion that the method seems affordable for an agile project, which is in a relatively favorable situation. Nevertheless, using REMMA might not be the best option if a project is in a dramatic situation.

Perceived Ease of Use of the Method (RQ2).(1) Self-efficacy. At first, we focus on identifying possible problems with understanding REMMA and any spontaneous questions asked during the interview. There was a question regarding the interpretation of the practice assessment scale and the rationale behind it, and about the source of the practices. There appeared ten questions concerning several terms used in the names of practices, e.g., “what is an elevator test?”. These required the interviewers to read aloud the description of the ambiguous practice from the REMMA assessment form. The remaining practices and context factors seemed self-explanatory. As the Fig. 1 shows, the majority of interviewees claimed that they would be able to conduct the assessment by themselves. The Agile Coach concluded the interview by stating that “from my perspective, it is a super tool for the Scrum Master, who can perform the assessment by himself without any great effort”. (2) Perception of external control. Taking into account the previously discussed cost-efficiency of REMMA, we can state that participants considered using the method in their projects. In addition, they suggested that the management might be interested in using the method to “get an overview of the projects.”

Usefulness of REMMA (RQ3).Output quality. Project A: The results of both the team members’ and Agile Coach’s assessments showed that all critical and the majority of important practices were performed in this project at the de facto standard level in basic assessment. There were only two important practices identified as never used. Both assessments indicated that there are certain important practices to be improved, and additional practices to be introduced, in the Knowledge sharing area. Project A was believed to be the best Agile project in the company concerning its process quality. The team members seemed very proud of their success in applying Scrum “everybody says that we do it by the book and even better”. Project B: The Agile Coach assessed the project as middling in applying the Agile principles. Around 80% of critical and ca. 75% of important practices were at least normally used. Two critical, one important, and two additional practices were performed at discretionary-use level. Moreover, two important and three additional practices were never used. There were also five negative influencers (practices insufficiently used) and one practice unfit for the context. The Agile Coach triggered some positive changes during his presence in the company. When we asked to comment on the differences between their assessment and the one of the Agile Coach, the team members easily identified the improvement steps that led them to the up-to-date status. Project C: It had the highest number of practices that required improvement. This state was correctly reflected in the results of the assessment—the project obtained the lowest score compared to the others. Only around 20% and 36% of practices were assessed to be de facto standard or normally used by team members and the Agile Coach respectively. In both assessments, there were 8–9 practices that negatively impacted the usage of other practices and 8 practices unfit for the context. Both assessments also indicated that Customer Involvement, Knowledge sharing areas had the lowest result. The assessment of the Planning area by the Agile Coach showed that the practices were at least normally used. While discussing the Agile Coach’s assessment with the team, they confirmed that the Agile Coach had tried to introduce them, but they were rejected soon after.

We can state that all the interviews approved the results of assessment provided by the method. They also confirmed the appropriateness of each major increase or decrease in influence and contextual assessments. Moreover, we analyzed the codes developed during the analysis of transcripts; we assigned them into categories of the project’s problems, strong points, and context factors. We wanted to find out if they were mirrored in the assessments. We observed that 100% of problems and strong points, described by interviewees, were reflected in the assessments results. In addition, we obtained examples of practice implementation for 75% of cases.

Result Demonstrability. Some of the remarks from interviewees concerned the usability of our prototype tool used to support the assessments. Most of them were related to graphical details, such as colors, or layout. We also received comments regarding different forms of presenting the results. Interviewees suggested that they should be provided in a form supporting a top-down approach to analysis. They stated that, first of all, that the method should be able to provide an overall result of assessment—accurately expressed by a single number. For instance, in REMMA the overall assessment is provided in the form of TL and PIP measures, or a chart summarizing the results of the basic assessment. The interviewees supported this idea with the argument that some people may not be interested in the details of assessment. Second, team members might want to receive a quick report on the current status of their project, without needing to analyze the results. On the other hand, agile coaches and team members who are working on improvement may like to receive more detailed reports. Such reports should help them identify potential problems in their project.

Job Relevance. All of the interviewees stated that they were interested in using REMMA in their projects. Moreover, two interviewees were eager to use it in another project right after the interview. During the sessions, we tried to identify which project roles could potentially benefit from applying REMMA in certain situations (see Fig. 1). The participants indicated that Scrum Master would be the role that would use the tool most extensively for supporting the whole improvement process (identification of problems, defining goals, monitoring the process). However, they also believed that a company owner (management) would consider the tool valuable, as it provides comprehensive information about project status. Overall, REMMA was recognized by project stakeholders as an appropriate tool to make the development team more aware of the current status of a project or to support retrospectives. According to the interviewees, the teams that are unaware of how agile practices shall be implemented may find REMMA particularly valuable.

4 Conclusions

To validate the approach to assessing the maturity of Requirements Engineering practices in agile projects (REMMA) proposed method we conducted an exploratory case study in one of the biggest software houses in Central Europe. The results of the study made us expect that the proposed approach might be useful for agile development teams to identify strengths and weaknesses of RE and provoke improvement. Besides, according to the study participants the method seems cost-effective and simple enough to be regularly used in agile software projects.

Finally, we are aware of the limitations of the case study as a validation method, but we believe that the promising results might provoke further implementation and evaluation of the method by practitioners in various organizations. Moreover, we hope that the paper will facilitate the discussion about the role of project context in the assessment of the agile-projects maturity.