Keywords

1 Introduction

Agile Software Development (ASD) is an iterative approach to delivering software products. The term “agility” implies adaptability [1], flexibility [2], and close collaboration with the customer [3]. An Agile approach assumes sensible values such as trust [4], responsibility [5] and loyalty [6]. Around half of organizations have now been applying Agile practices for over three years to adopt change and transformation management [7]. Moreover, the results from a survey conducted in 2018 among software industry practitioners show that 97% of respondents declared using Agile methods [8]. In fact, the benefits of adopting Agile practices have been reported in many studies [9,10,11], indicating an increase of team productivity, motivation and discipline, as well as overall software quality, just to name a few.

Indeed, software quality is an important aspect to be considered during the software lifecycle [12, 13], usually defined in terms of high-level attributes [14]. Alternatively, one can impose additional constraints on the behavior of the system. In other words, the required properties (attributes and constraints) are specified as non-functional requirements (hereafter, NFRs), in addition to functional requirements (FRs). Since the beginning of software development as a job role, NFRs have been recognized as critical factors that affect the acceptance and use of the products by the users [15].

In fact, to mitigate the risk of users’ dissatisfaction by misunderstanding or disregarding their expectations and needs, active user involvement is imperative in ASD [16,17,18]. However, one question arises naturally: Does this user engagement bring other risks, and does the development team need to find a balance between risk and benefits?

Undeniably, the search for the answer to this question has been the subject of vast research [19,20,21], since the introduction of the Agile Manifesto [22]. Nevertheless, few studies provide an evidence-based review and analysis on the subject of implementing NFRs, in particular regarding the issues that could arise with the advance of ASD, as well as the practices that have been documented as successful facilitators.

The values and principles followed in ASD also result in practices different than those used in more traditional software development methods. It includes requirements engineering practices [23], which e.g. assume continuous close cooperation with the customer [24], put more emphasis on face-to-face communication [25], and use less formal techniques like collaborative games [26].

Both researchers and practitioners have repeatedly noted the challenges in Agile requirements engineering. For example, the results from a Delphi study [27], performed in 2017 in a group of 26 experts, show that one of the recognized challenges is to “establish non-functional requirements”, which has been reported by prior other studies [25, 28, 29]. The comprehensive know-how with regard to the more detailed challenges and relevant counteractions is not available though. The only available secondary study focusing on NFRs in ASD at the time we started our research was the SLR by Alsaquaf et al. [30]. That SLR was considered by its authors as a starting point for further empirical studies and several primary studies were published since then. To systematize the current state of the art, in this paper, we put forward these two following research questions (RQs):

  1. 1.

    What issues affect the identification and implementation of non-functional requirements in ASD?

  2. 2.

    What practices facilitate the successful identification and implementation of non-functional requirements in ASD?

Therefore, the goal of this study is to review and analyze the existing studies and their outcomes and to summarize the documented issues and applied practices, in the extent of NFR identification and implementation, within the ASD context. To provide evidence-based and state-of-the-art answers to the above questions we conducted a systematic literature review (SLR).

By design, the results of this study are complementary to the existing body of knowledge by providing the following contributions to the software engineering discipline: the collections of (i) the current issues (challenges and problems), and (ii) the explicit practices that, respectively, affect and facilitate the identification and implementation of NFRs within ASD. Moreover, the findings in this paper entail useful implications for researchers and practitioners alike. In this context, while the former group might be interested in investigating the impact of particular issues on the success (failure) of ASD projects, the latter group might be willing to mitigate those issues by adopting the practices in the scope and content due to the current needs and priorities.

The remainder of this paper is laid out as follows. Section 2 describes the rationale behind implementing NFRs. Section 3 provides the description of the research methodology, applied to conduct the systematic literature review. The results are given in Sect. 4, followed by their discussion in Sect. 5. Finally, the paper is concluded in Sect. 6.

2 Rationale Behind Implementing NFRs

Generally speaking, non-functional requirements (NFRs), also known as quality requirements, define the users’ expectations and needs regarding a software product, as well as their particular notions of its qualities. According to Svensson et al. [31], the most important quality attributes in industrial practice relate to usability, performance, reliability, stability, safety, security/integrity, compliance, maintainability, reusability and interoperability. Unmistakably, NFRs have great importance in software product development [31,32,33].

Besides this, NFRs can also impose global constraints on a software product [34], arising from all of its parts as well as from interdependencies between them [35]. In other words, NFRs put constraints on how the product’s functions must work [36]. Overlooking or even neglecting information related to quality facets negatively affects the final product. Ironically, although it might be surprisingly different from common sense, NFR-related errors are still claimed to be the most difficult to correct, and the most expensive [37]. It is a major risk, especially considering that in recent years software defects have become the dominant cause of user outage [38].

Undeniably, both researchers and practitioners from ASD communities have seen the need to capture, document and prioritise NFRs [39]. For instance, Microsoft, the largest software and programming company worldwide [40], recommends capturing functional and non-functional requirements alike, since the former indicate whether the application does the right thing, while the latter determine whether the application does those things well [41]. Oracle, the second largest software corporation, argues that “the key to successful software development is that all stakeholders develop a clear and uniform understanding of application requirements” [42].

Furthermore, we also acknowledge the importance of NFRs as the major external quality facets of the software products from the user’s perspective [43]. The questions addressed in this study are narrowed to ASD, which assumes having the user(s) actively involved. If one compares Agile with traditional approaches, this involvement is not limited to the early stages of the development process. On the contrary, Agile development principles encourage active user involvement, being generally considered to contributing to user satisfaction [44, 45] and project success [46].

3 Methodology

We designed and executed the systematic literature review following the guidelines for SLR studies in software engineering elaborated by Kitchenham and Charters [47]. The definition of the search query and query execution in Scopus (phase 1 of SLR process) are shared with our other study aimed at identification of particular NFR-related requirements engineering techniques [48]. The inclusion/exclusion criteria were however defined with respect to this study’s aim and subsequent phases of the SLR process were conducted separately in each of two studies.

We chose to rely on a single publication database (Elsevier Scopus). Scopus was selected because it indexes a large number of journals and conferences [49] and enables a single search query to access items from a broad variety of publishers [50]. It is worth noting here that in several other SLR studies similar to ours (e.g. [30, 51]) similar strategies were applied, in particular exclusively relying on the Scopus database.

3.1 Inclusion and Exclusion Criteria

The papers were eligible based on the five following inclusion criteria:

  • peer-reviewed papers (I1);

  • papers in English (I2);

  • papers published since 2008 (I3);

  • papers related to the software engineering domain (I4);

  • papers covering Agile development and NFRs (I5).

The papers were screened prior to acceptance and were further rejected if they had any of the following exclusion criteria:

  • papers not providing any information about NFR issues or practices in ASD (E1);

  • papers not available for download, despite extensive search (E2);

  • papers reporting the same results covered by another source included in SLR - in such cases the latest paper was included (E3);

  • papers dedicated to a very specific subarea of NFRs (e.g. with proposals of advanced methods of establishing security requirements) (E4).

We focused on papers published since 2008 to include all works published in the last 12 years before the conduction of the SLR. We also decided to include papers dedicated to a specific project context (e.g. large-scale distributed development), but to exclude papers with very narrow scope (E4) e.g. with advanced dedicated analysis methods suggested as facilitating practices for security requirements, which are hard to consider as an issue or practice regarding the whole category of NFRs.

3.2 Search Query Definition

As we had performed some initial searches before planning the SLR, we were aware that sources dedicated to this topic of interest are rather scarce. This led us to the decision to cast a wider net and try to identify all sources focusing on NFRs in Agile, thus we used more generic keywords instead of those exactly matching our RQs (e.g. “challenges” or “practices”).

The following search string was used:

TITLE-ABS-KEY ((agile OR scrum OR lean OR xp OR kanban) AND (nfr OR “non-functional requirements” OR “quality requirements”)) AND PUBYEAR> 2007 AND (LIMIT-TO(DOCTYPE, “cp”) OR LIMIT-TO (DOCTYPE, “ar”) OR LIMIT-TO (DOCTYPE, “ch”)) AND (LIMIT-TO (SUBJAREA, “COMP”) OR LIMIT-TO (SUBJAREA, “ENGI”) OR LIMIT-TO (SUBJAREA, “MATH”) OR LIMIT-TO (SUBJAREA, “BUSI”) OR LIMIT-TO (SUBJAREA, “DECI”)) AND (LIMIT-TO (LANGUAGE, “English”))

The string includes various methods that could possibly be mentioned in the title, keywords etc. instead of the generic “Agile” term. We also provided alternative terms commonly used to denote an NFR. The types of documents mentioned in the search string match peer-reviewed papers. The specification of subject areas resulted from our knowledge, in particular on how some sources (especially the series that include conference proceedings as its volumes) are classified and indexed by Scopus. The search in titles, abstracts and keywords was chosen as the most comprehensive option available (Scopus does not enable searching the contents of full texts).

3.3 Search Strategy

We defined a process that comprised 3 main phases:

  1. 1.

    Execution of the search query.

  2. 2.

    Manual review of titles, keywords and abstracts of the papers retrieved from the search to exclude those not related to the topic of NFR in an Agile context.

  3. 3.

    Manual review of each remaining paper’s full text in order to decide whether to finally include it or not. Identification of information pieces relevant to our RQs and assigning codes to them.

3.4 Search Execution

The results of the 3 phases defined in the previous section were as follows:

Phase 1: The search was executed on November 28th 2019. Despite including several alternative keywords in the search string, the search returned only 159 papers. This confirmed our initial suspicions that the topic of “NFR in Agile” is not widely addressed in the scientific papers, at least those indexed by Scopus.

Phase 2: The results retrieved by the Scopus search engine (that include title, keywords and the abstract of each paper found) were manually reviewed. It allowed us to verify the findings against I5 criterion more precisely than in the case of relying on an automated search and to reject papers that reported nothing on NFRs (for example, several papers referring to “quality requirements” turned out to interpret this term as “well-documented/valid requirements” instead of “requirements regarding system quality”). As a result, 71 papers were retained at the end of this phase.

Phase 3: In this phase, the papers were reviewed and checked against exclusion criteria E1-E4. Finally, 44 papers were qualified to extract information. Moreover, we evaluated the papers with regard to (i) the use of appropriate and rigorous research methods, (ii) clarity and coherence of the research findings, and (iii) providing a validation of the proposed approach. During the review, apart from just deciding on the paper’s final classification, the fragments relevant to the RQs were identified and provided with codes to summarize the findings. Next, the codes were reviewed to identify similarities, and related codes were grouped into the more generic ones presented in the Results section.

4 Results

The final results of the SLR are presented in Tables 1 and 2. For each issue reported and facilitating practice suggested, a list of papers mentioning it is provided (“Sources” column). We also explicitly distinguish issues/practices related to requirements engineering activities (“RE” column) from those that should rather be associated with e.g. testing, architectural design or project management. Both tables are sorted starting from with the items quoted by the most sources. The elaboration of results with respect to the answers they provide to RQs is provided in 4.1 and 4.2.

Table 1. Problems and challenges affecting development of NFRs in ASD
Full size table
Table 2. Practices facilitating implementing NFRs in ASD
Full size table

4.1 What Issues Affect the Identification and Implementation of Non-functional Requirements in ASD?

The most frequently reported issue concerns neglecting NFRs (I1) i.e. the situation in which developers and/or stakeholders focus on the system’s functionality and do not identify NFRs in a sufficient manner, often postponing such task to a later stage of the project. Unfortunately, it often results in significant rework effort, as NFRs are not necessarily simple additions and are likely to substantially affect the system architecture. It should be stated that this issue, while mentioned by many papers, is not always based on experience or empirical findings but sometimes treated as “common knowledge” or quoted from other referenced papers. On the other hand, the frequent occurrence of such issue is confirmed by more general studies not dedicated to NFRs but listing the general problems and challenges related to ASD and/or requirements engineering (examples are given in Sect. 5.1).

A number of issues can be attributed to limitations of the simplified requirements documentation techniques (e.g. user stories, story cards) commonly used in Agile methods. In application to NFRs, such techniques can turn out to be insufficient to express NFRs in an unambiguous way (I2). Another reported shortcoming of such techniques is the difficulty in representing the dependencies between a given NFR and other related requirements (I4). An open issue of how to represent NFRs is also reported as a doubt explicitly expressed by Agile teams (I9). While in some projects, a workaround in the form of a separate document dedicated to NFRs is used, it can also cause difficulties as the project team can focus on the FRs typically stored in the product backlog and do not sufficiently rely on external documents including that for NFRs (I15).

NFRs are more difficult to capture and cause problems both for stakeholders and for the project team. The stakeholders may even not recognize their needs that have to be captured as NFRs (I3). The project team may in turn lack the knowledge and competencies necessary to identify and implement some NFRs, especially when advanced concepts related to e.g. security or performance need to be used (I7).

The elicitation and communication of NFRs is another category of issues. Requirements elicitation can fail to involve all of the relevant stakeholders (I8) and result in NFRs that do not reflect all viewpoints or even omit some important requirements. NFRs are also quite hard to express, thus their communication (both from the stakeholder to the project team, and between team members) can be prone to errors (I14). Moreover, in large scale development projects, involving multiple teams, additional communication problems are likely to arise (I12, I16, I17). Several drawbacks in handling NFRs can result in a situation of late detection of NFR infeasibility (I13), especially considering the lack of a feedback loop regarding NFRs (I19).

Several issues related to NFR traceability and verifiability are reported as well. A lack of NFR traceability mechanisms is claimed in general (I5), but also several more specific issues are described. Traceability of NFRs is even more important as NFRs are frequently affected by changes in FRs (I10). It is difficult to develop test specifications associated with NFRs, which are intended to verify their implementation (I6). Moreover the execution of such tests requires the associated FRs to be already implemented (I24). The cost-effectiveness of some tests is also disputed (I21). The manual verification of DoD can be cumbersome as well, especially in case of a lengthy checklist (I18).

The remaining issues are either related to project team members’ attitudes (I11, I23) or architectural design activities (I20, I22).

4.2 What Practices Facilitate the Successful Identification and Implementation of Non-functional Requirements in ASD?

A number of practices dedicated to the documentation of NFRs can be found in the literature, even though some of them seem to be mutually contradictory. The issue of the insufficiency of the popular Agile requirements documentation techniques can be addressed by utilizing modified or additional specification techniques (P1). Such techniques are to be applied to NFRs only (while FRs are still recorded as e.g. user stories). Some proposals include techniques adopted from plan-driven approaches.

Alternatively, other sources recommend making sure that NFRs are documented together with FRs, using the same, typical representations, e.g. user stories, Definition of Done, Acceptance Criteria (P4). There is also a kind of intermediate solution suggested - instead of specifying NFRs as epics, user stories etc. and mixing them with FRs, a similar but distinct structure dedicated to NFRs can be used (P14). Also, assumptions related to the implementation of NFRs are worth documenting using, e.g. a wiki-page (P21).

Focusing on NFRs in an early phase of the project (P3) is a suggestion that can possibly minimize the rework caused by omitted NFRs. Multiple roles and viewpoints should be involved to elicit and/or review NFRs (P7). A number of more detailed practices facilitating requirements elicitation from stakeholders can also be found, e.g. using proper terms (P11) or explaining the consequences of NFRs expressed by stakeholders (P20), especially in the case of “over-specification”. Another good idea is to educate and raise awareness about importance of NFRs (P8) - in general or with respect to some categories of NFRs which are not sufficiently recognized. Both stakeholders and project team members can be educated in such a manner.

As NFRs can be difficult to identify and even harder to implement, it is possible to strengthen the competencies of the project team by involving NFR specialists (P6). For example, a security expert can enable the elicitation of relevant security requirements and later verify their implementation. As the stakeholders may not be able to identify the NFRs themselves, external resources such as catalogues of NFR patterns/templates (P9) or dedicated supporting systems providing recommendations (P13) can be used as additional sources of NFRs.

Maintenance of traceability between FRs and NFRs is a frequently recommended practice (P2) that enables proper requirements management activities, including configuration and change management. Various solutions, including tool support, are proposed to ensure traceability maintenance. There are also other practices including the use of automated tools, in particular: tools to monitor the quality of the software under development, including the aspects expressed in NFRs (P5) and CI environments facilitating automated NFR testing (P22). Apart from tool-based testing, NFR-oriented code reviews (P19) and external tests conducted by an independent team (P23) can be practiced.

To minimize the risk of neglecting NFRs, several actions in the software project organization can be undertaken. They can concern the organization of the project team(s) (P10, P17); the development process - dedicating an iteration (P18) or a part of each iteration (P15) to the implementation of NFRs; or using multiple requirements registers, which make NFRs (or specific NFR categories) more visible (P12, P16).

5 Discussion

Non-functional requirements (NFRs) have become an important research area, mainly due to the abundance of project failures caused by neglecting quality attributes related to user values. While there is no consensus on the reasons for this, Maxim and Kessentini point out that NFRs “are not easy for stakeholders to articulate, but they know that the software will not be usable without some of these non-functional characteristics” [94]. Similarly, the four most frequent issues identified in our study concern neglecting (I1), a lack of (I3), or misunderstanding (I2) NFRs. Further to this, even when one manages to write them down, difficulties are encountered along the way while attempting to document particular qualities and their dependencies (I4).

Notwithstanding these observations, one question arises: Why have been NFRs disregarded? The first reason is the insufficient knowledge and low competence of the employees (I7), particularly in terms of their analytical skills and professional experience, reflected by their inability to perform a required task at a targeted level of proficiency. Moreover, their incompetence is also demonstrated by overlooking sources of NFRs (I8), vague definitions and obscure descriptions (I9). These burdens might be considered to be a result of ambiguous communication (I14).

The remaining aspects of the discussion, namely: comparison to works by other authors (5.1), study limitations (5.2) and implications (5.3) are respectively given below.

5.1 Comparison with Related Works

There are only a few sources dedicated to identification of NFR-related issues and/or practices in ASD. Alsaquaf et al. consider NFRs in a more specific context of Agile Large-Scale Distributed projects. These authors conducted an SLR study to summarize the challenges and practices mentioned in the literature [30] and further investigated additional challenges through a series of interviews with industry practitioners [52]. Behutiye et al. [85] consider NFRs in the generic context of ASD. They used situational method engineering to analyze NFR management practices and interviews to identify challenges and practices of NFR documentation [71].

The SLR by Alsaquaf et al. [30] uncovered 12 NFR-related issues and 13 practices used as solutions. We were able to identify more items of both categories. The results of both above-mentioned primary studies ([52] and [85]) were retrieved in our SLR study and included in its results, together with a wider set of issues/practices from other sources. It is worth to notice that 16 out of 24 issues found in our SLR were identified in a series of interviews dedicated to NFRs challenges in Agile Large-Scale Distributed (ALSD) development [52], which indicates that such challenges are not limited to ALSD, but apply to ASD projects in general. Such findings are also corroborated by reports in other sources we were able to retrieve in our SLR study.

A larger number of research studies on issues and/or practices is available, however their scope is wider and concerns e.g. the challenges of Agile requirements engineering in general. An SLR study by Inayat et al. summarizes existing requirements engineering challenges, Agile practices that address such challenges as well as additional challenges related to Agile practices [28]. Heikilla et al. identified, through a mapping study, Agile requirements engineering benefits as well as its problematic areas [95]. Medeiros et al. conducted a mapping study focused on Agile requirements engineering practices and techniques dedicated to requirements elicitation and documentation [51]. Schon et al. focuses on introducing the user’s perspective to ASD and joint application of ASD and User-Centered Design (UCD). They conducted an SLR study to summarize the related practices and identify essential aspects of Agile requirements engineering in the UCD context [27].

Apart from [30, 71] and [85], none these studies considers NFRs specifically and as such, they do not address detailed practices nor issues. Some of them enumerate single NFR-related issues, mostly mentioning the risk of neglecting NFRs in ASD projects [27, 28] and to the typical Agile documentation insufficient to successfully capture NFRs [28, 95]. It is worth noting here that usually, the issues/practices mentioned in such papers are not explicitly assigned to NFRs but to requirements in general, also including FRs, constraints, business goals etc. With respect to this, our study has a much more narrow scope, but within this scope provides much more detailed findings.

Finally, there are studies that focus on a single issue and/or practice, as they propose a new method, technique or practice to address some known issue, e.g. a lack of NFR traceability or difficulty in documenting NFRs of some kind. In our study, we made an attempt to include such contributions into the summary lists that provide the reader with an overview of the state of the art on NFRs in the ASD topic.

5.2 Limitations

While we try to minimize risks by following the guidelines by Kitchenham and Charters [47], we are aware that our study has several possible limitations remaining. A limitation of any SLR study is the potential bias in selecting the sources. This starts with the decisions regarding the publication databases to be searched and the search string to be used for this purpose.

Our study conducted the search in Elsevier Scopus only. Scopus is known to enable a single search query to access items from a large number of journals and conferences [49, 50]. However, it is possible that it misses some sources that could in turn be found in other databases. Moreover, scientific databases do not include so called “grey literature” which can potentially include industry experiences in non-scientific publications, such as reports and web articles.

We paid attention to the construction of the search string and included several synonyms and alternative terms to increase our chances of finding all of the relevant sources. The SLR is, however, strongly dependent on vocabulary and we cannot rule out that some authors used less common expressions which would lead to failure to find their papers. Moreover, as our search string implied that NFRs must be explicitly mentioned in the paper’s title, abstract or keywords, more generic papers (e.g. dedicated to ASD challenges or practices), which just mention an issue or practice related to NFRs among many others, could be missed.

The extraction of the data from the sources is also a task prone to bias, as it is done by humans, who interpret the contents of the sources. Following SLR guidelines minimizes such a threat, but cannot entirely eliminate it.

5.3 Implications for Research and Practice

This study has several implications for both researchers and practitioners. For researchers, the relatively small number of sources retrieved for the SLR indicates that there is a need for more studies regarding NFRs in ASD projects. Moreover, the issues and practices listed in the findings of our study can be considered by researchers as potential subjects of dedicated empirical studies, further exploring, e.g. the root causes of reported issues or the effectiveness of the facilitating practices.

Industrial practitioners can use our findings to anticipate issues in projects they participate in, and to select facilitating practices to be applied in their projects. Our study can also be used to raise awareness on NFRs in ASD, the related issues and practices as well as the overall importance of such a topic - which still tends to be neglected or lack sufficient attention.

6 Conclusions

In this paper, we explored the topic of the implementation of non-functional requirements (NFRs) in Agile Software Development (ASD), focusing on the issues and facilitating practices, gathered from the existing body of literature. The main motivation underpinning this study was to investigate the state-of-the-art in implementing NFRs within ASD projects, through a systematic literature review (SLR), in order to identify what issues have been documented, as well as by what means one can facilitate the implementation of NFRs. This research was driven by the guidelines elaborated by Kitchenham and Charters, a well-known and widely-applied research framework in the field of software engineering.

The number of industrial projects that deliver different and specific lessons regarding NFRs, makes comparison of the studies a time-consuming and intricate task, since they do not frequently deal with the same focus or goals. Nevertheless, we were able to collect and unambiguously classify a bulk of issues and practices, extracted from peer-reviewed scientific sources. Obviously, our report neither exhausts the topic nor provides external validity. However, we believe that the obtained results may serve as a useful reference repository to be used by both experienced and novice researchers, as well as senior and junior practitioners.

Moreover, a number of open issues and related research directions were identified through this study, which can be considered as an input for future work. A clear lack of consensus on which requirements documentation techniques should be used in order to specify NFRs. Some sources suggest using the techniques available in Agile methods e.g. User Stories (possibly with some adjustments), while others recommend introducing additional techniques (including those used in more traditional, plan-driven approaches). Such selection of the suitable techniques may be a context-dependent issue and requires further investigation. Other areas we identify as potential future work directions are: the relationship between requirements engineering and testing areas (test specifications to verify the implementation of NFRs, which can also be considered as part of NFRs documentation); and the facilitation of an unambiguous NFRs communication process.

We plan to address the latter topic by designing and testing an ontology-based approach, using Controlled Natural Language (CNL) [96] and the Fluent Editor [97], simulating and modelling requirements specification. Our observations gathered during professional work indicate that there is still a need to implement suitable methods and tools to support communication among different groups of stakeholders and development teams.