Keywords

1 Introduction

Agile methodology has emerged as an alternative to the traditional, sequential way of software development and is also responsible for promoting several changes in how software development organizations work. Instead of following a sequential model, the agile methodology has an iterative and incremental approach, where the product is divided into small parts that are continuously developed and delivered to the customer [1,2,3].

In 2005 Fowler [4] evidenced that the interest and search for agile methodologies has grown since the early 2000s, this growth in interest for agile methods is due to the success in reducing costs, responsiveness to change, meeting and understanding customer requirements as stated Rising [5], in improving product quality, point out Jain, Sharma, and Ahuja [6] and also processes, such as the testing process, according to Patuci and Morais [7].

Given this context, this work developed a study following the guidelines of a model proposed by Kitchenham [8] in the construction of a Systematic Literature Review (SLR). According to the authors, a systematic approach is predefined using a protocol to identify, evaluate and interpret the available evidence in primary studies related to one or more research questions. This study was inspired on the SLR [9]. Additionally, it was carried out to identify articles available in the literature that present evidence related to new research activities in software testing that can still be developed and discussed in the WBMA.

The structure of this paper is organized as follows. In the Subsect. 2.1 we introduce the event where the papers analyzed here were published, followed by the Subsect. 2.2, where we describe agile testing according to the Agile Manifesto. The Sect. 3 presents the research method used to conduct this study, as well as the steps and questions we aim to answer in this study. The Sect. 4 presents the research results, answering the research questions and analyzing the data. The Sect. 5 shows related and already conducted work that differs from this study. The Sect. 6 discusses the limitations and implications of conducting our analysis. And finally, the Sect. 7 brings implications of our study and points out opportunities for future work.

2 Background

2.1 Brazilian Workshop on Agile Methods

The Brazilian Workshop for Agile Methods (WBMA) is a scientific research track event at the Agile Brazil conference. It is an event about Agile Software development, where during all its editions, they received an expressive number of submissions of research papers from higher education institutions and also papers in partnership with the market, counts with the participation of a large number of students, researchers and professionals from various parts of the world.

One of the primary objectives of the event is to promote greater integration between industry and academia to stimulate the generation of ideas, innovation, and opportunities for all. The main topics of interest at the event are:

  • Adoption of Agile/Lean

  • Agile Methods for Systems and Software Architecture

  • Agile for the Development of Critical Systems

  • Human and Social Aspects in Agile Methods

  • Agile Project Management, Outsourcing and Governance in the Agile context

  • Development in Distributed and Global Agile Teams

  • DevOps and Continuous Delivery

  • Agile and Lean Teaching and Coaching

  • Scaling Agile for Large Corporations

  • Experimental Studies in Agile/Lean

  • Tools and Techniques in Agile/Lean

  • Conceptual Studies and Theoretical Foundations in Agile/Lean

  • Technical Debt Management

  • Innovation and Agile Entrepreneurship

  • Lean Startups

  • Agile/Lean in Large and Distributed Teams

  • Agile in Government

  • Metrics, Measurements and Mining Repositories in Agile contexts

  • Patterns and Anti-Patterns in the Agile/Lean application

  • User Roles (co-participation)

  • Agile Practices and Trends, Evolution and Revolution (Techniques and Management)

  • Agile Principles, Lean Manufacturing and other disciplines

  • Agile Testing and Quality: Techniques, Standards and Automated Support

  • Organizational Transformation and Cultural Aspects in Agile Business

  • User Experience (UX) em Agile/Lean

  • Test-driven Development

  • Refactoring Techniques

  • Continuous Inspection Techniques.

However, papers on other topics, as long as they are related to agile methods, are welcome at the event.

The event has space for full papers containing relevant results of original research and for short papers that address ongoing research, experience reports, bibliographic research, or opinion articles [10]. Among the topics of interest, it is relevant for this article to address the papers presented during the ten editions of the Workshop that fall under discussions related to software testing and quality.

2.2 Agile Testing

Software development using agile methodologies promotes severe changes in how software was traditionally built. One of its goals is to demonstrate that functionality is as defined in the requirements and to identify as many bugs as possible. This also includes how development stages are performed, such as software testing [11]. Traditionally people view testing as a phase that happens at the end of development. In agile most have changed it that the chunk of development done is smaller, but the testing still happens last. Nothing has fundamentally changed about how testing is done.

One way to see if this is the case is to ask people about their taskboards. If taskboards have a separate column for testing, it’s a sure sign that testing is still being thought of as a phase.

In contrast in agile, testing is just an activity that needs to happen, along with coding, documentation and everything else. Thinking about it like this makes it possible to consider the idea of doing testing tasks before development work. A great way to visualise this on a taskboard is that instead of having a separate column for test, rather just make testing tasks a different colour sticky note. Now put all the tasks in the “To do” column together. Challenge the team to see how many of the testing tasks they can do before any development tasks happen.

For example you can create test cases before any code is written. That way you know how you are going to test it before you build it. You could even create automated acceptance tests first. These should fail since there is no code yet, but once the code is written and the tests pass, the work is done, and there are no test tasks left. Working this way will remove the hurdle of testing always being behind. For some people this is a huge step, however just breaking the mentality that testing tasks follow development is a great start.

Another useful technique is the “Show Me” column. Put it after the “In Progress” column, before the “Done” column. Most teams do code reviews, documentation reviews or even test case reviews on each story. The idea behind the “Show Me” column is to do a review on every task as soon as that task is done. If tasks are small these are micro reviews that might only take a few minutes, but they ensure that at least two people in the team have seen every piece of work, and this can help catch and fix issues much earlier.

Developing quality software is one of the pillars of development in environments that adopt agile methods. These activities have received attention since the beginning of the history of agile development, and many agile practices rely on effective testing [12]. Agile testing is a software testing practice that follows the principles of agile software development [13]. It involves testing activities as early as possible, as stable versions of the code are made available, as agile methods feature incremental product development and delivery [14].

Furthermore, Agile Testing seeks to promote broader thinking about how quality should be adopted and who is responsible for it. Thus, attitudes such as testing at each stage of development instead of just testing at the end; preventing defects from happening instead of having strategies that focus only on finding them; and that the entire Agile team should be responsible for quality instead of just the testers, should always be explored and enhanced [15].

3 Research Method

For this work, we followed the guidelines of a model proposed by Kitchenham [8] in constructing a Systematic Literature Review to develop this mapping study. According to Kitchenham, an SLR is carried out to search for and quantitatively and qualitatively evaluate the studies already conducted in the area that have been published or made available to the scientific community and that have specific academic importance as a basis for research. To this end, it is necessary to determine research questions and, based on these questions, survey the essential data to answer them, thus guiding the reading and selection of articles to select those that will be part of the SLR. It is necessary, therefore, to establish inclusion and exclusion criteria before conducting the research.

The protocol used as a guide throughout the SLR for evidence collection was developed from the formulated questions and these criteria. One of the goals of this work is to present background and, from it, determine which new research activities in software testing can still be developed and discussed in the WBMA. Although it is not an SLR, we adopted the proposed model, so we followed the following phases to conduct this study: Planning and research, Analysis and studies, and Results.

The steps were organized as follows: in Planning and research, we formulated the review questions and defined inclusion and exclusion criteria and search strategy; in Analysis and studies, we performed the selection of studies, analyzed their quality and relevance to our research, and worked on the data extraction process; finally, in the Results step we presented the Analysis and presentation of results, as well as interpreting them.

3.1 Research Questions

One of the essential steps of this analysis activity was the definition of the research questions that led to the search for relevant documents aiming to answer the study’s central question: “What is the relevance of the papers published in the WBMA for the testing scenario in Brazil?”

This question was derived from five others with different focuses:

  • RQ1. Who are the main authors that have published articles (and other materials) related to software testing at the WBMA?

  • RQ2. What are the main institutions that have published articles (and other materials) related to software testing at the WBMA?

  • RQ3. What are the main topics discussed at the WBMA related to software testing?

  • RQ4. What are the main types of contributions to software testing in the WBMA?

  • RQ5. Which articles published in the WBMA related to software testing have the most impact (cited by)?

3.2 Search Strategy, Sources and Selection of Papers

Since we are performing this analysis only on papers published in WBMA events held from 2010 to 2019, we adopted the following inclusion and exclusion criteria:

Inclusion Criteria:

  • I1: All papers published in WBMA between 2010 and 2019;

  • I2: Articles with software testing topics as the main focus.

Exclusion Criteria:

  • E1: All papers that have not been published in the WBMA;

  • E2: Papers that were not related to software testing;

  • E3: Papers that could not be accessed, even after contacting the authors;

  • E4: Papers that are not written in English or Portuguese;

  • E5: Presenters’ notes, round table papers, workshop reports, invited papers, books presented or released at the event, theses and dissertations;

  • E6: Incomplete documents, drafts, presentation slides, and extended abstracts;

  • E7: Papers that treat software testing only as part of future work.

A significant number of papers (full papers and short) were published in the WBMA. We followed three filtering or selection steps to arrive at the final result of 12 selected articles. In the first step, we read the titles, abstracts, and keywords and had an initial selection of 22 articles. The data from these articles were organized and listed in a Google spreadsheetFootnote 1.

In the next step, the data were extracted from the introduction and conclusion of the works, totaling 18 articles, also listed in a Google spreadsheetFootnote 2. For the final step, the full articles were read and a total of 12 articles were selected, listed in detail in this spreadsheet onlineFootnote 3.

3.3 Data Extraction

The next step in this work is to answer the research questions. For this, the data from the initial 18 articles were extracted and organized, for this, we used a Google spreadsheet (links in session 2.2) and, thus, all the data collected that guided us to this mapping were organized. The information for each article is:

  • Identifieritem;

  • Title;

  • Year of publication;

  • Author, Institution and Federal Unit;

  • Type of contribution;

  • Topics;

  • Number of citations;

  • Empirical study.

4 Results

This section presents the results that were observed in our study analysis. All articles used to develop the results below respect the inclusion and exclusion criteria in Sect. 2.2. The papers were published in the issues between 2010 and 2019: 1 article in the years 2011, 2013, 2017, and 2018; 2 articles in the years 2010, 2014, 2016, and 2019, totaling 12 articles that are part of this study.

4.1 RQ1: Which are the Main Authors Who have Published Papers (and Other Materials) Related to Software Testing in the WBMA?

Many authors contributed to the event during the first decade of the WBMA’s existence. The selected papers were:

  • Contributions from 34 authors;

  • Sixteen different institutions (either from industry or academia);

  • Six other states in Brazil;

  • All regions of the country.

Among the 34 authors, the ones who stood out in the number of papers submitted and accepted at the event are:

  • Eduardo Martins Guerra (National Institute for Space Research and Technological Institute of Aeronautics) with three different papers;

  • Ivaldir Honório de Farias Junior (the Federal University of Pernambuco and Softex Recife) with two other papers;

  • The remaining authors contributed to 1 of the 12 papers analyzed.

Such researchers have been making significant contributions to the community outside of the WBMA, both for their output involving agile methods and software testing, these being:

  • Eduardo Martins Guerra has 468 citations in Google Scholar, 303 of them in the last five years, and his most cited article is focused on software testing. Information is taken from his Google Scholar profile pageFootnote 4;

  • Ivaldir Honório de Farias Junior also has essential studies in the area of testing and agile methods; his Google Scholar profile shows that among his most cited articles are studies done on software testingFootnote 5.

The complete list of authors and their test-related papers published in the WBMA can be seen in the spreadsheetFootnote 6. These data have been organized in Table 1.

Table 1. Authors who have published the most in WBMA
Full size table

4.2 RQ2: What are the Main Institutions that have Published Articles (and Other Materials) Related to Software Testing at the WBMA?

Analyzing Table 2, among the 16 institutions (industry and academia) that published, the Federal University of Pernambuco stands out, which published 3 of the 12 articles selected in this study, followed by the institutions Federal University of São Paulo, Federal Technological University of Paraná and National Institute for Space Research, with 2 published articles each. The other 12 institutions collaboratively participated in the production of 1 article each.

Consistent with the results of RQ1, UFPE has as a member of its team the researcher Ivaldir Honório de Farias Junior, one of the authors with the highest number of publications in the event. Besides, UFPE’s graduate program in Computer Science has the maximum CAPES concept (grade 7), which may contribute to the strong presence of UFPE in the event.

Table 2. Institutions that published the most in the WBMA
Full size table

There were contributions from 6 states and five regions of the country:

  • São Paulo is the state with the most significant number of publications, 5 in total;

  • Pernambuco, with 3 publications;

  • Paraná with 2 publications;

  • Rio Grande do Sul, Mato Grosso and Amazonas contributed with 1 paper each.

Figures 1 and 2 show these state and region distributions, respectively.

Fig. 1.
figure 1

Contribution to the WBMA by state

Full size image
Fig. 2.
figure 2

Contribution to the WBMA by region

Full size image

4.3 RQ3: Which Articles Published in the WBMA Related to Software Testing have the Most Impact (Cited By)?

The answer to this question can be obtained by checking the keywords in the articles surveyed. It is important to note that some articles do not have keywords and that even if they do, they may not be enough to categorize our research in a way that clarifies the main idea of the articles. As we also verified a certain redundancy in the keyword definitions of some articles, we decided to define the main topics according to what we proved to be the general idea of each work, respecting the fact that they may address more than one of the defined topics.

Thus, we observed that the most covered topics were:

  • “Study of Agile Methodologies in Testing,” cited by 7 of the published papers, representing 58.33% of the papers;

  • “Adaptation to the agile model,” cited by 6 of the published papers, representing 50% of the papers;

  • “Test Support Tool/Framework” cited in 4 of the submitted papers, corresponding to 33.33%.

This data shows that the first two topics address the agile model, where 3 of the papers appear in both.

In addition to these 3 points above, if we look at all the topics in more than one article, we can see five significant issues being addressed among the papers used for this study. The complete listing of articles is described in Table 3.

Table 3. Most discussed topics among the papers published at the WBMA
Full size table

4.4 RQ4: What are the Main Types of Contributions to Software Testing in the WBMA?

The answer to this question can be obtained by considering how the authors have classified their articles. The list with this classification can be seen in the “Type of contribution” tab of the spreadsheetFootnote 7. The idea behind this question is to answer the contributions of these papers published within the WBMA to software testing. Thus, it was found that the three main types of contributions of the analyzed papers were:

  • Improvements (of processes or practices) with 4 articles;

  • Use Experience (of type of process applied, application domain or purpose) and Tools with 3 articles.

The complete list (Table 4) also includes Best Practices followed by Definition, containing 1 article. One can infer that, during the ten years of the WBMA, there was a great concern on the part of the authors in contributing to the maturing of the, by then, new agile model, seeking to support its adhesion and constantly worrying about its impact on how the software testing process would be carried out.

Table 4. Main types of contributions to the WBMA
Full size table

5 Related Work

The Brazilian Workshop on Agile Methodologies study focused on software testing has not been developed so far; no similar studies were found in any of the bibliographic bases we used. This study contributes to the analysis of the WBMA, the authors, and the different topics related to software testing and identifies which areas still need study and which still have gaps even though researched. As well as a comparison of the progress of the studies in these ten years of existence.

6 Limitations and Threats

The study carried out presented significant limitations regarding its sample, these being:

  • The sample would be the papers on testing during the ten years of the event, there were different topics presented within the large area of testing, so we could not perform many comparative studies on the progress of a particular area during this period;

  • Another limitation that directly impacts the research results is the lack of studies on important topics in the area of software testing;

  • Finally, another significant limitation is the sample size, which presents a reduced number of articles and allows us to consider the results found only for the population in question.

7 Conclusion and Future Works

In this paper, we reviewed which are the software testing-related papers published at WBMA during the ten years of the event and what were their contributions to the community. We could present some information that can be used to provide people with a more detailed analysis of these papers to understand how much they have contributed to the evolution of software testing, always pointing in a better direction.

Section 2 provides the results of our mapping, where the most active authors and institutions showing participation and interest in the topics related to this field of study are shown in detail. It is essential to highlight the issues that appeared the most in our survey: Improvements (of processes or practices), Experience of Use (of type of process applied, application domain or purpose), and Tools.

Through this survey, we could notice the community’s desires during these 10 years, how to adapt to an agile context and how to insert software testing in this transformation process, either by new tools or by modifying the current process. This paper presents an overview of the importance of the event in Brazil and the positive impacts for the study of software testing.

In future work, we consider that a similar analysis in the other areas addressed by the WBMA could help define an understanding of the event’s impact on the Brazilian community, which areas need more investment in research and primary research opportunities.