1 Introduction

The complexity of software development activities makes it necessary to provide engineers with innovative tools and techniques to support them. In this context, the application of search and optimisation methods represents a golden opportunity to automate tedious tasks, providing novel mechanisms to support from planning, design or coding to testing and maintenance activities. Most phases of the software life cycle could greatly benefit from the so-called search-based software engineering (SBSE) paradigm [28], which also explains why the field has attracted great interest in both academy and industry.

Intense scientific activity within the SBSE international community is clearly reflected in specialised repositories like the CREST database [54], both general [26] and phase-specific surveys [1, 40, 41] and bibliometric analysis [24]. However, as already pointed out in 2008 [30], the exponential growth of publications in the last years makes it difficult to carry out a global analysis of the state of the SBSE community. For this reason, it is also possible to find more geographically focused studies, e.g. Brazil [14]. Significantly, this type of study is useful to take the pulse of specific communities, as well as to analyse their maturity and particular interests and interactions.

Since its conception in 2001, and specially in the last decade, SBSE has been actively investigated in Spain, and researchers have often contributed to the field with new topics and techniques. The recent appearance of specialised events and international collaborations seems to indicate that it is the right time to assess what has been done and which are the current and future interests of the Spanish community. A rigorous analysis in form of a systematic literature review (SLR) will provide a structured view of the field in Spain, as well as an indicator of its contribution to the global area of knowledge. With this aim, a rigorous review protocol has been designed describing the method to be followed, as well as the classification scheme that guides the data extraction process.

More specifically, this paper aims at responding to the following three research questions (RQs):

RQ1:

In quantitative terms, what is the research contribution of the Spanish SBSE community? From the more than 3700 papers initially retrieved, 232 primary studies were selected. A quantitative analysis of these relevant works, authored by 145 researchers from 19 institutions, identified publication trends and current collaborations.

RQ2:

Which are the software engineering (SE) activities addressed and the techniques explored to tackle these activities? Primary studies have been evaluated in terms of the optimisation problem, the search techniques and its empirical validation. From these outcomes, it is possible to provide an overview of the expertise of the community.

RQ3:

Which is the evolution and positioning of the Spanish community within the international community? The results have been contextualised with respect to the SBSE international community, taking the presence of the Spanish community in SBSE specialised events as a reference. We also discuss the evolution of topics with the information available in other similar literature reviews.

RQ4:

What are the current trends and open gaps for future research? The analysis of primary studies, especially their future research lines, pending work and open gaps, allows extracting and analysing current trends in the area, as well as those aspects still requiring further efforts.

Results of analysis show that the contributions of the Spanish SBSE community cover a wide range of problems and techniques. This fact has probably become a key factor to allow positioning itself as an important contributor to the international SBSE community. Nevertheless, there are other topics that are currently gaining attention from the international community, such as code refactoring or the optimisation of non-functional properties, but they still are rarely explored in Spain. Tool-oriented research and better experimental reproducibility are other aspects needing further efforts from the Spanish SBSE community to gain global visibility both in the industry and in the academia. The review has also served to identify open issues and current trends. Among the aspects to be improved, a closer approach to the industry would be desirable, in order to turn the transfer of results into a reality. Moreover, the interest in advanced techniques, such as many-objective optimisation, is highlighted.

The rest of the paper is organised as follows. Section 2 describes the review methodology, including the search procedure and data extraction process. Section 3 provides a quantitative analysis of publications, institutions and authors. The SE activities and the search techniques explored by the community are analysed in Sect. 4. Section 5 is focused on the evolution of the community and its position within the international scenario. Section 6 presents current and future trends, while threats to validity are identified in Sect. 7. Finally, Sect. 8 concludes.

2 Methodology

This SLR has been thoroughly conducted according to the guidelines specifically developed for systematic reviews within the field of Software Engineering [34]. The aim of this methodology—conveniently adapted for this community—is to perform an exhaustive procedure to obtain the list of authors and primary studies, from which relevant information will be extracted and analysed. In the next sections, the search procedure and the data extraction process are explained in detail. Note that a complete definition of this review protocol and classification scheme is fully available (see Additional Material).

2.1 Search procedure

The search procedure is depicted in Fig. 1. Notice that the search should consider two particularities of this review: (a) the search is limited to a very specific community, and (b) the review will cover both research contributions and members of the Spanish community. Therefore, the automatic search is seemingly not enough because, even when the search can be filtered by country, it might discard some relevant contributions. Thus, manual search of authors—including snowballing—and carefully scrutinising the outcomes is required to obtain a complete picture of the Spanish SBSE community.

Fig. 1
figure 1

Search procedure to identify the Spanish SBSE community

Full size image

A first step consists in conducting a number of pilot searches of authors and papers, both manually and automatically via search engines. These pilot searches serve to check how accurate the search strings are, as long as they were defined considering previous recommendations in the field [26]. Table 1 shows the resulting search string, together with the list of terms to be queried for each specific phase.

Table 1 Search string and terms
Full size table

The ACM Library, IEEE Xplore, Scopus and the Web of Science are the search engines used for the automatic search. As a result, 3378 references were obtained, from which 147 authors were identified. In the case of manual search, the institutional web pages of all Spanish research institutions were accessed, as well as personal web pages and the SISTEDESFootnote 1 repository. This search identified 87 authors. After combining both lists, the number of candidate authors was established in 181.

Next, the search for candidate papers was conducted per author using additional databases, namely CREST repository, Scopus, DBLP and Google Scholar. Three hundred seventy-three international papers were returned. Manual search allowed retrieving 62 papers in Spanish. Candidate papers were revised in terms of exclusion and inclusion criteria in order to obtain the primary studies, i.e. those that will be later analysed for the literature review. With this aim, exclusion criteria are defined as follows:

  • No Spanish institution was involved in the paper.

  • The study consists in a bachelor, master or PhD thesis. Similarly, contributions to doctoral symposiums and invited talks are excluded too.

  • The study is not properly indexed by scientific databases or its full content is not available online.

  • The study has to be written in Spanish or English.

In addition, the set of non-excluded candidate papers still needs to meet the following inclusion criteria:

  • Only papers describing a SBSE approach providing detailed information about the search technique and problem formulation can be included, independently of the specific type of research article (survey, theoretical proposal, experimental study, etc.)

  • Books, book chapters and papers presenting tools can be included.

  • The study should be published between 2001 (date when the term SBSE was coined) and \(28{\mathrm{th}}\) February 2019 (date when the search was concluded) to be included.

In the end, 232 primary studies were selected for this systematic review, authored by 145 members of 19 different Spanish institutions.

2.2 Data extraction

Primary studies are distributed among data extraction teams, each team consisting of the couple of authors of this review affiliated at the same institution. This distribution is conceived to avoid any conflict of interest: the person extracting the data has not (co)authored the primary study, or does not work for the institution indicated in the primary study. Extractors must fill in a data extraction form per primary study, which is organised according to the following top categories:

  1. 1.

    Manuscript Information about the publication, including list of authors and their affiliations, type of publication, name of publisher/event, year of publication and cites in Scopus.

  2. 2.

    Problem formulation The aim is to analyse the primary study from the software engineering perspective. Existing SBSE surveys [26, 31] and the ACM Computing classificationFootnote 2 are initially taken as reference to determine the project phase and type of problem. In addition, tool support is considered within this category.

  3. 3.

    Search perspective Information about the type and nature of the search technique applied to solve the problem. The classification is mostly based on taxonomies for metaheuristic techniques [7, 8]. The number of objectives is extracted too.

  4. 4.

    Empirical validation and reproducibility Information about the benchmarks used for experimentation, the availability of source code and the application of statistical tests to validate the significance of results [4].

Questions and disagreements during the extraction process are written down, and solved later by a different extraction team. Weekly meetings are organised among participants for coordination purposes and conflict resolution.

3 Quantitative analysis

In response to RQ1, we have performed a quantitative analysis of the 232 primary studies selected. With this aim, we analyse next the number and type of publications produced, as well as the community itself in terms of the participating authors and institutions, and their collaborations.

3.1 Publications

Figure 2 shows the cumulative number of publications since 2001 and, even when the most significant increase of contributions to SBSE began in 2007, more than 50% of works have been published since 2011. In fact, the first 50 works on SBSE were published in the period 2001–2008, and only 3 years later, the number of publications had doubled. Such an increase rate has been maintained over the last decade, as in 2014 the number of publications was already close to 150, while in 2017 the number of publications had already exceeded 200.

Fig. 2
figure 2

Cumulative number of publications per type

Full size image

The contributions, a total of 232 primary studies, are divided into international journals (77), international conferences (99) and workshops (14), Spanish conferences (36) and workshops (3), and book chapters (3). It is worth noting that the number of journal papers has doubled in last 5 years (2019 excluded), substantially exceeding the number of submissions to international conferences. The years with the highest number of journal publications are 2015 (10), 2016 (10) and 2018 (11). International conferences are the preferred type of contribution, representing 42.7% of the total of publications, closely followed by international journals (33,2%). We speculate that this small difference between journal and conference papers is in line with the Spanish system for the evaluation of the scientific productivity, where journal papers are significantly better considered—and rated—than other alternatives, in contrast to other countries. Notice that the most recurrent events are the Genetic and Evolutionary Computation Conference (GECCO), with 22 studies, followed by the Symposium of Search-Based Software Engineering (SSBSE) and the International Work Conference on Artificial Neural Networks, with nine papers both. Finally, it is worth mentioning that 82.1% of the primary studies published in a Spanish conference were submitted to the SISTEDES conference on Software Engineering and Databases and, more precisely, its track on search-based software engineering, which constitutes a real meeting point for researchers in this area.

Regarding the 77 (33.2%) journal papers, it is worth remarking the large variety of publications (39 different journals were found), the Journal of Systems and Software (JSS) and Information and Software Technology (IST) being the most commonly appearing journals with 13 and 8 publications, respectively. Both journals usually appear indexed in the first quartile (Q1) of the JCR (Journal of Citations Report) within the Software Engineering category. Also, other categories like Information Systems and Artificial Intelligence have relevance for the community with publications in journals like Information Sciences (5), Expert Systems with Applications (4) or Applied Soft Computing (3).

In relation to the impact of contributions, 183 of total primary studies are indexed by Scopus (78.9%), with a total number of 3,084 cites among all of them.Footnote 3 Table 2 shows the most cited papers (11), i.e. those with more than 50 cites. They accumulate 39% of the total number of cites among the primary studies. Unlike what was observed in the bibliometric study of the first decade of SBSE [24], the authors of the most cited works do not always correspond with the authors with the greatest number of publications (see Table 3 in Sect. 3.2).

Table 2 Articles with more than 50 cites
Full size table

3.2 Institutions, people and collaborations

Figure 3 compiles the overall numbers of the Spanish institutions in terms of the people involved (Fig. 3a) and the publications generated (Fig. 3b). From the 19 institutions working (or that have worked in the past) on SBSE, 17 are universities and 2 industrial centres. In this sense, considering the number of people having co-authored a primary study at any time, the Complutense University of Madrid provides the largest number of authors (21), followed by the University of Seville (15) and the Polythecnic University of Valencia (14). Looking at the ratio of publications per author, the current average is equal to 1.8. The University of Córdoba (4) and the University of Málaga (3.9) are the institutions with the highest ratios, and UAL (2.8), US (2.7), UPO (2.5), UCA (2.2), URJC (2) and UPV/EHU (2) are also above the average. It seems remarkable that the scientific contribution of industrial partners in the area is still limited, with the only presence of ATOS and Ikerland among the primary studies.

Fig. 3
figure 3

Spanish institutions working on SBSE

Full size image

Table 3 lists the authors with 10 or more publications within the SBSE field. It is worth highlighting that Francisco Chicano and Enrique Alba were also reported as researchers with a high impact in the first decade of the SBSE field [24]. This table also compiles information about the period of activity of these researchers, revealing a reasonable continuity over time in most cases. In addition, considering those authors from the list who are still actively working on SBSE over the last year, we have found that the University of Córdoba shows the best productivity performance with 0.6 publications per author/year, followed by the University of Mondragón (0.4), Almería (0.3), Málaga (0.3), Cádiz (0.2) and Seville (0.2). From the currently active universities, Málaga (2005–2018), Seville (2005–2018) and Alcalá (2006–2018) are the institutions that have been working on the subject for longer.

Table 3 Authors with 10 or more primary studies in SBSE
Full size table

Regarding the inter-institutional cooperation, as an average, each paper is written by 1.16 Spanish institutions and 0.59 international centres. This way, considering the primary studies, the average number of authors working at Spanish institutions per paper is 2.91, while the number of international authors per paper is 0.93. From the group of 112 primary studies co-authored by more than one institution (48.3%), 31 (13.4%) were produced as a collaboration between Spanish institutions only. It is likely to be caused by the fact that the members of the Spanish SBSE community have been traditionally dispersed among other more consolidated SE communities like requirements, software product lines, testing, or even others specifically dedicated to artificial intelligence, such as metaheuristics.

Figure 4 depicts the Spanish network in terms of current collaborations, the size of nodes being proportional to the resulting number of publications. Note that the University of Seville, the University of Alcalá and the University of Málaga have an outstanding collaboration network with other Spanish research groups. In terms of international collaboration, 39.2% of primary studies were written in collaboration with at least one author from a non-Spanish institution, having found a total of 95 researchers from 55 different international centres. The most recurrent collaborators with this community are M. Harman (University College London, 9 papers), P. Kruse (Berner and Mattner Systemtechnik, 8 papers), A. Egyed and R.E. López-Herrejón (Johannes Kepler University, 8 and 7 papers, respectively), and R.M. Hierons (Brunel University, 7).

It is revealing how the Spanish community has primarily worked with international authors, rather than looking for building an internal collaboration network. As a way to mitigate this, in 2015, the Spanish Track on SBSE was held together with the SISTEDES conference for the first time.Footnote 4 Since then, this event has served as a meeting and discussion point for the Spanish community. It is only from that moment when new initiatives were taken around, just like the search for financial support through the Spanish Research Network in SBSE (SEBASENet),Footnote 5 which is used as a source to put efforts together and support the international projection of their members, e.g. short-term mobilities abroad, sponsoring the International Summer School on SBSE, or the joint organisation of other international events.

Fig. 4
figure 4

Spanish collaboration network

Full size image

4 Systematic review of the field in Spain

In response to RQ2, this section presents the analysis of the SBSE studies in terms of the addressed SE activities and the applied search techniques. In addition, we also want to remark how experiments are validated by the community. Finally, note that the full list of activities and techniques mastered by each institution is compiled in the Additional Material.

4.1 Analysis by SE activities

Table 4 shows the software engineering tasks that have been addressed by the community, classified according to the software life cycle phase, and including the percentage of primary studies focused on them. The three most prolific areas are testing and validation (48.3%), analysis and design (14.7%) and integration and deployment (12.9%). In contrast, the coding phase has not yet been explored in any of its aspects. Also, notice that, whereas testing and validation and software planning are widely covered, efforts on requirements are mostly focused on the Next Release Problem, i.e. requirements selection. We have also observed 6 papers not fitting in any life cycle phase, and identified as “General SBSE topics” in Table 4. These papers are one of the original surveys on SBSE by Clarke et al.  [13], a primary study on the relationship between SBSE and the concept of People as a Service by García-Alonso et al. [25], a paper on the workflow for SBSE experiments by Del Águila et al. [15], and more recently, two surveys on interactivity [42] and many-objective optimisation [44] in SBSE and a paper on the development of interactive algorithms by Ramírez et al. [43].

Table 4 Studied areas in SBSE
Full size table
Fig. 5
figure 5

Evolution of the SBSE studies in the Spanish community by SE phase

Full size image

The interest of the Spanish SBSE community in each phase has changed over time. To illustrate this, Fig. 5 shows the evolution of the number of publications by SE phase from 2001 to 2019. Testing and validation, highly explored between 2007 and 2019 (119 papers), was a trendy area in 2009. Analogously, analysis and design was barely explored between 2005 and 2010 but, after an increasing interest in 2014, it became the hottest topic in 2016—together with testing and validation—with up to 10 papers. Just in 2015, requirements selection had been the topic of most interest for the community. Other like software planning and integration and deployment are, however, recurrent topics but with a low number of studies per year. Notice that, apart from the primary studies, tools have also emerged as a result of the research carried out in the community, improving the applicability and transfer of the techniques and methods developed. These applications cover different areas like testing (8), analysis/design (4), requirements (2) and maintenance (2). Further information about these tools, including their references and links for download, is available as part of the Additional Material.

4.2 Analysis by techniques

In this section, we analyse the search techniques primarily used by the Spanish community in its research. It is worth mentioning that global search methods are preferred and applied in 83.2% of primary studies. Only 6.9% of papers use local search methods, while the rest (15.5%) do not clearly describe their search algorithm. Also note that some studies make use of both approaches.

Table 5 Applied techniques in the primary studies
Full size table

Table 5 shows the distribution of primary studies in terms of the applied search techniques. It should be noted that there are studies applying more than one technique, so multiple values were allowed in this case during the extraction process. Population-based metaheuristics are the most frequently used techniques (88.4%), especially evolutionary algorithms. There is great variety of evolutionary models: genetic programming (GP) [19], scatter search (SS) [6], differential evolution (DE) [9] and estimation of distribution algorithms (EDAs) [48]. Ant colony optimisation (ACO) is the most popular swarm intelligence technique [12, 16] (9.5%), though particle swarm optimisation (PSO, 1.7%) [21] or artificial bee colony optimisation (ABC, 0.9%) [10] are also alternatives. It is worth highlighting the number of papers using exact search, 46, from which 34 studies are proposing exact search as their main resolution method. Notice that classic algorithms like breadth-first search (4.7%) and depth-first search (4.3%) are still preferred in this case. In addition, we have observed that the applicability of exact search has even increased in recent years, as well as the number of distinct problems addressed by using it.

In general, we do not observe any clear preference about which specific technique is more often applied to a particular SE problem or phase. Figure 6 depicts current relationships between SE activities and search techniques. Evolutionary algorithms predominate in software planning, where only population-based metaheuristics have been applied. For requirements engineering, exact techniques, single-solution based metaheuristics and SI, specially ACO, have more presence. Focusing on the analysis and design phase, problems related to software product lines have been mostly tackled with exact algorithms, whereas architectural design has been studied from an evolutionary perspective only. In testing, model checking appears as the problem for which greater variety of techniques can be found, SI and exact search being preferred over EC. In contrast, several EC variants are applied to address the remaining problems—test case prioritisation or selection, and test data generation, with special emphasis on GAs. This type of algorithm is also the most frequently found for software migration and portability, though some studies using exact search and PSO can be read too. Another popular problem within the integration and deployment phase is web service composition, for which exact search, single-solution-based metaheuristics and evolutionary techniques have been explored. As for maintenance topics, the only paper focused on software modularisation uses an exact algorithm, while refactoring studies include single-solution-based metaheuristics, GAs and ACO among the chosen techniques.

Fig. 6
figure 6

Sankey diagram showing the relationship between SE activities and search techniques

Full size image

It is also remarkable that about 32.3% of primary studies propose an ad hoc solution to solve the corresponding problem, meaning the interest of this community in contributing with more accurate and refined techniques. Table 6 compiles the list of algorithms found after the literature review, where multi-objective techniques like NSGA-II (14.2%) and SPEA2 (9.1%) are predominant. Some recent studies have applied CMSA [20, 50] (a hybrid approach, 0.9%) and SWAY [39] (a sampling method, 0.4%), which cannot be correctly classified into any category of Table 6.

Regarding the number of objectives, more than half of all the papers (58.6%) use single-objective algorithms. Since 2015 and 2016, multi-objective approaches became almost as frequent as single-objective proposals, however, probably because of the increasing complexity of the SE problems addressed. This coincides with the international trend, where multi-objective algorithms were also hardly applied in the very beginning of the field.

Table 6 State-of-the-art algorithms applied in the primary studies
Full size table

Regarding the type of the search model, most primary studies (66.4%) use standalone, non-hybrid, sequential algorithms, especially in comparison studies. Hybrid models, i.e. combining two different algorithms like GRASP and path relinking, are present in at least 9.1% of the reviewed papers. In addition, parallel search algorithms can be found in 8 studies, while interactive algorithms—i.e., those where a human is asked to participate in the search—and hyperheuristics are mentioned in 5 and 1 of the studies, respectively.

4.3 Empirical validation

Analysis of empirical validation provides a sense of the precision of evaluation of new proposals in the SBSE literature. Statistical tests have been used in 32.3% of the studies, most of them (62 out of 75) published after 2011, when the work by Derrac et al. [17] on the use of non-parametric statistics was published. Table 7 compiles the most frequently applied tests for all aforementioned categories. The Wilcoxon rank-sum test (a.k.a. Mann–Whitney U test) is preferred for pairwise comparison (15.5%), while Friedman is the most popular test for multiple comparison (4.7%) together with the Kruskal–Wallis test (3.5%). Also, Vargha–Delaney is found in 13 primary studies, being preferred the most recurrent method to measure the effect size. In this case, notice that effect size measurements are less common (20 studies), usually appearing in combination with a pairwise or multiple comparison. However, it is remarkable that they are becoming increasingly frequent in the last few years: 55% (11) of these studies have been published between 2017 and 2018.

In addition to presenting valid and accurate approaches, it is worth highlighting the need to propose reproducible proposals. In this sense, almost 81% of primary studies provide information about the used benchmarks. An average of 4 problem instances per paper is used, where academic samples are mostly frequent (37.9%) in comparison to industrial case studies (29.7%). Synthetic benchmarks are used only in 18.5% of cases. Source code or/and data of these instances are available in only about 28% of the studies, even though this is an essential factor to achieve reproducibility.

5 Community evolution within the international scenario

This section presents a discussion of the review findings with respect to the SBSE international literature, thus allowing us to give response to RQ3. To help understand the evolution of the Spanish SBSE community, and contextualise its development within the international framework, the most relevant literature covering the beginnings of SBSE [24, 30], other SBSE topic-specific studies [33, 37, 38, 40, 45, 49], as well as an analysis of the Brazilian community [14] have been reviewed.

Focusing on quantitative data (see Sect. 3), it can be observed that the number of publications in SBSE significantly increased after 2008, experiencing an explosion in 2011. A similar phenomena also occurred in other productive SBSE communities like Brazil [14, 24]. In fact, the authors of the Brazilian review [14] reported GECCO and SSBSE as the preferred conferences by SBSE authors in this country. It is noteworthy to observe that, after 8 years, the interest in these conferences still remains in the Spanish community with 7 and 3 contributions, respectively, in these last years. In the meantime, other international conferences have also raised the attention of Spanish authors like IEEE Congress on Evolutionary Computation (CEC) and European Conference on the Applications of Evolutionary Computation (EvoApps), both with 4 primary studies. Similarly, in terms of high-impact journals, IST and JSS were the most recurrent high-impact publications up to 2011 for both the Brazilian and Spanish communities. Since then, the range of journals accepting papers in this area has increased significantly. In fact, these journals coincide with some of the most recurrent among the international community too, as the recent publication of SBSE-specific reviews seems to indicate, e.g. in Information and Software Technology on SPL [37] and refactoring [38], and in Journal of Systems and Software on software requirements [40] and project scheduling [45], as well as in IEEE Transactions on Services Computing on service composition [33].

Despite the wide interest in testing in the first years [30], Harman et al. [27] already pointed out in 2014 that the ratio of testing studies with respect to SBSE publications had progressively decreased, reaching a stable ratio of approximately 50%. This might not necessarily mean a decline in interest in testing, but into the emergence of other SE topics and tasks under study. Until that date, the Brazilian study endorsed Harman’s assertion, 60% of contributions focused on testing, in contrast to other tasks like planning (16%) or requirements (8%). In Spain, however, the situation was slightly different from the beginning. While efforts in testing are reportedly greater than other tasks (52%), it is worth observing that the variety of areas covered was clearly superior, including planning (12%) and requirements (7%), but also analysis and design (9%) and integration and deployment (18%). Besides, this trend stayed hidden in Spain for some years, especially in 2015 (35.7%) and 2016 (22.2%). Again, over the last 2 years, Harman’s ratio for testing has been confirmed as the most studied area with 61.1% (2017) and 52.4% (2018) of SBSE publications, respectively, followed by maintenance (16.7% and 9.5%) and analysis and design (11.1% and 9.5%).

As for the search techniques, the preference of evolutionary algorithms observed in Spain until 2008 (63%) was consistent with the international situation (71%) [30]. In both cases, genetic algorithms appeared as the preferred approaches. Furthermore, swarm intelligence was broadly studied by the Spanish community, with up to 14 papers, and differs from other countries such as Brasil, with just two contributions. It is also remarkable how multi-objective algorithms aroused great interest of the international community from very early, reaching 48% of contributions in Brazil between 2009 and 2011. It was not until 2011 when multi-objective algorithms began gaining interest by the Spanish community, progressively (26% of the total number of publications by 2011). Within the international scenario, the application of multi-objective algorithms in recent SBSE literature is growing, e.g. 25% of SPL publications [37] and 35% of project scheduling studies [45] adopt a multi-objective perspective. In all reviews analysing the selected techniques, NSGA-II stands out as the preferred algorithm [40, 45], or even the only technique considered, as reported by Silva et al. [49] for mutation testing.

In terms of their impact, it should be noted that some studies authored by Spanish researchers have been acknowledged as highly relevant in their respective areas. More specifically, the application of GP to effort estimation by Dolado [19] is considered as one of the “key ideas” in search-based project management [23]. The work by Alba and Chicano [3] is still considered nowadays one of the first formulations of software project scheduling as an optimisation problem [23, 45]. In this regard, Rezende et al. [45] note that Spain is the third country with more contributions to the software scheduling problem, with 18.9% of the primary studies. Also in mutation testing—especially in mutant generation—the contribution of the Spanish community is significant (13%), according to Silva et al. [49].

Table 7 Statistical tests applied in SBSE literature
Full size table

Finally, we have also analysed the contribution of the Spanish SBSE community to specialised SBSE events. To quantify this, a list of SBSE tracks and special sessions at conferences, as well as workshops, was extracted from the SBSE repository. Then, the percentage of primary studies with respect to the total number of publications was determined from available proceedings. CEC (16%) and GECCO (11%) are the conferences where the Spanish community has more presence. Focusing on the SBSE track at GECCO, it is interesting to note that primary studies appear in 11 out of the 18 editions celebrated since 2002. In contrast, the participation in the SBSE symposium (3.5%) is quite low. The Spanish community contributed with at least one work in 5 editions, the first edition (2009) being quite productive with 3 out of the 17 papers accepted (17.7%). However, no more studies have been published since 2014, which seems to indicate that the Spanish community has opted for other conferences of a more general scope to share their advances. Among the considered workshops, the search-based software testing (SBST) workshop (4.7%) is the only one where primary studies were found. It can be explained by the maturity of SBST—the unique subarea with its own event—and its popularity within the Spanish community, which was present in the first two editions. After 8 editions without contributions, the interest seems to have been renewed with 3 out of the 5 papers presented in the last 2 years. No primary study has been presented at other relevant workshops for the SBSE community, such as RAISE (Realizing Artificial Intelligence Synergies in Software Engineering) and GI (Genetic Improvement). However, it should be noted that RAISE covers other areas of AI beyond SBSE, a fact that could influence the lack of studies. The outcomes for the GI workshop are consistent with our analysis, since this topic did not appeared among the studied areas in SBSE (see Sect. 4.1).

6 Trends and open gaps

After the analysis of the selected primary studies, we can observe a great variety of search techniques used to cover a large number of SE activities within the software life cycle, as well as pending work and potential future research lines. This section responds to RQ4 by identifying current trends and open gaps in the Spanish SBSE community.

6.1 Trends

One observed trend is related to the increasing interest in multi-objective algorithms, which seems to be a consequence of the multiple criteria that a software engineer has to face when selecting the best solution for a problem. As a result, many-objective optimisation techniques allow considering an increasing number of objectives, that is, more than three objectives. From a mathematical point of view, there is no difference between the multi- and the many-objective formulation of a problem. Differences appear in practice because traditional multi-objective algorithms, e.g. NSGA-II, have very low performance when applied to a large number of objectives. The SBSE Spanish community seems to be aware of these challenges with up to five recent papers in the application of many-objective optimisation (e.g. [32, 44]), and future research should be headed towards the use of many-objective techniques along the entire software life cycle.

Most of the software engineering problems have a subjective component that is difficult to model only using a formula. Optimisation techniques allow the expert to select a solution based on objective criteria after the search ends. However, including the human in the search procedure (‘in the loop’) makes it possible not only to take into account subjective criteria, but also to reduce the computational effort required to compute the preferred solutions [42]. This is the goal of interactive algorithms, which have still a modest presence in the Spanish SBSE literature.

We can also observe a rebirth of the use of exact techniques in the SBSE community in general, and in the Spanish community in particular [29, 36]. Integer linear programming was used at the beginning of the SBSE domain and was discarded due to their poor scalability properties. For large instances this method was not able to provide a solution and heuristic an meta-heuristic methods became as popular as they are in this domain. However, recent advances in the exact methods community, the increase of the computation power, machine parallelism and the combination of exact and heuristic methods have brought exact methods again to the first line [22]. In this vein, it is worth noting that the SBSE community is opening up to other methods beyond evolutionary algorithms or swarm intelligence. Examples are SWAY [11] or novelty search [35], among others.

6.2 Open gaps

Regarding the gaps in the Software Engineering activities, we observe that the Spanish community has never contributed to the coding phase, which includes generation of source code and manipulation of existing source code to adhere to coding standards and styles. Certainly, the Spanish community has worked on software refactoring, which implies also a source-code manipulation, but only when maintenance problems had been detected. This is probably the most noticeable gap but there are still some other activities that have not received much attention by this community, even though some works on these topics have been published eventually (less than 0.5% of the primary studies). They are related to requirements prioritization, modularisation and service-oriented architectures.

The Spanish community has a preference for population-based metaheuristics. There is, however, one population-based approach not covered in the works of this community: Genetic Improvement. There is no work of the Spanish community in any of the related events on Genetic Improvement. This is a clear and important gap, due to the importance of the field in the last years. The other main gaps regarding the techniques are in the group of single-solution based metaheuristics. Variable Neighborhood Search is a single-solution-based metaheuristic used in one single primary study in recent years (2018). More remarkable is the absence of Iterated Local Search (ILS) in the primary studies. ILS is a very simple technique requiring only a local search and a perturbation method. It reported very good results in the combinatorial optimisation field. We think this is an encouraging method yet to be used in the Spanish SBSE community.

Reproducibility of empirical studies is a must in Science. With only 28.5% of the primary studies providing data to reproduce the experimentation, reproducibility becomes an important issue to improve in the future work. Not only data, but also the source code of the implemented algorithms should be made available either as additional material or in open source repositories. Long-term and externally controlled dissemination procedures should be ensured, in contrast to personal web pages, as they tend to have a limited lifetime. Empirical validation is also a pending task. However, on a positive note, it is worth remarking that completing some statistical validation is becoming a popular practice, about 51% of the studies published since 2015 applying tests to validate their proposals.

Nowadays, the different areas of knowledge become intermingled to provide more comprehensive solutions, and SBSE is seemingly not an exception. In this sense, it is likely that the use of machine learning together with the search possibilities offered by SBSE could provide an interesting alternative to obtain solutions that are not only optimal or quasi-optimal, but also better adjusted to the organisational policy or the historical particularities of the development team. Both approaches—SBSE and machine learning—can naturally complement each other with other research lines of recent applicability, such as the aforementioned interactive approaches or the development of more explainable models.

We observed that the presence of industrial partners in the studies is still limited, with only two industrial partners among the 19 institutions. Further cooperation with Spanish companies would mainly have three benefits: 1) the Spanish software industry could increase productivity, quality, and competitiveness by including search techniques in their products; 2) academic partners would be able to face real-world problems, making an actual transfer of ideas to the industry, getting feedback and improving research outcomes; and 3) industrial benchmarks would increase their presence in research studies. We speculate that industrial partners could be interested in automating tasks related to source code manipulation. This can be specially true if the company is applying some agile methodology, where the planning phase is less important compared to the coding and testing phases. Increasing the number of industrial partners in the SBSE field could, thus, increase the interest in the coding phase, which has not been explored yet in Spain (see Sect. 4.1). An Academia-Industry collaboration might conflict with reproducibility, since companies are often reluctant to share problem details, implying the need to make extra efforts to successfully find replicable scenarios under confidentiality agreements. On the other hand, it would help to find problems more adjusted to the industrial reality, as well as to evolve the area as the needs of the companies do.

7 Threats to validity

Threats to validity mostly relate to the search procedure and the selection of primary studies, as well as to the outcomes obtained from the subsequent data extraction process. Section 2.1 introduces the thoroughly applied search procedure, and the complete review protocol is publicly available (see Additional Material). Automatic searches from diverse sources were combined, each database requiring the adaptation of the search string syntax. Several pilot searches were conducted based on a wide range of terms in order to obtain accurate search strings. In addition, manual search was intended to prevent missing relevant results. However, notice that papers presented at Spanish events are not usually indexed by scientific databases and, consequently, proceedings can be unreachable. For example, only papers after 2012 are indexed by SISTEDES, while research published by AEPIAFootnote 6 is not usually available online. The search string may seem to be missing terms as “empirical”, “industrial” or “case study”. However, the search is driven by authors, instead of papers, as outlined in Sect. 2.1. Thus, we initially performed an automatic search for authors with the search string shown in the paper, and then we completed it with a manual search. We speculate that it is not likely that we have missed relevant papers reporting empirical or industrial studies. Besides, after preliminary pilot searches, the search string including additional terms found many false positives, i.e. papers that are not strictly related to the software engineering field.

The classification scheme used for data extraction has been meticulously adapted from a well-known survey on SBSE [26] and the ACM Computing classification. Additional surveys and reviews have served as a reference regarding the search techniques [7, 8] and best experimental practices [4]. As for the extraction process, primary studies were randomly distributed for review, though conflicts of interest were considered at this point. To avoid any misclassification, reviewers were allowed to use ‘N/A’ (not available or not applicable). Except for severe disagreements, reviewers of a primary study were at the same institution in order to facilitate their communication.

8 Conclusions

This literature review reveals an active community in Spain working on search-based software engineering, comprising 145 authors from 19 institutions. The community strongly contributes to the development of the overall field, especially in the last decade, with more than 200 publications presented in international conferences and journals, as well as in Spanish conferences. The outcomes of the literature analysis reveals that the community closely collaborate with researchers from other international institutions. In addition, the birth in 2015 of a new Spanish Track on SBSE, jointly held with the main Software Engineering Spanish conference, has strengthened the collaborations at the national level and permitted the application for financial support, including a Research Network that has also supported international events like the International Summer School on search-based software engineering.

With respect to primary studies, it is worth mentioning that most phases of the software development life cycle are covered. Similar to what is observed in the global SBSE area, testing and validation happen to be the most studied topics, though important contributions have also been made to other phases like software analysis and design, and integration and deployment. Regarding the search techniques, population-based metaheuristics are preferred, in line with the practices of the international community. In contrast, other SE tasks and activities like non-functional requirements or refactoring, which are of great interest outside the Spanish community, have not been studied yet.

As for the search techniques, apart from evolutionary algorithms and swarm intelligence, single-solution metaheuristics and exact techniques are also common in the researchers’ toolkit, showing the use of alternative proposals to reach competitive results. Many-objective and interactive optimisation algorithms seem to be two more hot topics in the Spanish community, and future work will probably continue including these two sorts of algorithm. However, there is a lack of proposals applying complementary approaches like machine learning, which is becoming a recurrent practice in other forums.

Urgent challenges remain for the Spanish SBSE community. Some important examples are the improvement of the empirical validation and the reproducibility of their experimental studies. Besides, leading a further cooperation with industrial partners would allow these groups to face realistic scenarios with an increasing complexity as companies demand the solution of new SE problems.

9 Additional material

The systematic review protocol is provided separately as additional material. This document thoroughly details the method followed for the development of this paper. For the search process, the query strings and data sources that guided the search strategy are listed. The selection of candidate and primary studies and the data extraction process are explained, including data extractors and the mechanisms for conflict resolution. In addition, the protocol specifies the classification scheme for the categorisation of data extracted from the analysed studies. Due to space constraints, the entire list of primary studies, and their full references, are provided as additional material, too.

Additional material is available from the website https://www.uco.es/SEBASENet/SLR2019.