Keywords

1 Introduction

Regression Testing (RT) is foremost the crucial part of software life cycle. Whenever software changes RT is done in order to ensure the proper functioning of a software [13]. With the changes in a software, the size of test suite also varies. As the software evolves, the magnitude of test suite size becomes quite large. Rationally it is not feasible to execute all the test cases due to limited time and cost constraints. Therefore, it becomes important to deduce a mechanism to perform efficient RT with maximum number of faults revealing test cases in minimum time. So, test case optimization techniques are required to perform early fault detection in affordable time and cost [3]. Regression Test Case Optimization (RTCO) is further categorized into three main branches of test case optimization namely; Retest All [15], Test Case Selection [4] and Test Case Prioritization (TCP) [14]. In Retest All technique, all the test cases in original test suite are executed. This technique is time consuming and very expensive. Regression Test Case Selection executes subset of a test suite. Whereas, regression TCP techniques [6] deals with maximum and early fault detection by ordering the test cases based on priority given to them. Determining the faults covered by test case in advance is very difficult and time-consuming process, since each test case must be executed in advance to determine its fault coverage.

In the recent times Nature Inspired Metaheuristic Optimization Algorithms (NIMOA) have gained huge popularity [22]. Most of the NIMOA belong to the domain of Stochastic Algorithms (SA). SA are those algorithms whose outcome does not depend on its starting point as they follow a random search. Keeping in view the success of SA numerous algorithms for TCP in RT have been proposed in recent times [6, 7, 12, 21]. In 2012, a new algorithm namely, Flower Pollination Algorithm (FPA) was developed by Xin-She Yang [1]. FPA is inspired by the pollination process in flowering plants. Since, FPA is an optimization algorithm [8] therefore, in this paper TCP was done using FPA without any prior knowledge of faults covered by the test cases.

2 Related Work

Numerous studies have surfaced recently that adopted FPA in different domains like engineering, wireless sensor networking, image and signal processing, communication, structural design, computer gaming, software engineering and global function optimization [2, 11].

Alsewari et al. [1] proposed a strategy for Test Case Minimization based on FPA. Their strategy is termed as Test Generator Flower Pollination Strategy (TGFP). Their research is based on t-way testing. Where ‘t’ represents the interaction strength between parameters. For different interaction strength their strategy performed well. Nasser et al. [10] proposed a hybridized FPA for generating t-way test suite. The main hybridized features used in their research are local search, mutation operator and elitism feature. It is shown in their work that elitism based FPA performed better than other variants of FPA.

Kabir et al. [5] proposed a technique to minimization test suite for t-way testing by using adaptive FPA. Their comparative analysis with existing techniques shows that their algorithm showed improved performance. Nasser et al. [9] proposed a Flower Strategy (FS) for generation of t-way test suite based on FPA. The research also critically performed the comparison of adoption Optimization Algorithm (OA). Their results show that FS outperforms OA based strategies.

Yadav and Dutta [21] used Genetic Algorithm (GA) to prioritize the test cases by using statement coverage as input to the algorithm. Average Percentage of Statements Covered (APSC) metrics is used to compute the efficiency of the algorithm. Their approach is compared with other techniques of prioritization viz., random prioritization and reverse prioritization. Their study gives optimum results.

Panwar et al. [12] proposed an improved version of Ant Colony Optimization (ACO) algorithm to prioritize the test cases. Triangle classification problem is used in this study. Their proposed technique is compared with different ordering of test cases viz., reverse ordering, random ordering, optimal ordering and original ordering. The antennas of ants are used to share the information for proposed approach. Their approach achieves better results as compared to other orderings of the test cases.

3 Flower Pollination Algorithm (FPA)

FPA was developed in the year 2012 by Xin-She Yang. There are four main rules for flower pollination process as discussed below [22].

  1. I.

    Global pollination is achieved through biotic and cross-pollination process, and pollen- carrying vectors transfer pollens through Levy flight (Rule 1)

  2. II.

    Local pollination is achieved through abiotic and self- pollination (Rule 2)

  3. III.

    Reproduction probability is equivalent to Flower Constancy (FC) that is proportionate to the relationship between two flowers involved (Rule 3)

  4. IV.

    Switch probability p ε [0,1] controls the global and local pollination (Rule 4)

Figure 1 represents the FPA developed by Yang [22].

Fig. 1
figure 1

Flower pollination algorithm (FPA)

Full size image

NIMOA are analyzed by the way they explore their search space [22]. In principle all algorithms have two main components i.e., exploitation and exploration which are also known as intensification and diversification respectively. Exploitation is a local search process which can leads to early convergence rate and may also get stuck at local optimum since the final solution depends on starting point. Whereas, exploration explore the global search space leading to a diverse set of solution. Exploration does not get stuck at local optimum [22]. FPA is one of the newest NIMOA [2]. Flower pollination is mainly related to the transfer of pollens, the transfer is achieved through pollinators viz., birds, insects, bats and other animals [23].

Pollination can be achieved in two ways biotic and abiotic. Nearly 90% flowering plants are a member of biotic pollination, the pollens are transferred with the help of vectors. Whereas 10% pollination belong to abiotic form where no vector is needed for transfer of pollens. The transfer of pollens take place with the help of water and wind. Various vectors develop FC [22]. FC can be defined as the affinity that is developed between the vector and the specific flower species. FC thus result in maximum transfer of pollens and reproduction of the same flower species. Biologically, the objective of flower pollination can be explained as the survival of the fittest and optimum reproduction of plants in number as well as fitness. Therefore, this natural process encourages this research to use FPA for TCP for RT.

4 FPA for TCP

This paper uses FPA for TCP to reduce overall time of execution in RT. FPA is used to prioritize test cases from the original test suite (TSo) to give maximum fault coverage in minimum time of execution. The algorithm uses code coverage criteria as an input to prioritize test cases in TSo. Original population of test cases is randomly generated and represented as TSo = (T1, T2, …, Tn). Figure 2 shows FPA for TCP.

Fig. 2
figure 2

FPA for TCP

Full size image

5 Explanation of FPA for TCP

Original population of ‘n’ test cases is generated and the test suite is identified as TSo = (T1, T2, ……, Tn). For each test case its code coverage is considered as the fitness function f(xi), where xi = (x1, x2, …., xn). The code coverage of test cases in TSo is used to prioritize the test cases. Test case with highest coverage is maintained as g*. If multiple test cases have same coverage, then First Come First Serve (FCFS) policy is adopted. A switch variable (p = 0.8) [23] is used to perform local (exploitation) or global (exploration) pollination [23]. Random number ‘rand’ (value ranging between 0 and 1) is used to determine whether local pollination will take place or global pollination will occur. If rand < p, global pollination is performed and the fittest reproduction is represented as g* at tth iteration. In global pollination, the fittest reproduction is ensured through pollens that can travel to long distance. Therefore, for t + 1 iteration the pollens/test cases are picked from the tth iteration. Here the pollens are represented as test cases. In flower pollination vector has affinity towards the pollens similarly test cases have affinity towards the fault. Those test cases with higher fault detection capability are of great value and are carried forward to the next generation of software. If we represent the fittest as g*. Then the FC and first rule (Rule 1) can be mathematically formulated as shown in Eq. (1).

$$ X_{i}^{t + 1} = X_{i}^{t} + \varUpsilon \,L\left( {g* - X_{i}^{t} } \right) $$
(1)

In Eq. (1) X t+1i is the ith pollen/test case at t + 1 iteration, X ti symbolizes the ith pollen or the ith test case at iteration t, L is the strength of the pollination and has a certain probability of distribution. In this paper all test cases/pollens are considered to have FC or affinity toward the vector carrying them for pollination. Hence giving equal chance to all the test cases/pollens to participate in pollination process (TCP). For this reason, L (i.e., step size) is equated to one. To control the step size, ϒ i.e., the scaling factor is used. It is assumed that uniform scaling is performed by vector to visit pollens. Therefore, ϒ is equated to one. Local pollination described in (Rule 2) and FC can be formulated as shown in Eq. 2.

$$ X_{i}^{t + 1} = X_{i}^{t} + \varepsilon \left( {X_{j}^{t} - X_{k}^{t} } \right) $$
(2)

Pollens are transferred from different flowers, but these flowers belong to a single plant species. This simulates the FC in a small neighbourhood. Here j = i and k = i + 1 in the tth iteration. This emulates the FC behaviour in small area. A local random walk variable ε is derived from a uniform distribution. This paper takes the value of ε as 0 or 1 representing either the test case is selected or not selected for further prioritization; since it may be the case that in a given local area two test case cover the same code. Therefore, redundant test cases are required to be discarded. Here redundant test cases are those test case which give same code coverage. Therefore, fifty percent of the chances of test cases with same coverage being selected is reduced. This result in a reduced and prioritized set of test cases represented as test suite TS1. TS1 is exercised upon the application under consideration to determine the number of faults covered by prioritized test suite TS1. The test cases now in TS1 are prioritized again to give test suite TSp. TSp gives maximum fault coverage in reduced time of execution by removing those test cases that give redundant fault coverage. Effectiveness of the final prioritized test suite TSp is determined with the help of Average Percentage of Fault Detected Metrics (APFD). Let TS be the test suite having n test cases and let F be the set of m faults identified by TS. Let Fi be first test case in original test suite TSo which identifies fault i. Therefore, APFD for TSi is formulated as shown in Eq. (3).

$$ {\text{APFD}} = 1 - \frac{{F_{1} + F_{2} + F_{3} + \ldots \ldots + F_{m} }}{nm} + \frac{1}{2n} $$
(3)

6 Implementation

FPA for TCP is implemented in Java using Eclipse IDE. Three applications namely, Puzzle Game [16], Area and Perimeter [18] and Tritype [17] are used, which were written in Java programming language. Area and Perimeter calculates the area and perimeter of various mathematical shapes and tritype is a classic triangle classification problem. TestNG tools is used to design the test scripts for all the applications [19]. EclEmma tool is used to get the code coverage for applications used [20]. For the validation of results five versions of Puzzle Game, eight versions of Area and Perimeter and six versions of Tritype applications were created. Each version was seeded with a unique fault. Details of the applications used for experimental evaluation are shown in Table 1.

Table 1 Details of applications used
Full size table

7 Results

For the explanation of empirical evaluation only one application is considered namely, Puzzle Game. The original test suite (TSo) for Puzzle Game contains 33 test cases. Test cases are designed in no order viz., {T1, T2, T3, T4, …, T25, T26, …, T33}. There are five versions of Puzzle Game application with unique faults induced in each version viz., {F1, F2, …, F5}. TSo for Puzzle Game is given as input to the FPA for TCP, the following sequence of test cases is given as the output from FPA i.e., TS1 = {T3, T5, T8, T14, T25, T27, T30}. So, the output of FPA is a test suite TS1 consisting of seven test cases. TS1 was exercised on all the versions of Puzzle Game application to identify the faults covered by TS1. Table 2 shows the faults identified by TS1 for Puzzle Game.

Table 2 Faults identified by TS1 for puzzle game
Full size table

In Table 2, circled cells represent the test case that first identifies a given fault, which is also used to compute APFD metrics. It is evident from Table 2, that test case T3 identifies maximum faults i.e., F3, F4, F5. Hence, only T3 test case is enough to identify faults F3, F4 and F5. Test case T3 is followed by T5 and T8 that identifies faults F2 and F1 respectively. Therefore, the prioritized sequence in final test suite i.e., TSp = {T3, T5, T8} respectively. Table 3 shows the final test suite TSp for Puzzle Game with the faults identified by TSp.

Table 3 Faults identified by TSp for puzzle game
Full size table

TSp gives complete fault coverage as was given by TSo. APFD metric value for TS1 = {T3, T5, T8, T14, T25, T27, T30} for Puzzle Game application using FPA is shown in Eq. (4).

$$ {\text{APFD}} = 1 - \frac{8 + 5 + 3 + 3 + 3}{33\, \times \,5}\, + \,\frac{1}{2\, \times \,33} $$
(4)
$$ \begin{aligned} & {\text{APFD}} = 1- 0. 1 3 3 3 3+ 0.0 1 5 1 5\\ & {\text{APFD}} = 1- 0. 1 4 8 4 8\\ & {\text{APFD}} = 0. 8 5 1 5 2\,\,{\text{in}}\,\,0. 4 8\,\,{\text{s}} \\ \end{aligned} $$

Figure 3 shows the priority order and time of execution of TS1. As it is evident from the Fig. 4 that all the test case for TS1 failed. Hence, TS1 was efficient in covering all the faults. Time of execution of TS1 is 0.48 s. Similarly, for the Prioritized final test suite TSp = {T3, T5, T8} its APFD metric value for Puzzle Game application is shown is Eq. (5).

Fig. 3
figure 3

Order and time of execution of TS1

Full size image
Fig. 4
figure 4

Order and time of execution of TSp

Full size image
$$ {\text{APFD}} = 1 - \frac{8 + 5 + 3 + 3 + 3}{33 \times 5} + \frac{1}{2 \times 33} $$
(5)
$$ \begin{aligned} & {\text{APFD}} = 1 - 0.13333 + 0.01515 \\ & {\text{APFD}} = 1 - 0.14848 \\ & {\text{APFD}} = 0.85152\,{\text{in}}\,0.286\,\,{\text{s}} \\ \end{aligned} $$

Figure 4 shows the priority order and time of execution for TSp. Time of execution of TSp is 0.268 s which is comparatively lower than the execution time of TS1.

For the validation of the results APFD metric is computer for different ordering of test cases viz., FPA order (TS1), FPA order (TSp), Reverse FPA Order of (TSp), Random Order of (TSo) and Reverse Random Order of (TSo) for Puzzle Game application. It is also evident from Fig. 4 that the Prioritized Test Suite (TSp) covers all faults with reduced test suite size and reduced time of execution as compared to other ordering.

In Fig. 5 the random order of execution for original Test Suite (TSo) in 0.94 s is shown.

Fig. 5
figure 5

Order and time of execution of random ordering of Tso

Full size image

In Table 4 APFD results and time of execution of all the application for different ordering is shown viz., FPA order (TS1), FPA order (TSp), Reverse FPA Order of (TSp), Random Order of (TSo) and Reverse Random Order of (TSo). It is evident from Table 4 that the APFD results for FPA order (TSp) and FPA order (TS1) are better as compared to other ordering for all the three applications used in this study.

Table 4 APFD results and time of execution for various ordering of test suites in all the applications
Full size table

In Fig. 6 APFD results and time of execution of all the application for FPA order (TS1), FPA order (TSp), Reverse FPA Order of (TSp), Random Order of (TSo) and Reverse Random Order of (TSo) is graphically represented.

Fig. 6
figure 6

APFD results for various ordering of test suites for all the applications

Full size image

8 Threat to Validity

The threat to validity for this study depends on various factors. Few aspects are that, FPA follows a random search mechanism which may result in an inefficient test suite giving minimal fault coverage and hence may not guarantee efficient results for TCP. In the industrial environment the size of applications and number of actual faults can vary from large to very large including the variation in the operating environment, whereas in this study small size application with few seeded faults have been used. The value of control variables has been manipulated to suite the need of this study, which may have otherwise led to inefficient results.

9 Conclusion and Future Work

In this research, FPA is used for TCP in RT. FPA uses code coverage of test cases without using any information of faults covered by the test cases. FPA is used to prioritize/order the test cases for RT to give maximum fault coverage in minimum time of execution. Three applications written in Java are used for empirical evaluation. Test scripts for all the applications were designed using TestNG tool. For all the applications three test suite ordering were considered namely, original test suite (TSo) and two orderings generated by FPA (TS1) and (TSp). FPA for TCP initially generated prioritized test suite (TS1) which was further reduced and prioritized (ordered) to get test Suite (TSp). For validation of the results APFD metrics is used. APFD value for different orderings of test cases in three applications is calculated. The results of APFD metrics for FPA order (TS1) and FPA order (TSp) outperform the results given by Reverse FPA Order of (TSp), Random Order of (TSo) and Reverse Random Order of (TSo). Since FPA for TCP converged early for all the applications used in this study, hence, only one iteration is used. Therefore, it can be stated that stopping criteria of FPA for TCP depends on the type and size of the application being used. For future validation of results an empirical evaluation of FPA for TCP will be performed with different NIMOA.