1 Introduction

Information technology is playing a crucial role in today’s fast moving life. Almost everything is more or less dependent on software or driven by software technology. This dependence on software has increased the demand of reliable software in no time. Reliability of software is governed by the amount of testing time and resources (effort) consumed during the software development phase. Software testing is one of the most important phase of software development life cycle as it generally consumes more than 50 % of the total budget. Also release and quality of the software is dependent on the testing phase. Measuring the quality of software is a very difficult job in the software industry. It is important to measure reliability of software before release, also quantifying reliability help software firms in quantifying the behaviour of the system and allocate testing resources. More is the testing greater is the chance of getting a reliable software. But on the other hand to match with the pace of market firms don’t want to spend too much of time on software testing. Software failure is said to be occurred if on giving a specific input we don’t get the desired output, on the other hand software fault is incorrect code, data definition or statement in a program that has affected the program in such a way so that it will not give the desired output Musa (Musa 2004). During the testing phase of software development life cycle, testers learn and improve their skills. This phenomenon of learning is the result of familiarity of the testers with the software program. Testing team finds faults and developers remove them during testing phase but this phase consumes large amount of testing resources i.e. CPU hours, manpower etc. To find out the relationship between faults detected and removed during testing various testing resources (effort) based software reliability growth models (SRGMs) have been proposed under different set of assumptions in the last four decades (Goel and Okumoto 1979; Huang et al. 2007; Kapur et al. 2011; Kuo et al. 2001; Lin and Huang 2008; Musa 2004; Pham 2006; Chatterjee and Singh 2014). Yamada et al. (1986) first introduced effort based SRGM. Huang and Kuo (2002) developed a SRGM by incorporating logistic testing effort function. Kapur et al. proposed (Kapur et al. 2007, 2008) proposed a SRGM with testing effort based learning process. They further extended their concept to propose a flexible SRGM with testing effort based learning process. Kapur et al. (2009) proposed a unified framework for modelling testing effort based SRGM. Ahmad et al. (2010) proposed a S- shaped SRGM with testing effort under imperfect debugging phenomenon. Inoue and Yamada (2013) proposed a testing effort based lognormal SRGM. Zhao et al. (2005, 2012) proposed a testing effort based SRGM with imperfect debugging. Zhang et al. (2012) proposed a queuing theory based SRGM incorporating testing effort. They further proposed a two-step fault detection and correction model for testing effort based on queuing approach Zhang (2015) and Zhang et al. (2013). Peng et al. (2014) proposed a two stage detection correction based SRGM with testing effort under imperfect debugging environment. Zhang et al. (2014) proposed a unified framework for modelling SRGM with testing effort under the effect of imperfect debugging. Recently Li et al. (2015) incorporated S-shaped testing-effort functions into software reliability model with imperfect debugging. Software reliability modelling with testing effort is not limited to single version only. Researchers are working on to include testing effort in software reliability growth modelling for multiple versions of a software. The study of software reliability modelling with testing effort for multiple version is proposed by Singh et al. (2012). Recently Kapur et al. (2015) proposed generalized framework for a software up-gradation model with testing effort and two types of imperfect debugging.

Testing is done to remove faults from the software and hence ensuring reliability. But it consumes large amount of resources such as manpower, CPU hours etc. which incurs cost to the developer. On the other hand if faults are not removed in the testing phase, then users will face failures during the operational phase and the cost of debugging faults in the operational phase is much higher than that in the testing phase (Musa 2004; Singh et al. 2015). Prolonged testing increase the software reliability but increases the software development cost and also delays the software release which causes loss in terms of market opportunity. However spending insignificant amount testing time for error removal reduces the software development cost, but increases the risk of higher number of failures in the operational phase. Hence determining the optimal release time is very important from developer’s perspective. Literature available in this field, concentrates mainly on minimizing the total expected cost, satisfying the reliability requirement at the lowest total cost as well as sensitivity analysis of the optimal release time (Huang and Lin 2010; Huang and Lyu 2005; Kapur et al. 2011; Kapur and Garg 1990; Li et al. 2010; Lo et al. 2005; Pasquini et al. 1996; Pham 2006; Zachariah 2015).

Although it is impossible for a testing team to come out with software which is completely free of errors. But in order to ensure that users face least number of errors during operation phase, now software firms continue removing the faults even after release (Apple 2015; HP 2015). During this post release phase software firms update the software by providing patches to users. A patch, sometimes also called a fix, is a small program of software that is used to fix errors that successfully deceived the testing team during pre-release testing phase of software. After software release, there is either a decrease in the failure rate function or it remains constant, depending on whether there is a growth in software reliability during the operational phase or not (see Figs. 1, 2). This phenomenon can be understood as follows. If the removal of detected faults was not possible during the operational phase then the failure rate remains constant. On the other hand if the detected faults during testing phase are removed and similar failures do not occur again, then the software reliability increases. Yang and Xie (2000) emphasize that the latter is more likely in practice where errors encountered are corrected, bundled as a software patch and presented to the user.

Fig. 1
figure 1

Software reliability growth without patching

Full size image
Fig. 2
figure 2

Software reliability growth with patching

Full size image

The topic of security patch management is closely related to general software update management, which has received attention. Cavusoglu and Zhang (2008) studied the security patch release and management problems from the perspective of both vendor and the firm. A game-theoretic model was developed by them to study how the interaction between vendor and firm balances the cost and benefits of patch management. Subsequently, Okamura et al. (2009) and Luo et al. (2015) considered a non-homogeneous bug-discovery process and studied both the periodic and aperiodic patch/update management models. Recently a cost model for finding optimal release policies for security patch management was developed by Dey et al. (2015). Arora et al. (2006, 2008) proposed a cost model in which the trade-off between releasing a buggy software product and investments in patching it later is shown. They also show the benefit of releasing the software early and patching it later. Jiang et al. (2012) also proposed a software scheduling policy where they show the advantage of releasing the software early and to continue testing after the release. The idea was to increase the testing base from a limited number of testers to significantly large number of testers which contributes to customer side testing of the software. Kapur and Shrivastava (2015) proposed a generalized framework to determine the optimal release and testing stop time and showed the benefit of early release and continuing testing even after release.

Practically it is not feasible to release an update every time a fault is reported by the end user. Therefore an important problem for the developer is to determine how and when to release updates and stop testing to remedy faults in its software. Also the existing literature in software release time problems (Kapur et al. 2011; Peng et al. 2014) did not take account the difference between the user’s operational environment and the software testing phase for determining optimal release time of software. However several authors have worked on the reliability modelling in different phases of a software and various reliability assessment methods have been proposed for the operational phase (Huang et al. 2005; Huang and Lin 2010; Jain and Priya 2005; Pasquini et al. 1996). A generalized framework was proposed by Huang et al. (2000) to prove that fault detection rate changes from testing to operational phase. A SRGM was proposed by Zhao et al. (2012) to show that due to environmental changes and inherent fault detection rate there is a change in software reliability from testing to operational phase.

The above literature review shows that academic research on software update/patch management is still at an early stage. However, the current software industry has started implementing the concept of releasing early and updating the software by providing patch release. The biggest advantage of early release is capitalizing market opportunity while post release testing ensures higher software reliability. Although software testing and updating activities are very common in practice, not many attempts can be found on modeling their effects on total cost incurred during the lifecycle of software in an integrated manner. In this paper we have taken the first step towards filling this gap by considering both software testing and updating activities jointly to propose a generalized framework for developing a testing effort based cost model to determine the optimal release and patching time of software to minimize the total expected cost. Rest of the paper is organized as follows. Section 2 deals with model assumptions, notations and formulation of the proposed cost model. Section 3 deals with the modeling of the testing effort based cost function with patching. In Sect. 4, numerical example is provided using a real life data set to validate the proposed work. Finally conclusion has been drawn in Sect. 5.

2 Model formulation

This section deals with model formulation. The applied SRGM and its assumptions are firstly described to provide a basis for the model of software failure process. Then, the software testing procedure and updating policies are described in order to minimize the total expected cost. Section 2.1 describes the notations that are used in modeling the proposed framework.

2.1 Notations

m(W(t)):

Expected number of faults removed in the time interval (0,t]

W(t):

Cumulative testing effort in the interval (0.t]

w(t):

Current testing-effort expenditure rate at testing time t. i.e. \(\frac{d}{dt}W(t) = w(t)\)

b :

Constant

b i :

Fault detection rate in ith phase

a :

Constant, representing the number of faults lying dormant in the software at the beginning of testing

ν, k :

Parameter of Weibull distribution

F(W(t)):

Failure distribution function

Fi(W(t)):

Failure distribution function in ith phase

\(\overline{W}\) :

Amount of testing-effort eventually consumed

W i (t):

Amount of testing effort consumed in ith phase

m(T lc ):

Number of faults removed during the lifecycle of the software

τ :

Release Time of the software

τ i :

ith patch release time

c 1 :

Cost of testing per unit testing effort expenditure

c 2 :

Market opportunity cost

c 3 :

Cost of detection and removal of faults by testers and developers respectively before the release of the software

c 4 :

Cost of removing faults reported by user after software release in pre patching period

c 5 :

Cost of removing faults reported by user after patch release in post patching period

2.2 Assumptions

The proposed model is based upon the following basic assumptions:

  1. 1.

    The process of fault removal follows non-homogeneous Poisson process(NHPP) throughout the software lifecycle.

  2. 2.

    The software system might fail at random owing to the faults lying in it.

  3. 3.

    The expected number of faults detected in the time interval (t, t + Δt) is directly proportional to average number of faults remaining in the software.

  4. 4.

    Fault detection leads to its immediate correction.

  5. 5.

    The fault detection/correction rate with respect to testing effort intensity follows independent and identically distribution function, where

    $$F(W(t)) = \int\limits_{0}^{W(t)} {f(x)dx}$$
  6. 6.

    All faults are removed perfectly.

  7. 7.

    Total numbers of faults lying dormant in the software are finite.

  8. 8.

    Lifecycle of the software is finite.

  9. 9.

    Cost of patching is negligible.

  10. 10.

    Market opportunity cost which is assumed to be monotonically increasing, twice continuously differentiable convex function ofτ. Since the qualitative conclusion of the study is not much affected by the actual functional form of market opportunity cost, therefore we will use the form used by Jiang and Sarkar (Jiang et al. 2012).

2.3 Effort based software reliability growth model

The SRGM developed in this paper considers the time dependent variation in the consumption of testing resources. To elucidate the testing effort in this study, we used Weibull function. The underlying assumption is described as, “The testing effort rate is proportional to the testing resources available”.

$${\text{i}}.{\text{e}}.\frac{dW(t)}{dt} = v(t)\left[ {\overline{W} - W(t)} \right]$$
(1)

where ν(t) is the time dependent rate of consumption of testing resources, with respect to remaining available resources.

If v(t) = v.k · t.k−1, we get Weibull function as:

$$W(t) = \overline{W} \left( {1 - e^{{ - vt\,^{k} }} } \right)$$
(2)
$${\raise0.7ex\hbox{${\frac{dm(W(t))}{dt}}$} \!\mathord{\left/ {\vphantom {{\frac{dm(W(t))}{dt}} {\frac{dW(t)}{dt}}}}\right.\kern-0pt} \!\lower0.7ex\hbox{${\frac{dW(t)}{dt}}$}} = b(W(t))(a - m(W(t)))$$

where \(b(W(t)) = {\raise0.7ex\hbox{${f(W(t))}$} \!\mathord{\left/ {\vphantom {{f(W(t))} {(1 - F(W(t))}}}\right.\kern-0pt} \!\lower0.7ex\hbox{${(1 - F(W(t))}$}}\) is the failure detection (fault removal) rate.On solving the above equation under the initial conditions of m(t = 0) = 0 and W(t = 0) = 0 we get

$$m(W(t)) = aF(W(t))$$
(3)

Here F(W(t)) is the testing effort dependent probability distribution function for fault correction times. It can be noted that F(W(t)) so defined satisfy all the properties of probability distribution functions.

  1. 1.

    In this paper, we have used Weibull type testing effort function which satisfies the property that, at

    $$t = 0,\,W(t) = 0\;{\text{and}}\;F(W(t)) = 0.$$
  2. 2.

    For \(t > 0,\,W(t) > 0\;{\text{and}}\;F(W(t)) > 0.\)

  3. 3.

    As t increases, W(t) also increases indicating monotonically increasing nature of F(W(t)). Similarly the continuity of F(W(t)) can also be explained.

  4. 4.

    As testing continues for an infinitely large time i.e. \(t \to \infty ,\,W(t) \to \overline{W} ,\) the corresponding value of distribution function F(W(t)) is \(F(\bar{W})\). Here \(\overline{W}\) is very large positive number representing the upper bound on the availability of the amount of testing resources available. Therefore, \(F(\bar{W})\) can be assumed to be of order 1.

3 Effort based cost model for single patching

In this case lifecycle of the software is divided into three phases viz. pre release testing phase [0, τ], post release pre patching phase [τ, τ 1] and operational phase after patch release [τ 1, T lc ] as shown in Fig. 3 below. Also it is to be noted that since we have assumed that fault detection process follows NHPP throughout the software lifecycle, hence it will follow NHPP in each phase.

Fig. 3
figure 3

software cycle with single patching

Full size image

Here \(\tau_{1} = \tau + \tau_{1}^{{\prime }} ,\) where \(\tau_{1}^{{\prime }}\) is the time duration between release and first patch time.

In this section we will discuss the total cost incurred for detecting/removing ‘a’ number of bugs which were found during the software lifecycle. The total cost is sum of testing cost, market opportunity cost and the cost incurred for removing faults in each phase which are described below.

Testing cost Testing cost associated with testing effort expenditure is given by

$$c_{1} W\left( \tau \right)$$
(4)

Market opportunity cost The market opportunity cost refers to the market related cost-or the opportunity loss cost due to late entry into the market.

Market opportunity cost is given by

$$c_{2} \tau^{2}$$
(5)

where τ 2is the functional form of market opportunity cost c 2 as in Jiang et al. (2012).

Phase 1: [0, τ] (Pre-release testing phase)

In this phase testing team works to detect failure/fault, i.e. this is the testing period before the release of the software. The tester detects the faults and notifies about it to the developing team for removal process. The total number of faults removed in this interval is given by

$$m(W(\tau )) = a\,F_{1} (W_{1} (\tau ))$$
(6)

Cost incurred due to detection and removal of the faults in this phase it is given by

$$c_{3} \,m\left( {W\left( \tau \right)} \right)$$
(7)

where F 1(W 1(τ))is the rate by which the faults are removed from the software in the interval [0, τ].

Phase 2: [τ, τ 1] (Post release phase before patching)

This period refers to the post release phase before patching where bugs are reported by the users and fixed by the developing team of the software company. Total number of faults detected in this phase is given by

$$\begin{aligned} m(W(\tau_{1} - \tau )) &= (a - m(W(\tau )))\,F_{2} (W_{2} (\tau_{1} - \tau )) \\ &=\, a(1 - F_{1} (W_{1} (\tau )))\,F_{2} (W_{2} (\tau_{1} - \tau )) \\ \end{aligned}$$
(8)

where a(1−F 1(W 1(τ))) represents the remaining number of faults that are left undetected in pre-release testing phase and  F 2(W 2(τ 1 − τ)) is the fault removal rate in post release phase i.e. in the interval [τ, τ 1].

Cost incurred in this phase is given by

$$c_{4} \,m\left( {W\left( {\tau_{1} - \tau } \right)} \right)$$
(9)

Phase 3: [τ 1, T lc ] (Post patching phase)

This interval refers to the post patching period till the life cycle of the software. Here we have considered a situation where firm signs a bond with the user that they will remove the bugs throughout the software lifecycle. In this interval, users face the failure in the software due to remaining number of faults which remain undetected even after patching period. Total number of faults detected in this phase is given by

$$\begin{aligned} m(W(T_{lc} - \tau_{1} )) =\, & (a - m(W(\tau_{1} )))\,F_{3} (W_{3} (T_{lc} - \tau_{1} )) \\ = &\, a(1 - F_{1} (W_{1} (\tau )))\,(1 - F_{2} (W_{2} (\tau_{1} - \tau )))\,F_{3} (W_{3} (T_{lc} - \tau_{1} )) \\ \end{aligned}$$
(10)

where, τ 1 denotes the optimal patch release time and T lc refers to the life cycle of the software. Here a(1−F 1(W 1(τ)))(1–F2(W 2(τ 1−τ))) represents the remaining number of faults which are left undetected in first and second phase and  F 3(W 3(T lc  − τ 1)) is the fault removal rate in post patching phase i.e. in interval [τ 1, T lc ].

Cost incurred in this phase is given by

$$c_{5} \,m\left( {W\left( {T_{lc} - \tau_{1} } \right)} \right)$$
(11)

It may be noted that we do not release patch to the users during the last interval [τ 1, T lc ], since at some point management needs to release a new version of the software. On combining cost incurred in each phase we get total cost incurred is given by

$$\begin{aligned} C(\tau ,T_{lc} ) &= c_{1} \,W(\tau ) + c_{2} \,\tau^{2} + c_{3} \,m(W(\tau )) \\ & + c_{4} \,m(W(\tau_{1} - \tau )) + c_{5} \,m(W(T_{lc} - \tau_{1} )) \\ \end{aligned}$$
(12)

4 Numerical example

Based on our assumption that all the faults are identically and independently distributed we get that the cumulative fault distribution function follows exponential distribution in each phase given by \(F_{i} (W_{i} (t)) = 1 - e^{{ - b_{i} W_{i} (t)}}\), where i denotes the ith phase of software lifecycle. This analysis can be similarly carried out using any other distribution function. Also the effort function follows Weibull distribution given by \(W(t) = \overline{W} \left( {1 - e^{{ - vt\,^{k} }} } \right)\). For estimation of parameter of the effort function and mean value function, we have used data set provided by Obha (1984). This data set contains 328 number of faults which were removed in 19 weeks by consuming 47 h of CPU time. The parameter estimation values obtained by using statistical package for social sciences (SPSS) software are described below.

Total amount of testing effort eventually consumed (\(\overline{W}\)) = 715.7, scale parameter (v) = 0.003, shape parameter (k) = 1.116, total number of faults (a) = 570.9, fault detection rate (b) = 0.017, software lifecycle (T lc ) = 100. It is important to note here that \(W({T_{lc}})={\overline{W}}\). For numerical illustration purpose we have assumed that the fault detection rate in each phase is equal i.e. b i  = b and testing effort function has equal rate in each phase i.e. W i (t) = W(t). Based on the experience of the testing team we assume cost parameters as follows: Cost of testing per unit testing effort expenditure c 1 = 5, market opportunity cost c 2 = 1, Cost of debugging during testing phase c 3 = 3, Cost of debugging during patching phase c 4 = 8 and cost of debugging a fault after patching (operational phase) c 5 = 8. These values can be changed for different scenarios. In general cost per unit fault during a given period is directly proportional to the ratio of resources consumed to the fault removed in that period. It is to be noted that since support cycle of software is much longer as compared to any other phases of software lifecycle, hence it consumes high amount of resources. Also the number of faults removed during this period is low, as maximum number of faults is debugged during testing and patching phase. Hence cost per unit fault removal during post release period is highest. Based on the similar arguments magnitude of cost per unit fault removal in different phases can be described. Now based on the above assumptions we will describe the phase wise fault detection with the associated cost.

Phase 1: [0, τ]

This is the testing period before the release of the software. Total number of faults removed in pre-release testing phase [0, τ] are given by

$$m\left( {W\left( \tau \right)} \right) = a\,F_{1} \left( {W_{1} \left( \tau \right)} \right) = a\left( {1 - e^{ - bW(\tau )} } \right)$$
(13)

where F 1(W 1(τ)) is the fault removal rate in pre-release testing phase.

Therefore cost of removing m(W(τ))number of faults is given by

$$c_{3} \,m(W(\tau )) = c_{3} \,a(1 - e^{ - bW(\tau )} )$$

Phase 2: [τ, τ 1] (Post release phase before patching)

This period refers to the post release phase before patching where bugs are reported by the users and fixed by the developing team of the software company. Total number of faults detected in this phase is given by

$$\begin{aligned} m(W(\tau_{1} - \tau )) &= a\,(1 - F_{1} (W_{1} (\tau )))\,F_{2} (W_{2} (\tau_{1} - \tau )) \\ &= a\,e^{ - bW(\tau )} (1 - e^{{ - bW(\tau_{1} - \tau )}} ) \\ \end{aligned}$$
(14)

where F 2(W 2(τ 1 − τ)) is the fault removal rate in post release phase.

Cost incurred in this phase is given by

$$c_{4} \,m(W(\tau_{1} - \tau )) = c_{4} \,ae^{ - bW(\tau )} (1 - e^{{ - bW(\tau_{1} - \tau )}} )$$

Phase 3: [τ 1, T lc ]

This interval refers to the post patching period till the life time of the software.Number of faults removed in [τ 1, T lc ] is given by

$$\begin{aligned} m(W(T_{lc} - \tau_{1} )) = & a(1 - F_{1} (W_{1} (\tau )))\,(1 - F_{2} (W_{2} (\tau_{1} - \tau )))\times F_{3} (W_{3} (T_{lc} - \tau_{1} )) \\ = & a\,e^{ - bW(\tau )} e^{{ - bW(\tau_{1} - \tau )}} (1 - e^{{ - bW(T_{lc} - \tau_{1} )}} ) \\ \end{aligned}$$
(15)

where, τ 1 denotes the optimal patch release time and T lc refers to the life cycle of the software. F 3(W 3(T lc  − τ 1)) is the fault removal rate in post patching phase i.e. in interval [τ 1, T lc ].

Cost incurred in this phase is given by

$$c_{5} \,m(W(T_{lc} - \tau_{1} )) = c_{5} \,ae^{ - bW(\tau )} e^{{ - bW(\tau_{1} - \tau )}} (1 - e^{{ - bW(T_{lc} - \tau_{1} )}} )$$

On combining cost associated in each phase with testing and opportunity costs, we have the total cost function given by

$$\begin{aligned} {\text{Total}}\,{\text{cost}} = C(\tau ,T_{lc} ) = & c_{1} \,W(\tau ) + c_{2} \,\tau^{2} + c_{3} \,a(1 - e^{ - bW(\tau )} ) \\ & + c_{4} \,ae^{ - bW(\tau )} (1 - e^{{ - bW(\tau_{1} - \tau )}} ) \\ & + c_{5} \,ae^{ - bW(\tau )} e^{{ - bW(\tau_{1} - \tau )}} (1 - e^{{ - bW(T_{lc} - \tau_{1} )}} ) \\ \end{aligned}$$
(16)

Using the cost parameter values defined above for each phase in Eq. (16) we get total cost function which is a function of two variables i.e. τ and τ 1. On optimizing the total cost function by using MAPLE software we obtain the optimal cost incurred is 3411.675, release time (τ) = 19.9 weeks and \(\tau_{1}^{{\prime }} = 3.14\) hence, optimal patch time (\(\tau_{1} = \tau + \tau_{1}^{{\prime }}\)) = 23.04 weeks. The graph for optimal release and patching time is given below in Fig. 4. According to the testing data set (19 weeks) used for numerical illustration and optimal results we can infer that firm should test for one more week before release of the software and release the first patch after 3 weeks. In Table 1 we have summarized the number of faults removed and effort consumed in each phase by using the above mentioned values.

Table 1 shows that out of 570 faults 359 faults are removed by developing team in pre release phase which is of 19.9 weeks. Out of the remaining number (570−359 = 211) faults, 30 are removed during the post release before patching phase which is of 3 weeks. We can say that the first patch is released for 30 faults. The remaining number (211−30 = 181) of faults are removed during post patching period, this phase is the longest of all as it extends up to software lifecycle. In the current work we have discussed the cost model for single patching but on the similar lines of the proposed model we can extend it for ‘n’ number of patches. Faults remaining after the release of first patch can be removed by releasing more patches.

Fig. 4
figure 4

Graph for optimal release and patching time

Full size image
Table 1 Phase wise faults removed and effort consumed
Full size table

5 Conclusion

Decisions related to software release and testing are some of the most important aspects of policy making in software industry. Generally firms release their software after spending significant amount of time on testing so as to ensure that software has achieved a specified reliability level which in turn reduces the chance of software failure in field. But delay in release of software results in loss of market opportunity. In the current market situation firms are releasing early to gain high market potential and fixing the faults later in the post release phase. Keeping this in mind in our proposed work we formulated a generalized testing effort based cost model under different testing environment to determine the optimal release and patch time of software. Based on the proposed policy, project managers can plan the release and testing time of software. This additional flexibility can help firms significantly to gain an advantageous position in a competitive marketplace. Future work may include other factors like change-point, error generation and warranty in the proposed framework to make it more practical.