Introduction

Value of information analysis provides a framework for analysing uncertainty within economic analysis, by focussing on the value of reducing uncertainty through further information [1]. Such analysis adopts a Bayesian approach to sensitivity analysis [2, 3]. Within the health economics literature, there has been much focus on the estimation of EVPI (e.g., 2–4). EVPI is a measure of the reduction in opportunity loss associated with obtaining perfect information (no uncertainty) on a parameter and can be seen as a measure of decision sensitivity.

EVPI can be calculated for all parameters within a model (global EVPI). Alternatively, EVPI can be calculated for a partial set of input parameters (X i or X p ). This is termed the expected value of partial perfect information (EVPPI or partial EVPI). Parameters for which the decision over optimal treatment is sensitive will have higher EVPPI, although for all parameters EVPPI will vary substantially by a decision maker’s willingness to pay for an additional unit of health benefit (λ).

In this paper, alternate methods for estimating EVPPI are described in detail in terms of the individual steps required to obtain estimates. These methods are then applied to input parameters from a case study to demonstrate how and when these methods can give similar results.

Methods

Notation

This paper adopts standard notation relating to treatment options, costs, effects, cost effectiveness and parameters.

T is the set of alternative treatment options with an individual treatment option represented by t j . Thus, we wish to determine which treatment option is optimal.

E t1 is defined as the expected value of health benefits (e.g., QALYs) from treatment t 1 and C t1 as the expected value of costs. The net monetary benefit (NMB) for t 1 is defined as:

$$ {\text{NB}}_{{t_{1} }} = \lambda E_{{t_{1} }} - C_{{t_{1} }} $$

where λ = a decision makers’s maximum willingness to pay for a unit of health benefit.

The incremental net benefit (INB) when comparing two treatment options (t 1 and t 2) is defined as:

$$ {\text{INB}}_{{t_{1} t_{2} }} = \lambda {\left( {E_{{t_{1} }} - E_{{t_{2} }} } \right)} - {\left( {C_{{t_{1} }} - C_{{t_{2} }} } \right)} $$

The treatment with the greatest net benefit (NB) can be considered the optimal treatment (t*).

Let X, represent the set of k data parameters (X 1,…., X k ) used to estimate the cost and effects of the alternative treatment options. X p is a subgroup of parameters within X, whilst X i represents an individual parameter. X c i and X c p denote the complement sets of input parameters, i.e., all members of X other than X i or X p.

Expected value of perfect partial information

The expected value of perfect partial information (EVPPI) for an individual parameter X i is defined as:

$$ {\text{EVPPI}}_{{{\text{X}}_{{\text{i}}} }} {\text{ }} = {\text{ }}E_{{X_{i} }} {\left[ {{\text{max}}_{t} E_{{X|X_{{i{\text{ }}}} }} {\left( {{\text{NB}}_{t} |{\text{X}}_{t} } \right)}} \right]}{\text{ }} - {\text{ NB}}_{{{t^{*}}{\text{ }}}} $$

EVPPI for a subgroup of parameters X p is defined as:

$$ {\text{EVPPI}}_{{{\text{X}}_{{\text{p}}} }} {\text{ = }}E_{{{\text{X}}_{p} }} {\left[ {\max _{{t{\text{ }}}} E_{{X|{\text{X}}_{{{\text{p }}}} }} {\left( {{\text{NB}}_{t} |{\text{X}}_{{p{\text{ }}}} } \right)}} \right]}{\text{ }} - {\text{ NB}}_{{{t^{*}}{\text{ }}}} $$

EVPPI cannot be solved in a closed form. Thus, all methods of estimating EVPPI require integration using either Monte Carlo simulation or quadrature: numerical methods for estimating the area under the curve for functions that cannot be solved through integration. In this paper, five different proposed methods of estimating EVPPI are outlined.

The first two methods described are appropriate only in specific circumstances relating to the characteristics of the probability density functions of input parameters and their relationship with INB. In many instances, the requirements for these methods are not met and hence the methods are inappropriate for calculating EVPPIs for all input parameters. This is especially the case for Markov models. Hence, it is necessary to adopt more complex methods that can be applied in the general case. Three such methods are described. Two of these methods are based on the mathematical definition of EVPI and involve solving double integrals, neither of which is in closed form. The inner or nested integration involves estimating the incremental net benefit with different fixed values of X i . The outer integration then determines EVPPI through integration across the probability density functions for X i . An alternate method has been suggested that involves avoidance of the second integral by assuming X i is constrained to its expected value. This method is not based on the mathematical definition of EVPPI.

Methods of estimating EVPPI

Unit normal loss integral method (UNLI)

EVPI can be defined alternately as the integral of the incremental net benefit function (INB, defined as the net benefit of the optimum treatment less the net benefit of the alternative) for INB < 0 with respect to its density function:

$$ {\text{EVPI}} = {\int\limits_{ - \varpi }^0 {f({\text{INB}}){\text{INB}}\;{\text{dINB}}} } $$

If the uncertainty around INB is normally distributed (INB∼N (μINB, \( \sigma ^{2}_{{{\text{INB}}}} \))), we can evaluate the EVPI by exploiting the following result:

If Y~N (μ, σ2):

$$ {\int\limits_{ - \varpi }^0 {y\frac{1} {{{\sqrt {2\pi \sigma ^{2} } }}}{\text{exp}}{\left\{ { - \frac{1} {{2\sigma ^{2} }}{\left( {y - \mu } \right)}^{2} } \right\}}{\text{d}}y = \mu \Phi {\left( {\frac{{ - \mu }} {\sigma }} \right)} - \frac{\sigma } {{{\sqrt {2\pi } }}}{\text{exp}}{\left( { - \frac{{\mu ^{2} }} {{2\sigma ^{2} }}} \right)}} } $$

where \({\Phi}\) is the cumulative density function of a standard normal random variable.

Hence we have

$$ {\text{EVPI}} = \mu _{{{\text{INB}}}} \Phi {\left( {\frac{{ - \mu _{{{\text{INB}}}} }} {{\sigma _{{{\text{INB}}}} }}} \right)} - \frac{{\sigma _{{{\text{INB}}}} }} {{{\sqrt {2\pi } }}}{\text{exp}}{\left( { - \frac{{\mu _{{{\text{INB}}}} ^{2} }} {{2\sigma ^{2}_{{\,{\text{INB}}}} }}} \right)} $$

Thus, if INB is normally distributed, global EVPI can be solved from the above [4, 5].

A recent review of the use of modeling in research prioritization found no examples of where the EVPPI has been estimated using the unit normal loss integral method [5]. Recent work in estimating EVPPIs has ignored this method, primarily due to the focus on situations where the distribution of INB is non-normal. However, in situations when the global EVPI cannot be estimated through the unit normal loss integral method (UNLI), it may still be an appropriate method for calculating EVPPIs for parameter(s).

INB may be non-normally distributed and the relationship between INB and some parameters may be non-linear. However, consider a normally distributed parameter X i that has a linear relationship with INB. The relationship between the parameter and the expected INB can be expressed as:

$$ E{\left( {{\text{INB}}\left| {X_{i} } \right.} \right)}{\text{ }} = {\text{ }}\alpha + \beta X_{i} $$

Writing \( {\text{Z}}_{i} = {\text{ }}\alpha + \beta X_{i} \) the EVPPI for X i can now be defined as:

$$ {\text{EVPPI}}_{{X_{i} }} = {\int\limits_{ - \varpi }^0 {zf_{{Z_{i} }} (z)\;{\text{d}}z} } $$

with f zi (z) the density function of Z i .

Since Z i ~N (μ i , \( \sigma ^{2}_{i} \)) with u i = α + βE(X i ) and \( \sigma ^{2}_{i} \) = β2var(X i ), it follows that

$$ {\text{EVPPI}}_{{X_{i} }} = \mu _{i} \Phi {\left( {\frac{{ - \mu _{i} }} {{\sigma _{i} }}} \right)} - \frac{{\sigma _{i} }} {{{\sqrt {2\pi } }}}{\text{exp}}{\left( { - \frac{{\mu ^{2}_{i} }} {{2\sigma ^{2}_{i} }}} \right)} $$

Thus, EVPPI for X i can be evaluated from the above equation as follows.

  1. 1.

    First we require knowledge of α and β. These can be estimated through two Monte Carlo simulations by holding X i constant at two values at the extreme ends of its range (x l and x h ).

    1. (a)

      Hold X i constant at x l and estimate E(INB|X i  = x l ) using a MCS with all other parameters (\( {\mathbf{X}}^{c}_{i} \)) random.

    2. (b)

      Hold X i constant at x h and estimate E(INB|X i  = x h ) using a MCS with \( {\mathbf{X}}^{c}_{i} \) random using the same random seed as previously.

    3. (c)

      Now, by definition:

      $$ \begin{aligned}{} & \beta {\text{ }} = \frac{{E{\left[ {{\text{INB}}\left| {{\left( {X_{i} = {\text{ }}x_{h} } \right)}} \right.} \right]} - E{\left[ {{\text{INB}}\left| {{\left( {X_{i} = {\text{ }}x_{l} } \right)}} \right.} \right]}}} {{x_{h} - x_{l} }} \\ & \alpha {\text{ }} = E{\left[ {{\text{INB}}\left| {{\left( {X_{i} = {\text{ }}x_{l} } \right)}} \right.} \right]} - \beta x_{l} \\ \end{aligned} $$
  2. 2.

    Now we estimate the following: u i  = α + βE(X i ) and \( \sigma ^{2}_{i} \) = β2var(X i )

  3. 3.

    From this we estimate EVPPI

    $$ {\text{EVPPI}}_{{{\text{X}}_{i} }} = \mu _{i} \Phi {\left( {\frac{{ - \mu _{i} }} {{\sigma _{i} }}} \right)} - \frac{{\sigma _{i} }} {{{\sqrt {2\pi } }}}{\text{exp}}{\left( { - \frac{{\mu ^{2}_{i} }} {{2\sigma ^{2}_{i} }}} \right)} $$

One can also derive EVPPIs for a sub-set of parameters that have the same desired properties using the above approach. Consider the situation where X p represents a sub-set of X of size j in which all parameters are normally distributed and have a linear relationship with INB. Thus,

$$ E{\left( {{\text{INB}}\left| {{\text{X}}_{{\text{p}}} } \right.} \right)}{\text{ }} = {\text{ }}\alpha + {\sum\limits_{i = 1,...,j} {\beta _{i} X_{i} } } = Z_{p} $$

Z p ∼ N (μ p , σ 2 p ) with \( u_{p} = \alpha + {\sum\limits_{i = 1,...,j} {\beta _{i} E{\left( {X_{i} } \right)}} } \) and \( \sigma ^{2}_{p} = {\sum\limits_{i = 1,...,j} {\beta ^{2} \text{var} {\left( {X_{i} } \right)}} }, \) it follows that

$$ {\text{EVPPI}}_{{X_{p} }} = \mu _{p} \Phi {\left( {\frac{{ - \mu _{p} }} {{\sigma _{p} }}} \right)} - \frac{{\sigma _{p} }} {{{\sqrt {2\pi } }}}\exp {\left( { - \frac{{\mu ^{2}_{p} }} {{2\sigma ^{2}_{p} }}} \right)} $$

UNLI is therefore an appropriate measure of EVPPI for variables that are normally distributed and have a linear relationship with INB. UNLI may also work well as an approximation in certain situations, specifically when the relationship between the parameter of interest and incremental net benefit is linear and it is approximately normal. This is of major importance given that whenever a distribution for a continuous parameter is obtained from a moderately large sample size, it can often be well approximated by a normal distribution.

Single MCS method

Felli and Hazen have shown that, if INB is multi-linear in \( {\mathbf{X}}^{c}_{i} \) (or \( {\mathbf{X}}^{c}_{p} \)) (i.e., all parameters within the complement set have a linear relationship with INB), EVPPI for X i (or X p ) can be estimated as follows (2,3):

  1. 1.

    Generate one random value \( {\mathbf{X}}^{{(j)}}_{p} \) from the joint distribution of X p

  2. 2.

    Calculate the net benefit for each treatment option t using parameter values X p  = \( {\mathbf{X}}^{{(j)}}_{p} \) and \( {\text{X}}^{c}_{p} \) = E(\( {\text{X}}^{c}_{p} \)|X p  = \( {\mathbf{X}}^{{(j)}}_{p} \)). This will give the value of E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \))

  3. 3.

    Obtain the maximum of the net benefits calculated in step 2, max t E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \))

  4. 4.

    Repeat steps 1 to 3 J times. EVPPI is estimated by \( \sum ^{J}_{{j = 1}} \) = 1 max t E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \))/J−NB t*

The single MCS method may also work well when the relationship between all parameters and INB is approximately linear.

Two-stage MCS method

Several authors have suggested a method of calculating EVPPIs that involves solving both the inner and outer integration through a two-stage Monte Carlo simulation (e.g., [5–7]). This differs from the single MCS method in that a second MCS is used in step 2 to estimate E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \)) It is conducted as follows:

  1. 1.

    Generate one random \( {\mathbf{X}}^{{(j)}}_{p} \) from the joint distribution of X p

  2. 2.

    Conduct a MCS, by repeatedly sampling from the conditional distribution of \( {\mathbf{X}}^{c}_{p} \)|X p  = \( {\mathbf{X}}^{{(j)}}_{p} \), to estimate E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \)) for each treatment option t, keeping X p fixed at \( {\mathbf{X}}^{{(j)}}_{p} \)

  3. 3.

    Obtain the maximum of the net benefits calculated in step 2, max t E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \))

  4. 4.

    Repeat steps 1–3 J times. EVPPI is the estimated by \( \sum ^{J}_{{j\; = \;1}} \) max t E(NB t |X p  = \( {\mathbf{X}}^{{(j)}}_{p} \))/J − NB t*

Quadrature method

A second method rooted in the mathematical definition of EVPPI has been suggested that requires fewer repeat simulations than the two-stage MCS method [8–9]. Instead of a two-stage Monte Carlo simulation, the outer integration across the probability density functions of X i can be achieved through numeric quadrature. Estimating the EVPPI of X i would require the following approach:

  1. 1.

    A set of values X (1) i ,…, X (N) i is determined for the parameter of interest. The values should be equally spaced across the individual’s parameters probability function with a high degree of coverage.

  2. 2.

    For \( X^{{(j)}}_{i} \) conduct a MCS by repeatedly sampling from the conditional distribution of \( {\mathbf{X}}^{c}_{i} \)|X i  = \( {\text{X}}^{{(j)}}_{i} \), to estimate E(NB t |X i  = \( {\text{X}}^{{(j)}}_{i} \)) for each treatment option t, keeping \( {\text{X}}^{{(j)}}_{i} \) fixed at X i

  3. 3.

    Obtain the maximum of the net benefits calculated in step 2, max t E E(NB t |X i = \( {\text{X}}^{{(j)}}_{i} \))

  4. 4.

    EVPPI is estimated by \( \sum ^{N}_{{j\; = \;1}} \) max t E(NB t |X p = \( {\mathbf{X}}^{{(j)}}_{{\text{p}}} \)) f(\( {\text{X}}^{{(j)}}_{i} \)) w j  − NB t* The weights w j are obtained using a quadrature method such as Simpson’s rule.

(In Simpson’s rule we have w j  = s j (\( {\text{X}}^{{{\text{(2)}}}}_{i} \) − \( {\text{X}}^{{{\text{(1)}}}}_{i} \))/3, where s j  = 4 when j is even and s j  = 2 when j is odd, with the exceptions of s 1 = s N  = 1. We must also have N odd).

Note that in step 1 the greater the number of values chosen and the higher the degree of coverage are, the more precise the estimate of EVPPI. In the following sections 101 different values of each parameter are used and values cover at least 99.99% of the probability density function. For single parameters X i this can be preferable to the two-stage MCS approach, as J typically needs to be larger than N. In principle, the same approach can be used to estimate the EVPPI for multiple parameters X p , but is less desirable as quadrature becomes unwieldy in high dimensions.

Difference method

In an evaluation of treatments for Alzheimer's disease, an alternative formulation for estimating EVPPI was adopted with the EVPPI for X i estimated by the difference between global EVPI given uncertainty in all parameters and global EVPI when X i is fixed [10]. Similar methods have been used in an evaluation of management strategies for urinary tract infection [11]. The method defines EVPPI as follows:

$$ {\text{EVPPI}}_{{{\text{X}}_{i} }} = {\text{EVPI}} - E_{{{\text{X}}\left| {{\text{X}}_{i} } \right.}} {\left[ {{\text{max}}_{t} {\left\{ {{\text{NB}}_{t} |{\text{X}}_{i} = E{\left( {{\text{X}}_{i} } \right)}} \right\}}} \right]} - {\text{max}}_{{t }} {\left[ {E_{{{\text{X}}\left| {{\text{X}}_{i} } \right.}} {\left\{ {{\text{NB}}_{t} |{\text{X}}_{i} = E{\left( {{\text{X}}_{i} } \right)}} \right\}}} \right]} $$

Thus, the approach involves the following

  1. 1.

    Estimate global EVPI as described in Sect. 4.5.2.

  2. 2.

    Estimate the NB for all treatment options by conducting MCS by keeping the parameters of interest (X i ) fixed at their expected values and by sampling from the probability density functions of all other parameters (\( {\mathbf{X}}^{c}_{i} \)).

  3. 3.

    Calculate the expected value over all replications of the net benefit of the optimum therapy subtracted from the maximum net benefit over all therapeutic options.

  4. 4.

    EVPPI is the difference in the values obtained in step 1 and step 3.

Case study

The case study is an evaluation of entacapone for the treatment of advanced Parkinson’s disease [12]. Analysis was conducted within a Markov model that assumed three distinct states of Parkinson’s disease severity: mild/moderate disease, severe disease and death.

All patients were assumed to be in the severe state at onset of treatment (Fig. 1). For patients receiving entacapone, transition probabilities were required for improvement from severe to mild/moderate disease, progression from mild/moderate to severe disease and death. For patients receiving usual care only progression from severe disease to death was required

Fig. 1
figure 1

Design of Markov model for evaluation of entacapone

Full size image

Analysis compared usual practice with and without the inclusion of entacapone. Usual therapy was assumed to include levodopa used in combination with other anti-Parkinsonian medication. The model was based on a 6-month cycle. A 5-year time horizon was chosen, which is relevant for a chronic disease such as Parkinson’s disease. All outcomes were discounted at 5%. Analysis was taken from the perspective of the health care system.

Input parameters and their associated probability distributions are detailed in Table 1. Drug costs were assumed fixed. The probability of mortality during each cycle was obtained from national population data and was also assumed fixed.

Table 1 Probability density functions for input parameters
Full size table

Analysis

Analysis focuses on estimating the EVPPI for each parameter within the decision analysis. The difference, quadrature and two-stage MCS methods are used to estimate EVPPI for all parameters within the model.

The UNLI method is appropriate for estimating EVPPI for cost and utility parameters as they are assumed normal and are linear in INB. The single MCS method is an appropriate method for estimating EVPPI for PPROGRESS, as INB is linear in all parameters except PPROGRESS.

The UNLI and single MCS methods are used as approximations for EVPPI for specific parameters as detailed above.

The single MCS method is appropriate for parameters where INB is multilinear in the complement set. Thus, in the case study it would only be appropriate for PPROGRESS as the relationship between PPROGRESS and INB is not linear. The degree of this non-linearity was assessed by regression analysis whereby the INB was estimated for a range of values of PPROGRESS by conducting MCS with the value of PPROGRESS assumed fixed at various values. Analysis found that although the relationship was non-linear the estimated linear function was associated with a high R 2 (0.985) (Fig. 2).

Fig. 2
figure 2

Relationship between the probability of disease progression and incremental net benefit

Full size image

The UNLI method is appropriate only for parameters that are normally distributed and are linear in INB. The variable PIMPROVE is linear in INB, but is not normal. However, the UNLI method may give a close estimate of EVPPI for PIMPROVE as the variable is well approximated by normals of the same mean and variance (Fig. 3).

Fig. 3
figure 3

Comparison of beta distribution for PIMPROVE and the approximate normal distribution

Full size image

The number of MCS conducted will affect the accuracy of the predicted EVPPI due to the associated Monte Carlo error. For the base analysis all Monte Carlo simulations involved 5,000 replications. To assess the accuracy of each method with respect to Monte Carlo error, analysis was repeated for a subset of parameters (utility and transition probabilities) using an extreme number of replications (5 million).

Results

Table 2 compares the estimates of EVPPI for each parameter and set of parameters based on the alternative methods. The difference method gave substantially different values from the other methods and can be dismissed as a true measure of EVPPI. The other four methods gave broadly similar values for EVPPI for most parameters. PPROGRESS had the highest EVPPI, followed by utility parameters and PIMPROVE. Cost parameters had little information value.

Table 2 Estimates of EVPPI based on alternative formulations
Full size table

Table 2 also provides evidence of the appropriateness of both the single MCS and UNLI methods as proxy methods for estimating EVPPI. It is shown that under the scenarios arising within this model, these methods do give close approximations to EVPPI.

Table 3 compares the estimates of EVPPI from the four methods based on 5,000 and 5 million replications. The values obtained from using 5 million replications differ modestly from analysis based on 5,000 replications suggesting that, in this instance, a MCS based on 5,000 replications may be sufficient. The results from the UNLI, quadrature and two-stage MCS are very similar, confirming that each method is estimating the same variable. Thus, the difference between methods can simply be put down to error with respect to integral measurement.

Table 3 Estimates of EVPPI based on alternative formulations and number of replications
Full size table

Conclusions

The estimation of EVPPI for parameters and subsets of parameters is an essential component in the analysis to identify the value of obtaining further information given decision-making under uncertainty. In addition, EVPPI has been argued to be a theoretically correct measure of the sensitivity of a study’s results [2, 3]. However, to facilitate such usage, EVPPI has to be accurately measured.

In this paper, five alternate methods for estimating EVPPI have been identified, described and applied to a case study. All measures are subject to Monte Carlo error. As the number of replications used to estimate EVPPI increase, the appropriate method for the estimation of EVPPI will converge to the same value.

Within this paper a number of issues relating to the conduct of probabilistic analysis and the estimation of EVPPI are raised. One method for estimating EVPPI, the UNLI method, is only appropriate when a variable is distributed normally. In this study, normal distributions are used to characterise uncertainty with respect to costs and utilities. Clearly if other distributions were used then the UNLI method would be less appropriate, though it may still give approximate results if distributions were well approximated by normals.

The case study used in this paper is simple to aid in demonstrating the methods used and to allow replication of results if desired. This contributes to the relative linear nature of the model, which in turn leads the single MCS method to provide useful approximations of EVPPI. Clearly, in more complex models a higher degree of non-linearity may be observed and this method may not provide as accurate a measure on EVPPI. In addition, in more complex models we may wish to assume some degree of correlation between parameters, which would likely further enhance the non-linear nature of the model.

More complex models will also lead to a requirement to increase the number of replications required to estimate EVPPI. In the case study, 5,000 replications were used to estimate EVPPI as a base case. It is likely in more complex models that a greater number of replications will be required to provide precise estimates of EVPPI. The level of precision can be estimated by the standard error of the estimate of EVPPI, which will decrease as the number of replications increases.

Results are presented as EVPPI per patient. For estimation of the expected value of sample information and the calculation of optimal sample size, population estimates of EVPPI are required, which involves weighting EVPPI by the potential patient population affected by the decision at hand. EVPPI does however provide decision makers with a measure of parameter importance, which can assist to some degree in determining areas for further research and consideration.

Of the methods previously proposed in the literature, the difference method is clearly an inappropriate method for estimating EVPPI. It is not rooted in the mathematical definition of EVPPI. However, it had been argued that if the relationship between a parameter and the outcome of interest is not markedly non-linear then the difference method would be a suitable means to estimate EVPPI [10]. The results from this study dispute this proposition as there is a substantial difference in values obtained from this and the other methods. Thus, this method is demonstrated empirically to be an inappropriate measure of EVPPI.

The single MCS and UNLI methods are computationally efficient methods of estimating EVPPI. However, they are formally appropriate for estimating EVPPI in specific limited circumstances relating to the mathematical relationship between input parameters and INB. However, both methods are shown to work well in certain other situations. The single MCS method is shown to work well when the relationship between all parameters and INB is approximately linear. The UNLI method is shown to work well if X i is approximately normal. However, it is clear that in more complex models these approaches will provide less accurate proxies for EVPPI.

The quadrature and two-stage MCS methods can be considered general methods for estimating partial EVPPI as they can be applied in all circumstances. The methods are comparable. By increasing the number of MCS used, both methods would return similar values converging to the true value of EVPPI. Both methods are, however, computationally complex.

Previous studies have explored the alternate methods for estimating EVPPI (e.g., 13). However, studies have tended to ignore both the quadrature method and the UNLI method as outlined in this paper. Given that these methods are shown here to be of use in many situations, a further examination of methods as conducted in this paper is warranted. In addition, although the difference method has been shown elsewhere to be incorrect [13, 14], it is worth reiterating this point to avoid future use of the inappropriate method.

Other methods for estimating EVPPI may be developed that will have to be evaluated under similar criteria. For instance an approach may be possible that uses a hybrid of the quadrature and UNLI methods if INB was linear in a parameter, though the parameter was strongly non-normal. In addition, Oakley has used a combination of quadrature and non-parametric regression to efficiently learn the relationship between parameters and the expectation of INB [15].

The conclusion reached is that where appropriate EVPPI should be estimated using either the single MCS or UNLI method. When neither of these methods is appropriate, the quadrature or two-stage MCS methods should be used. However, given the computational complexity of these methods, further work examining the appropriateness of using either the single MCS or the UNLI methods as approximations for EVPPI may be worthwhile. The difference method should clearly not be used.