FormalPara Key Points for Decision Makers

• Decision makers are increasingly interested in linking reimbursement strategies to the degree of uncertainty in the evidence base.

• Current methods of economic evaluation provide only highly aggregate information on the distribution of risk between decision makers, patients and manufacturers.

• The Net Benefit Probability Map (NBPM) allows decision makers to understand the distribution of risk between decision makers and manufacturers over time.

• The NBPM is a potentially useful tool for supporting decision makers’ assessments of patient access and risk-sharing schemes.

1 Introduction

Reimbursement decision makers and analysts increasingly recognize the importance of considering uncertainty in the estimates of costs and outcomes of therapies, in resource allocation decision making. There is a general consensus that cost-effectiveness acceptability curves (CEACs) [1] are a useful addition to incremental cost-effectiveness ratios (ICERs) in the reports of economic evaluations, with an assessment of decision uncertainty (using CEACs or alternatives) a necessary part of any economic evaluation [2–4]. Healthcare funders also now acknowledge the role of uncertainty in the evidence base for new technologies when explaining why technologies have been refused funding or approved under tightly defined circumstances [5].

Uncertainty in the cost effectiveness of a technology is important to reimbursement decision makers because the greater the uncertainty in the estimate of cost effectiveness, the greater the risk of making the wrong decision; that is, reimbursing a technology that will reduce population health or failing to reimburse a technology that would increase population health. Recognition of this importance is demonstrated by the increased interest in and use of ‘risk-sharing schemes’, to support the introduction of innovative technologies into healthcare systems [6–8]. In many cases, these risk-sharing schemes allow patient access alongside evidence development and further reflect the importance of uncertainty to decision makers [6, 9].

These ‘value of information’ methods consider how additional information can reduce the probability of making an incorrect decision; that is, the cases in which the optimal-seeming treatment under current information would not in fact turn out to be optimal. Across all possible cases of the world, the value attributed to increased information is related to the expected net benefit of the optimal-seeming treatment in the cases where the additional information is, and is not, available. In even those analyses that consider the role of time, for instance, the Expected Value of Sample Information (EVSI) [9] and the Expected Net Present Value of Sample Information (ENPVSI) [10], they produce an aggregated monetary value for this benefit that is then compared with the cost of the research. These analyses are useful in assessing the burden of decision uncertainty; that is, the value of delaying a funding decision to allow further research to report.

This article has a different aim to those dealing with value of information approaches and considers the presentation of information to decision makers rather than the methods used to assign value to information. Specifically, this article argues that the disaggregation of decision uncertainty may provide decision makers with a more granular assessment of the value of alternative reimbursement strategies, including access with evidence development schemes, alternative payment strategies such as patient access schemes and changes in time-sensitive assumptions, such as the choice of discount rate for costs and benefits.

This article considers how decision makers might be provided with more information about the timing of costs and benefits, and the distribution of the decision uncertainty over time. In that sense, it focuses not on the value attributable to additional evidence (as the EVSI and ENPVSI) but the separation of health effects and uncertainty over time. The disaggregation considered uses information that is already produced for existing analyses. We propose a new method for reporting the results of probabilistic cost-effectiveness analyses, the Net Benefit Probability Map (NBPM. The NBPM disaggregates the uncertainty in the net benefit over time and captures when uncertainty is unequally distributed between the short, medium and long term. Using exemplar analyses based around a hypothetical secondary prevention therapy for breast cancer based on a previous Markov model [11], we explore the scope for the additional information provided by the NBPM (threshold set to £30,000 per quality-adjusted life year (QALY)), to inform population healthcare resource allocation decisions.

2 Uncertainty in Cost-Effectiveness Analysis

Current methods for representing uncertainty in probabilistic sensitivity analyses (PSAs) focus on the uncertainty in overall cost effectiveness, as represented by the discounted net benefit [12]. There are two algebraically equivalent formulations for net benefit. In the first formulation, the net monetary benefit (NMB) for each option can be calculated by subtracting the discounted in-period cost from monetized health; obtained by multiplying the health (typically represented as QALYs) in each period by the ceiling ratio (λ), for example, a cost per QALY. In the second formulation, the net health benefit (NHB) is calculated by converting costs into a health metric, rather than converting health into a monetary metric; the two metrics differ only in scale (NMB = λ NHB).

Once we have the net benefit for each option, we can calculate the incremental net benefit (INB) as the difference between options. This figure represents a sum across the total period, or time horizon, of the model. When comparing two alternatives, the more efficient outcome will be the one with a positive INB, measured on either the health or not monetary benefit scale. When comparing multiple alternatives, the most efficient outcome will be that with the largest NHB or NMB.

A related tool is the ICER. Here, the total discounted costs and total discounted benefits are calculated for each option. Pairs of options are then compared by considering the ratio of (total) incremental costs to (total) incremental benefits, the ICER, between them. Identifying which of a pair of options is a more efficient option requires knowledge of both the ICER and the signs of at least one of incremental costs and incremental benefits. To see this, it suffices to note that the ICER comparing any treatment A with any treatment B (as baseline) will be the same as the ICER comparing treatment B with treatment A (as baseline), but the sign of incremental costs/benefits is different in each case. As there is no simple decision rule for cost effectiveness using only ICERs, the net benefit formulation is preferred when considering a large number of different scenarios, for instance, within a PSA.

To represent uncertainty within a PSA, CEACs present the likelihood of each of the options considered having the largest net benefit at a given ceiling ratio. The closely related cost-effectiveness frontier (CEF) is often overlaid upon CEAC diagrams and identifies which option has the highest expected net benefit at each ceiling ratio [3].

Whilst ICERs, CEACs and CEFs are informed by the flow of net benefit in each period, each aggregates net benefit across all time periods. Even though the CEACs and CEFs assess decision uncertainty, they only consider the total uncertainty at the time horizon. As such, they are insensitive to when the uncertainty rests and where the uncertainty falls. In the same way that the location and timing of costs and benefits are considered to be an important issue [5, 13], the location and timing of uncertainty may be an important consideration in a reimbursement decision. To the degree that it can be captured within specific parameters, analyses such as the expected value of perfect parameter information may be useful; but they maintain the aggregation of uncertainty over time, unless it is implicit in the definition of the parameter (e.g. ‘risk of long-term recurrence’).

Claxton and colleagues recently described the potential importance of recognizing the distribution of health effects over time [6]. They showed how plotting the cumulative net health effect of a technology over the time horizon of the analysis allowed decision makers to understand that costs and benefits from healthcare do not occur simultaneously, and that many healthcare interventions can be usefully thought of as an investment. However, the remainder of their report focussed upon the relationship between investing in a new technology and investing in additional evidence regarding the value of that technology; rather than communicating information on the location of uncertainty between the initial investment and subsequent returns, and its distribution over time to decision makers.

As described above, current methods for presenting the results of cost effectiveness analyses aggregate the net health effects over time and uncertainty. Prior to Claxton et al., the distribution of the net benefit over time was not reported, even though this could be easily done by plotting the net benefit over the time horizon of the analysis. Similarly, analyses to date have not shown how the uncertainty in the net health effect is distributed over time; an omission that is maintained in the work of Claxton et al. [6].

In cost-effectiveness models, for each time period we simulate expected costs and outcomes for patients treated with the index and the comparator technology. Over the time horizon of the analysis we have a stream of expected costs and outcomes for each technology and for each application of the model within the PSA. The analysis includes the simulated cost and outcome figures for each realisation of each option, at each time point up to and including the final time horizon. The discounted incremental net health effect can be calculated for each simulation of the comparison, for each time period in the analysis. What is lacking is an accessible means of presenting this information to decision makers.

3 The Net Benefit Probability Map

Presenting information in three dimensions, in this case, INB, time and the ceiling ratio, is inherently problematic. Maps are well established as a means to present three dimensions of information in two dimensions. For example, in topological maps, the third dimension captures the height of the ground above sea level using contours; in the simplest contour map, only the contour at sea level is displayed. In essence, each contour allows a horizontal ‘slice’ through a three-dimensional surface. Alternatively, taking a vertical ‘slice’ parallel to an axis holds one dimension constant to explore the relationships between the other dimensions.

A CEAC can be viewed as vertical slices of this type. The CEAC selects an INB dimension that identifies the probability of cost effectiveness, and takes a vertical slice through the surface at a time equal to the time horizon. The CEAC therefore displays changes in only the ceiling ratio, holding the time horizon constant. The CEF selects a similar vertical slice, and combines INB dimensions to display the probability of cost effectiveness for the net benefit-maximizing option.

If the expected magnitude of INB is important to decision makers, then this could also provide an alternative diagram, with vertical slices again taken at a series of ceiling ratios. The Incremental Net Health Benefit-Break Even Curve (INHB-BEC) displayed in Fig. 1 can plot the expected INHB over the time horizon of the analysis. We observe that by 23 years the total impact of the new treatment has become positive, that is, using it is likely to produce more health than it is expected to displace elsewhere in the healthcare system, under the assumption that health is displaced as an immediate impact of costs being incurred. Whilst decision makers are likely to be interested in how long they should expect it to take for the investment in a new technology to pay off, such a curve provides no insight into the risk that it will not pay off in any given time scale. What is required is a means of communicating the information on the expected net health effect and the uncertainty around that expectation at any point in the time horizon of the evaluation.

Fig. 1
figure 1

Incremental Net Health Benefit-Break Even Curve (INHB-BEC)

Full size image

The NBPM takes simulation data for INHB and identifies the uncertainty at each time horizon for the analysis. The diagrams show cumulative probabilities for INHB by identifying deciles at each time horizon. This NBPM diagram displays the time horizon (on the horizontal axis) and INHB (on the vertical axis). Figure 2 shows an illustrative NBPM, with the expectation of the total net health effect displayed as a solid black line and the deciles displayed as broken grey lines. In this way, the uncertainty in outcomes can be presented in a fashion whereby decision makers can ‘read off’ the changing probability distribution of net health effect as the time horizon of the analysis is modified. Whilst simple fund/do not fund decisions are correctly based upon expected net benefit [3], such black and white decisions are rare in practice and rely upon decision makers being risk neutral. Decision makers will frequently need to consider more nuanced options such as whether to await more research, whether to engage with alternative payment schedules and whether to weight costs and benefits differently. They may also, on occasion, choose to consider whether the level of risk involved is sufficient to cause concern; if so, the standard assumption of risk neutrality may be violated [4].

Fig. 2
figure 2

Net Benefit Probability Map (NBPM) for a standard case

Full size image

4 NBPM and Decision Making

4.1 Research and Reimbursement

There is an increasing interest in the potential of further research as a supplementary decision option for reimbursement authorities considering a new (and typically costly) technology [14]. It is increasingly understood that the time it takes for a research study to report is an important consideration in establishing its value from a reimbursement perspective [10, 15]. Likewise, it is important to stipulate what access, if any, patients who are not included in evidence development will have to the treatment in question [10]. The NBPM allows the decision maker to read off the likelihood that a technology will have demonstrated whether it is cost effective within a time period. In the illustrative example (Fig. 2), we would expect such a scheme to run for nearly 13 years before the accumulated evidence would be expected to demonstrate that the technology was more likely than not to be cost effective.

4.2 Discounting Costs and Benefits in Reimbursement

As the net benefit at a point in time will depend on the degree to which costs and benefits are discounted, any changes in the methods used to discount costs and benefits will impact on the NBPM produced. Whilst there is a long-standing (and continuing) debate on whether analyses for reimbursement decisions should attach a different weight to costs and health benefits [13, 16, 17], the NBPM can complement this debate by providing input on the impact of different assumptions surrounding discounting. These include the recommendations in the UK, where the National Institute of Health and Clinical Excellence recently mandated the use of differential discount rates for technologies that have long-term payoffs, such as vaccinations [18]. More generally, decision makers may require submissions to explore the impact of alternative discount rates.

Figure 3 displays the NBPM for the exemplar technology under differential discount rates; 3.5 % for costs and 1.5 % for benefits. The adoption of a differential discount rate has three distinct effects. First, the INHB increases from 0.48 to 0.86; secondly, the expected time to break even reduces from 12.5 to 11 years; and the uncertainty in the INHB increases substantially. The 95 % credible range for the expected INHB is −0.07 to 1.02 using an equal discount rate, compared with −0.08 to 1.61 under the differential discount rate. Whilst the effect is marginal in this example, the increase in the risk of a negative INHB due to the differential discount rate, translates into an increase in the value of further research. By making this type of effect more transparent, the NBPM allows for a clearer examination of the impact of alternative discounting mechanisms.

Fig. 3
figure 3

Net Benefit Probability Map (NBPM) for a standard case with differential discounting

Full size image

4.3 Impact of Alternative Payment Strategies

Using the same hypothetical technology as in the previous section, we examine the impact of a change in the pricing mechanism for this technology and how this can be shown using an NBPM. Specifically, as part of a patient access scheme, the manufacturer has agreed that the health system only pays for patients whilst they are responding to therapy. Figure 4 is the NBPM for the technology under the patient access scheme.

Fig. 4
figure 4

Net Benefit Probability Map (NBPM) for patient access scheme

Full size image

The adoption of the patient access scheme strategy substantially reduces the time to break even, from 12.5 to 5.5 years. Note that the distribution of the uncertainty over time is changed and this will impact upon the absolute value of investing in further research and the relative value of an ‘only in research’ compared with an ‘only with research’ strategy.

5 Discussion

Healthcare systems are increasingly under pressure to provide funding for innovative technologies. These technologies tend to be characterized by high acquisition costs, the potential to make valued contributions to patient health in areas of relative unmet need and uncertainty about their actual impact on health. Against this background, healthcare systems have become increasingly interested in the uncertainty in the evidence base for the effectiveness and value of innovative new technologies and the risk of making the wrong decision. There is increasing interest in bespoke appraisal and payment mechanisms [7, 8, 19, 20] to better balance the risks and potential benefits that innovative technologies can represent.

In this article, we have highlighted the highly aggregated nature of the outputs from standard economic evaluations, especially with regard to the distribution of uncertainty over time, and argued that this aggregation curtails decision makers’ ability to fully understand the impact of the emerging evaluation and payment mechanisms. We have noted that disaggregate information on the timing of and uncertainty in the estimate of net benefits is generated by a standard PSA, and what is required is a mechanism for the effective communication of this information to decision makers.

Any analysis relies upon the aggregation of uncertainty to render the information communicated sensible to decision makers. As an illustration, INB is an aggregation of several items of potential interest, including the likelihood of selecting the most cost-effective outcome, the option that maximizes net benefit, the value of the threshold and the distribution of INB (or its descriptive statistics). Any or all of these dimensions might be deemed important within decision making and so a multitude of diagrams are possible. Indeed, it is arguable that a fully informed decision maker will require uncertainty to be represented in multiple ways.

Many attempts have been made to display uncertainty, including the probability of cost effectiveness versus the threshold value within the CEAC [1], or the same information plus ICERs/optimal-seeming outcomes in the CEF [3]. Other attempts have been made to plot distributions, for example, by plotting INMB values (including a confidence interval) versus the threshold value [21].

As a simple extension of the concept of the CEAC, the INHB-BEC shows how long it takes for the return on investment to break even under a variety of threshold values. However, it does not provide the decision maker with insight into the uncertainty associated with the expected return on investment. This information operates in three dimensions: INHB, time and probability. We propose the use of net monetary benefit probability contours to allow the presentation of all three pieces of information (for a single threshold value) in a manner that is readily interpretable by decision makers, the NBPM.

We have shown how the NBPM appears capable of enriching a decision maker’s consideration of reimbursement decisions in the context of delaying reimbursement for further research; applying differential discount rates for costs and benefits; and alternative payment strategies under patient access schemes. These factors have been previously cited by decision makers as being of relevance [7, 16, 22, 23].

As the primary purpose of the NBPM is to aid in the presentation of existing information rather than the generation of a great deal of new information, it is unclear to what degree it would potentially affect the decisions made in practice. If no change was observed in practice with the use of NBPM information, its use might be associated with easier decision making. However, if a change was observed, then this might instead relate to the ability of decision makers to more easily operationalize their priorities over items that they have previously declared to be important. It is possible that this might lead to a greater priority for technologies with shorter times to break even over those with longer times. If this more accurately reflects organizational priorities, the quality of decision making from the viewpoint of the decision maker will have improved. Whilst such an outcome might be considered ‘perverse’, it remains incumbent on the decision makers to justify both their procedures and the decisions made.

6 Conclusions

It has long been accepted that healthcare reimbursement decisions are based upon much more than simple efficiency considerations. The role of uncertainty and risk in these decisions is increasingly being codified in decision-making processes [24], along with the consideration of values attached to the health of patients affected by funding decisions. In this context, the NBPM disaggregates the information that is generated by cost-effectiveness analyses to provide decision makers with a more nuanced understanding of the evidence on the likely value of investing in innovative new technologies, the risks associated with those investments and the impact of changes in certain values used to appraise technologies, and payment strategies on population health over time.