1 What is the scope and purpose of economic evaluation?

Discussions regarding the appropriate scope of costs and benefits for inclusion in economic evaluation in healthcare have been an important strand in the literature for much of the last two decades. Almost from the beginning of the use of cost-effectiveness analysis in healthcare, there have been discussions regarding whether productivity costs should be included and—if so—how. Relatively quickly, the discussions moved on to whether so-called unrelated medical care costs should be included, which in turn was extended to consideration of future non-medical costs. Subsequently, attention shifted to capturing the quality-of-life effects of therapies on carers and then, more broadly, on members of the family/household. This discussion developed into ‘spillover effects’—the impacts of interventions on members of the household of the individual receiving therapy. Most recently, the Second Panel on Cost Effectiveness in Health and Medicine proposed the use of ‘impact inventories’—described as a “structured table that contains consequences both inside and outside the formal healthcare sector” [1,2,3,4,5,6,7,8]. I argue that, with current approaches to health technology assessment, including spillover effects in economic evaluations will exacerbate an already substantial inequity in the value and voice of those who bear the opportunity cost of technology funding decisions, and therefore, spillover effects should not be routinely included in economic evaluations.

When the purpose of economic evaluation in healthcare is to identify, quantify and value the impact of ill health and treatments for ill health, to argue that some impacts are in scope and others are not is at best arbitrary. Individuals wishing to understand the value of investing in preventive treatments and positive health-related behaviours will benefit from having the most comprehensive account possible of the impacts. Equally, government ministers considering additional taxation to fund new healthcare interventions are likely to appreciate the most comprehensive account of the expenditure on the population they serve. Realistically, there may be practical difficulties in quantifying and valuing some impacts, but these are challenges for the implementation of economic evaluation in specific contexts, not principles that would exclude such factors from the  its scope.

Given these observations, why be cautious? The need for caution flows from the observation that the purpose of economic evaluation is rarely ‘to identify, quantify and value the impact of ill health and treatments for ill health’. Economic evaluations in healthcare are most frequently undertaken to inform decisions about whether a specific technology should be funded from a specific more or less fixed budget, the budget having been allocated through public or private processes to provide healthcare to a defined population. When this is the function of economic evaluation, consideration of the appropriate scope of costs and benefits for inclusion has a normative component that derives from the policy objectives of providing healthcare.

2 Economic Evaluation, Healthcare Coverage Decisions and Equity

The above role for economic evaluation is most frequently formalized through health technology assessment (HTA) processes, such as the Common Drug Review in Canada, the Pharmaceutical Benefits Scheme Advisory Committee in Australia and the UK’s National Institute for Health and Care Excellence (NICE) Technology Appraisals [9,10,11]. Typically, HTA agencies are charged with reviewing identified technologies and recommending for or against funding from the available budget. As these agencies cannot modify the health system budget, they tend to assume it is effectively fixed. This assumption places their use of cost-effectiveness analyses firmly in the ‘supply side model’. Introducing a new technology that has a positive budget impact requires that some current healthcare activity is discontinued [12]. The cost-effectiveness threshold represents the estimate of the marginal productivity of the healthcare system, and decision makers assume the least valuable healthcare activity is displaced. For HTA agencies, what care is actually displaced and hence which patients are affected, is unobservable. The observable impact of their decisions is constrained to the patients and carers who will benefit from the technology being assessed if it is funded. This introduces an immediate inequity and procedural injustice in the practice of HTA. The patients who could benefit from the technology being appraised have ‘voice’ in the process [13], whereas those who could be harmed—through displaced healthcare funding—do not.

The practical importance of this inequity and procedural injustice depends, in part, upon the assumptions made by decision makers with regard to the displaced healthcare and the characteristics of the individuals and households who would have benefited from that care. As indicated above, the conventional assumption is that technologies deemed to be cost effective displace technologies that produce less health. The focus is on the health produced by the technologies, not any characteristic of the individuals who receive or lose health. The implied equity position is often described as ‘a QALY is a QALY is a QALY’, although this position is not dependent upon the measure of health being the quality-adjusted life-year (QALY). The key point is that decision makers are indifferent to who receives and who loses health; it is maximizing the health produced that matters.

When the scope of costs and benefits included in economic evaluation is expanded, the additional considerations are frequently associated with the characteristics of the individuals or households affected. It is not necessarily the case that society would support preferential access to healthcare being a function of these characteristics. Consider the impact of incorporating productivity costs. This will depend upon the socioeconomic status of the individuals who would benefit from the technology being appraised. Therapies for diseases that are more prevalent among high-income groups will appear more attractive, whereas those that are more prevalent among low-income groups will appear less attractive. The incorporation of out-of-pocket costs has the potential to similarly favour higher-income groups, e.g. the amount that a tenured professor spends on mobility aids will likely be higher than that of an assistant professor, which will in turn likely be greater than the expenditure of the cleaner who empties their bins each day. Including unrelated medical care costs could also have a differential impact across the socioeconomic spectrum. Individuals with greater life expectancy will, all things being equal, accrue greater unrelated costs simply by living longer. Further, when care is consumed more intensively by those with less severe needs (the inverse care law) [14], the impact of including unrelated medical costs will be unequally distributed across the population.

The impact of incorporating carer quality of life might vary by socioeconomic status. Carers who can afford respite care may carry a smaller burden than those who cannot. Households where the carer is a child may incur a greater burden with long-term sequalae [15] than those where an adult is the carer. Similarly, the scope for spillover effects will vary systematically by household characteristics. The scope for spillover effects is inherently some function of household size; single-person households have constrained scope to exhibit spillover effects.

The introduction of factors that vary systematically with the characteristics of the individuals or households affected by the decision requires explicit consideration of their normative implications. Diseases are not uniformly distributed across the socioeconomic spectrum, so specific changes in scope will have quantifiable impacts upon the probability of a technology being funded and thus potentially change the distribution of health across the population. Cookson et al. [16], among others, have started to describe methods by which the equity impact of technology funding decisions can be captured. These approaches seek to capture the impact on the distribution of health that would flow from the adoption of the technology being appraised across the ‘equity groups’ of interest, e.g. by socioeconomic status.

There is a second consideration that is not readily open to quantitative analysis. The individuals whose healthcare will be displaced to fund the new technology, and the households within which they reside, have characteristics that will impact upon productivity costs, out-of-pocket costs, unrelated medical costs, carer quality of life and spillover effects. Unfortunately, decision makers typically do not know whose healthcare will be displaced. As the scope of the costs and benefits included in cost-effectiveness analyses for HTA increases, so too does the scale of the assumptions that decision makers must make about these unidentified individuals. Could the cumulative magnitude of the ‘expanded scope’ considerations be greater for the unidentified individuals? It is not difficult to identify investment–disinvestment pairs in which the effect of considering the extended scope factors for the disinvestment population could reverse a positive or negative technology assessment recommendation. For example, productivity costs for an advanced cancer therapy are likely to be a lot smaller than for a psychiatric intervention, whereas the spillover effects might work in the opposite direction [17]. Decision makers should understand that decisions made on the basis of ‘extended scope’ economic evaluations, where the inclusion of the additional factors has a material impact upon the decision, may have the reverse of the intended impact unless the evaluation identifies those individuals and households who will bear the opportunity cost.

3 Summary

When economic evaluations are undertaken for purely descriptive purposes, it is appropriate to capture all credibly related costs and outcomes. Arbitrary limitations on scope are difficult to justify. When economic evaluations are undertaken to inform resource-allocation decisions, then the appropriate scope for costs and benefits should rely on the scope of costs and benefits identified as relevant by the decision maker for whom the analysis is undertaken. When the decision maker is operating on the basis of allocating a fixed budget, the expansion of the scope of costs and benefits beyond direct costs funded from that budget and health accruing to the treated individual may lead to unintended effects on the distribution of health and access to healthcare in the population for which the decision maker is responsible. Further, where the decision-maker’s objective includes consideration of the extended scope, ignorance of the characteristics of the individuals and households who will bear the opportunity cost means it is not possible for the decision maker to know whether the total impact of their decision will be positive, negative or zero.

4 Rebuttal to Brouwer

Dr. Brouwer’s commentary [18] recognizes the importance of considering the individuals and households affected by opportunity cost. His statement “If the introduction of a new technology is believed to lead to the displacement of existing care” (Sect. 5 [18]) may be usefully recast as “If the function of economic evaluation is to inform health technology coverage decisions for a specified population from a fixed budget …” (paragraph 2, above). Brouwer rightly states that, under these circumstances, spillover effects and costs will apply to the displaced care. He goes on to make two important statements—one positive, the other normative—“spillover effects can (positive) and should (normative) be included in such a context.” [20]. He recognizes that often decision makers do not know whose healthcare will be displaced but defends his normative stance, arguing that (1) ignorance about displaced care has not stopped decision makers from using economic evaluations to make decisions and (2) if decision makers take account of spillover effects, we will learn more about spillover effects in general and hence improve the evidence base for describing the spillover effects of displaced care [18].

The first argument strikes me as problematic if we wish to pursue evidence-based decision making. It ignores one of the key lessons from empirical literature on supply side cost-effectiveness thresholds [19,20,21,22]. The gap between the ‘expert judgement’ about the opportunity cost and the empirical estimates is such that historical coverage decisions have likely substantially harmed population health. As researchers, should we not counsel caution on the basis of this experience and avoid encouraging decision makers to forge ahead with a systematic expansion of the scope of ignorance in the evidence base for their decisions? What is needed is evidence on the spillover burden across the spectrum of disease and by socioeconomic characteristics—a spillover atlas, if you will. This would allow analysts to provide decision makers with insight into whether the spillover effects for the identified beneficiaries were above, below or around the average. Analysts might even provide ‘opportunity cost’ scenario analyses to examine the circumstances where spillover augmented net benefit might be the reverse of the base-case analysis.

The second argument is problematic for a number of reasons. First, it suggests we should be content to risk sacrificing patients’ health—without their consent—to gather information about the magnitude of spillover effects from healthcare. Second, it assumes the populations for whom economic evaluations are undertaken are likely to be representative of the populations whose care is likely to be displaced. The probability of having a new technology developed is not equal for all patient populations. A review of the diseases for which new drugs are licensed shows a systematic over-representation of cancers and rare diseases [23, 24]. Licensing drives the work programmes of coverage decision-making processes, and therefore the evidence on spillover effects obtained from these processes will not be representative. Finally, even if we had no ethical concerns and we believed the data generated would be representative of both the ‘winners’ and the ‘losers’ from coverage decision processes, would this not be a very inefficient research design? Here again, the development of a spillover atlas to support an evidence-based rather than assumption-informed approach [25] strikes me as a more efficient and robust strategy for  moving the spillover effects agenda forward.

The literature on the measurement and valuation of spillover effects represents one of the most exciting developments in the methods of economic evaluation over the last decade. Brouwer et al. [18] have done the discipline a great service in reminding us that all effects in the population of interest, whomsoever they fall on, should be captured. In due course, these methods will improve the quality of coverage decision making and hence the value of societies’ investments in healthcare. However, as our experience with assumed cost-effectiveness thresholds should teach us, making decisions about the net impact of new technologies in the absence of credible evidence on opportunity cost may do substantial harm [26].