Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Health technology assessment (HTA) examines the consequences of the application of health technologies aimed at better informing decision-makers. As such, HTA has become a topic of great interest, albeit not without controversy. HTA advocates argue that it promotes efficiency of resource allocation, while critics state HTA is simply a means to restrict access to new and costly technologies (O’Donnell et al. 2009).

Over the past decades, different countries have established HTA organisations to better inform healthcare policies and clinical practice. HTA agencies have gained space in taxation-based and social health insurance systems. In fact, most high-income countries (HICs) utilise some form of HTA process to facilitate decision-making and priority setting within their health systems (Bulfone et al. 2009; Castro 2011). Recent examples of HTA agencies in the developing world have also emerged (Castro 2012).

HTA, although important, is only a part of the process of decision-making (Cleemput et al. 2012). Beyond scientific evidence, decision-making also requires value judgements (Eddy 1990; Tunis 2007; Cleemput et al. 2011). Neither HTA reports nor the results of cost-effectiveness analyses should be blindly used to make decisions.

While multiple studies and publications have examined the role of HTA as a data collection process (Heyse et al. 2001; Briggs 2001; Briggs et al. 2002; Hoch et al. 2002), there is still limited knowledge of how decision-makers used this data, as well as the challenge of incorporating other criteria in an explicit manner. Authors like Drummond and Sorenson (2009) have suggested a “divorce” of the evidence produced and decision-making process, since many HTAs and economic evaluations published in the literature have been performed with no specific decision-maker in mind.

Even in countries where formal HTA activities are ongoing, and in most low- and middle-income countries (LMICs), many resource-allocation decisions are still based on non-transparent choices that reflect competing interests of governments, donors and other stakeholders (Glassman et al. 2012). Frequently, decision-making is inconsistent and unstructured. Important criteria such as budget impact, equity and disease severity have not always been taken into consideration, and if they have, it is not often clear how they have impacted a final decision (Baltussen and Niessen 2006). This can lead to implicit and covert rationing through waiting lines, low quality and inequities (Glassman et al. 2012).

Multi-criteria decision analysis (MCDA) has emerged as a tool to support decision-making in healthcare (Miot et al. 2012) attempting to move beyond the evidence generation/collection phase of the process. MCDA methods are designed to help people make “better” choices when facing complex decisions involving several dimensions. “MCDA are especially helpful when there is a need to combine ‘hard data’ with subjective preferences or make trade-offs that involve multiple decision-makers” (Dolan 2010). In theory, MCDA allows a structured and objective consideration of the factors that are both measurable and value based in an open and transparent manner (Baltussen and Niessen 2006; Dolan 2010) thus could be considered an important step towards rational priority setting in developing countries (Baltussen et al. 2007; Miot 2012).

The aim of this chapter is to demonstrate the potential of, as well as the challenges associated with, using MCDA for resource-allocation decisions by presenting case studies carried out by well-established institutions in Colombia, the region of Lombardy in Italy and Belgium.

2 The Case Studies

2.1 Testing MCDA in Colombia

The Colombian Regulatory Commission for Health (CRES) operated until December 2012, as the coverage decision-making body. Arguably, CRES was disbanded because of a lack of “legitimacy”, and the Ministry of Health and Social Protection (MoHSP) regained reimbursement decision-making powers. This institutional instability created the opportunity to test MCDA methods.

The MCDA framework Evidence and Value: Impact on Decision-Making (EVIDEM) developed by Goetghebeur et al. (2008) was the one used by CRES in Colombia before its disbandment when attempting to implement a more systematic priority-setting process. EVIDEM is an open-source generic framework intended to help judge the value of interventions from two perspectives: the value system of the evaluator (decision-maker) with regard to the importance of each criteria (weights) and the performance of an intervention on preselected decision-making criteria (scores).

EVIDEM includes core quantifiable and contextual qualitative criteria considered important in decision-making; this approach has been tested and used in several countries (Guindo et al. 2012; Goetghebeur et al. 2010, 2012; Tony et al. 2011; Miot et al. 2012). The framework also includes detailed protocols for the collection, analysis, synthesis and reporting of evidence for each decision criterion (by criterion HTA report). Appraisals are transformed into a holistic MCDA value estimate which allows for ranking and cross comparison of healthcare interventions.

2.1.1 Methods

The methodological approach taken in Colombia is similar to the steps followed by previous applications of EVIDEM for coverage decision-making (e.g. Miot et al. (2012) in South Africa, Tony et al. (2011) in Canada) (Fig. 8.1).

Fig. 8.1
figure 1

Scheme of work for piloting EVIDEM (Source: Adapted from Goetghebeur et al. (2012))

Full size image

During a preparatory stage, investigators conducted literature searches and produced HTA reports for each intervention of interest, followed by panel sessions of decision-makers to contextualise criteria to be used, establish a committee perspective (weighting of criteria), appraise each intervention (scoring and consideration of criteria) and discuss the results and process.

2.1.1.1 Selecting Criteria and Assigning Weights

During October 2012 CRES led an independent initiative aimed at selecting criteria for coverage decision-making during three workshops involving 11 senior decision-makers (academics, researchers and civil servants) with broad experience of working in the context of the local health system. Participants were asked to identify additional contextual criteria considered relevant for resource-allocation in Colombia. After three voting rounds in two nominal group sessions, a final list with 15 criteria was produced, 13 from the EVIDEM core model and two added contextual criteria (bold) (Table 8.1).

Table 8.1 Final list of criteria and weights for the Colombian-modified version of EVIDEM
Full size table

Once the panel had agreed on the final criteria and their definitions, participants were asked to weight each criterion irrespective of any healthcare intervention. A participatory process was implemented by CRES, who organised meetings with various stakeholders (academics, patients associations, citizen councils and representatives from the medical societies) around the country and asked them to assign weights. A total of 201 citizens weighted each of the 15 criteria (CRES 2012).

2.1.1.2 Assembling the Evidence for Selected Technologies

Four technologies were selected for the pilot: primary prophylaxis (PP) for severe haemophilia A (SHA), zinc supply for diarrhoea prevention, anastrozole as first-line therapy for hormone receptor-positive postmenopausal women with metastatic breast cancer and ticagrelor + acetylsalicylic acid (ASA) for patients with acute coronary syndrome (ACS) without ST elevation and moderate to high cardiovascular risk. Technology selection was partly based on convenience with availability of published local HTA summaries for these interventions. In addition, all three non-haemophilia-related technologies were considered as potentially cost-effective, while prophylaxis was not. At the time of running the pilot, no reimbursement decisions had been made as to whether they would be publicly reimbursed.

The clinical practice guidelines by Perry et al. in 2012 (anastrozole), Florez et al. (zinc) (2013) and Senior et al. (ticagrelor) in 2013 were used to produce the HTA reports. In the case of PP, the HTA report was based on a recent cost-utility analysis (CUA) and a literature review (Castro et al. 2013). The adapted EVIDEM MCDA matrix was used to assemble the HTA information of the four technologies in Spanish. All reports contained the relevant information organised considering the criteria and weights developed by CRES in 2012.

2.1.1.3 Appraisal of Interventions and Discussion

Because CRES was dissolved before appraising the value of any intervention, a new focus group was organised as a mock reimbursement decision committee to appraise the value of the four interventions in August 2013. The focus group was designed to mimic a resource-allocation decision-making committee, 12 organisations were identified as containing potential sources of participants (government, insurers, providers, patients groups, academics, healthcare professionals, people’s advocates and lay members). Senior policy-makers and “high profile individuals were selected to assure legitimacy and ‘buy in’ of the pilot”.

Since traditionally resource-allocation decision-making occurs as a centralised process in the country, all eligible participants were located in Bogotá. The feasibility and usefulness of using and incorporating HTA and EVIDEM to inform resource-allocation decision-making were explored during a 2-h focus group held at the Health Technology Assessment Institute-IETS through a set of open-ended questions. All participants were asked to consent to participate and to be recorded for transcription and to declare potential conflicts of interest.

To appraise the healthcare interventions, respondents were presented with MCDA evidence matrices which prompted HTA summaries and ask to score each criteria individually on a four-point cardinal scale (0–3), where 3 represents the highest level of fulfilment of each decision criterion and 0 the lowest (EVIDEM v2.0). The calculation of the MCDA value estimates was done by combining normalised weights and scores for each individual using a linear model with 1 being the highest value for an ideal intervention and 0 the lowest. Averaged results compiled at the group level were presented to participants at the end of the session to promote discussion.

To promote discussion, participants were presented with a hypothetical scenario where only those two technologies with the highest scores were to be reimbursed by the healthcare system. Questions such as “was there enough information to make resource-allocation decisions in Colombia?” and “what changes or improvements could be added to the processes and methods presented in the pilot for future implementation”? were asked to participants to gather their inputs, concerns and expectations.

The focus group was recorded, transcribed verbatim and uploaded to ATLAS-ti7 to assist content analysis. In order to interpret emerging data rather than simply describing it, no preliminary hypothesis was considered. Labels such as sufficiency of information, methods concerns, methods comparison, validity of information, incorporation of HTA into decision-making and the specific value of each intervention were predefined as the relevant categories that served to inform the aims of this chapter.

2.1.2 Results

Seven people attended the invitation to participate. Participants represented a broad range of stakeholders within the Colombian health system, from members of the MoHSP, academics, insurers, patients and professional associations to lay members of society. All participants were skilled workers with at least one postgraduate degree. No representatives from hospitals or people’s advocates participated in the meeting, although they were formally invited to attend. Scoring the four technologies of interest using the MCDA evidence matrix took an average of 11.15 min (range 7–18 min) per healthcare technology per participant.

MCDA value estimate calculation indicated that zinc ranked first (0.904) followed by anastrozole (0.822), PP for SHA (0.794) and ticagrelor (0.708) (Table 8.2). Perceived value of interventions varied across participants [zinc (0.782–0.986), anastrozole (0.698–0.934), PP (0.595–0.977) and ticagrelor (0.449–0.945)], reflecting the diverse perspectives and interpretation of presented evidence of participants.

Table 8.2 Results of the EVIDEM comparative value of interventions by criterion
Full size table

In answer to the question could EVIDEM be used in Colombia to assist resource-allocation decision-making, participants found EVIDEM was a means of incorporating HTA into decision-making and also of prioritising different health interventions for resource-allocation. The final consensus was that a mixed methods approach including an appraisal based on an MCDA evidence matrix completed by a financial exercise with a detailed Budget Impact Analysis (BIA) examining the opportunity costs would be ideal for Colombia.

Participants also identified limitations regarding the adequacy of information presented in the EVIDEM summary. Some specific criteria represented more challenges than others for interpretation and valuation. Some doubts emerged when independently valuing each criterion. There was risk of double counting information (consideration of the same evidence in multiple criteria), since no strategy was considered to consistently synthesise HTA evidence to avoid it.

Another limitation of this pilot relates to language differences between the original EVIDEM tools used (matrix and by criterion), published in English and the non-validated Spanish versions presented to participants. The method used to elicit weights applied by CRES in 2012 clearly departed from the 1 to 5 scale originally used by EVIDEM, shall it had an impact on the final results could be as well an important limitation.

Since institutional HTA has been in place for less than 3 years in Colombia, there were still concerns and considerations among participants of the methods to conduct HTA, for instance, the validity of data used for modelling, the use of QALYs when conducting CUAs or the reliance on ICERs alone to inform decision-making but also on how to incorporate HTA results into decision-making. Nevertheless, the pilot is one of the first initiatives within the country to combine HTA and MCDA for more explicit priority setting.

2.2 Institutional HTA/MCDA Approach in Lombardy

Lombardy is a region in the north of Italy with 9.8 million residents served by a healthcare system involving 145,000 workers, 220 hospitals and 2700 pharmacies and an annual health budget of €17 billion. In 2008, the Lombardy Healthcare Directorate (LHCD) issued a policy for an HTA programme to maximise healthcare benefits to citizens by promoting more efficient and evidence-based healthcare resource-allocation and sustainable diffusion of technologies. The HTA programme was therefore based on principles of accountability, orientation to health outcomes, transparency in decision-making and sustainability.

Value for health is defined as health outcomes expected when the National Health Service (NHS) reimburses a health technology over other competing alternatives. The healthcare directorate started the programme with the naming of two representative committees (one for priority setting for emerging technologies and one for appropriateness of diffused technologies), alongside a policy for managing conflict of interest and a website platform to support it and collect contributions from hospitals, companies and experts. The programme mainly addresses prioritisation and appraisal tasks, while technical assessment is limited to the contextualisation of third-party HTA reports into an MCDA evidence matrix.

2.2.1 Historical Perspective and Rationale for Developing and Implementing an MCDA-Based Appraisal Process

The Lombardy government recognised an opportunity to strengthen the HTA programme taking into consideration methods developed by the European network of HTA (EUnetHTA) and the EVIDEM collaboration. EUnetHTA focuses on facilitating knowledge sharing, efficient use of resources and promoting good HTA practices in Europe. It publishes Core Models, guidelines and other resources to streamline assessment practices. The EVIDEM collaboration developed a pragmatic decision-making framework and some tools to help bridge MCDA and HTA in order to clarify appraisal practices.

The LHCD then developed an information framework incorporating adapted versions of both the EVIDEM set of criteria and the EUnetHTA Core Models (version 1.0); this was in order to build a complete, coherent and operational HTA-MCDA application aimed at structuring assessment reports for appraisal activities. A modified set of criteria from EVIDEM was inserted into the EUnetHTA framework under the top level (Domains) and also over the middle-level hierarchy (Topics). The EUnetHTA ontology structure was maintained, except from “Health problem relevance” and “Technology solution relevance” which were merged in the Lombardy’s version as “Technical relevance”, while “Effectiveness” was split into “Efficacy” and “Effectiveness” in order to comply with the original Lombardy regulation, issued 2 years before the publication of Core Models 1.0.

The Lombardy HTA-MCDA application ontology has been implemented as web-based tools for both the quantitative and qualitative stages of the prioritisation and appraisal process. This has helped to clarify processes and better communicate results to hospitals. It has been field tested internally and applied to most HTA projects from 2012 onwards (Migliore et al. 2014; Tringali et al. 2014).

2.2.2 The Appraisal Process

The process starts with a submission from hospitals, manufacturers, independent clinical experts or other bodies (e.g. the region itself or the Italian Agency for Healthcare research and quality-AGENAS). The framework is used by committed members formed by experts selected according to their expertise and declaration of vested interests. The framework contextualises participants on HTA reports and also provides the tools to support participants’ personal judgement. Potential committees’ decisions can be rejected, further assessed or directly approved for reimbursement within Lombardy NHS. Committees’ decisions are usually translated into formal acts.

The appraisal stage proceeds as follows, developers present structured proposals to committee members and then HTA reports are produced internally or adapted from third parties by the region or local hospitals. For each appraisal each committee member is asked to judge on the relative importance of 8 general “domains” (for emerging technologies) or 15 more specific “criteria” (for diffused technologies, for which more information usually is available) through a personal weighting operation using an online form.

The weighting method is always a direct and anchored rating scale. For emerging technologies, each committee member is asked to assign eight to the domain considered as the most important and one to the least important one and then to distribute weights to the other domains to ensure differentiation among domains and avoid flattening of judgements towards a same level of perceived importance. Average weights by domain are obtained after summing up scores by participant and dividing them by number of participants. Alternative methods for weighting, like hierarchical point allocation or pair-wise comparison between individual domains/criteria, have been initially considered but put aside since committee’s expertise with MCDA methods was still at an early stage, and the face validity of the simple eight-point scale weight elicitation method was deemed satisfactory (for more details on weighting methods, see Chapter 4).

The same process is applied by members of the appropriateness committee with 15 quantitative criteria. The individual weightings are then discussed online and during meetings, and each member can modify his/her own weightings. After these are approved, final weights are calculated and members are given access to the full documentation available.

Each member individually scores the performance of the proposed technology for each domains/criteria and with respect to available alternatives of care using an online form, with a predefined scoring system from 0 to 4, where 0 = absence of relevant information, 1 = comparative lesser value, 2 = comparative similar value and 3 or 4 = comparative (slightly or highly) better value. Members also provide a mandatory comment for each score. Uncertainty of scores was not initially modelled and was left to the discussion within committees, but a revision is planned with the introduction of a three-level classification of uncertainty for each assigned score.

Individual scores and comments are then elaborated into a judgement draft, in two parts

  1. 1.

    Priority (or appropriateness) index for the NHS. A linear additive model is used for the analysis of individual value contributions (normalised weights × scores) for each domain/criterion to provide an index from 0 to 1 representing the overall value of the technology as follows:

    1. (a)

      From 0 to 0.25 when the estimate’s averages are among 0 and less than 1, in this case the intervention cannot be evaluated in a robust way.

    2. (b)

      Between 0.25 and 0.50 when the estimate’s averages are among 1 and less than 2, here the relative value of the intervention is less or equal to the value of alternatives.

    3. (c)

      Between 0.50 and 1 when the estimate’s averages stand between 2 and 4 and the proposed intervention has a better overall comparative value than alternatives.

  2. 2.

    Qualitative analysis of comments written for each domain/criterion. Comments are categorised by two reviewers, with resolution of disagreement by consensus, and analysed within a descriptive report, where more frequent and robust arguments are proposed as possible motivations for the decision.

Both priority/appropriateness indexes for the NHS and categorised comments drafts are discussed and revised to verify the coherence between scores and comments, to eliminate ambiguities and to identify further areas of assessment. After revision, the index and the motivations are approved and sent to administration for consideration for policy-making.

For some of the appraised technologies, judgement was repeated by two or three independent subgroups of the committees in a blinded way to measure reproducibility of indexes; since this has always been very high, there have been no cases where there was a need to revise the final judgement. Intra-rater and inter-rater variability of committee members in expressing weights has also been explored and showed a high degree of consistency among voters (yet unpublished work). This internal analysis is now being replicated and extended by an independent academic group. Results of this analysis will help inform the updating of the regional HTA policy in the near future.

2.2.3 MCDA Outputs in Lombardy

From 2012 several diagnostic and interventional technologies have been prioritised, i.e. recommended or refused using the MCDA approach presented above in Lombardy. Figure 8.2 depicts the list of healthcare technologies prioritised for reimbursement during 2012–2014 in Lombardy (more information available at http://vts-hta.asl.pavia.it).

Fig. 8.2
figure 2

Value of healthcare interventions for healthcare programming in Lombardy

Full size image

Most of the proposed technologies have been rejected; in other cases a positive appraisal was followed by reimbursement, sometimes with restrictions for an appropriate use, i.e. indication of specific centres, patient selection procedures and provisional tariffs linked to a conditional reimbursement (upon verification of prospected outcomes as registered in real-life patients).

Note that laser endo-microscopy and presepsin reached a similar value of the priority index, but different decisions were issued by the regional administration on the basis of the overall comments of the committee’s members. XXX denotes a medical device for which an administrative appeal decision is pending against the rejection from reimbursement.

Figure 8.3 is an illustration from the final appraisal document for the trans-vascular aortic valve implantation (TAVI) for aortic stenosis procedure.

Fig. 8.3
figure 3

Extract from the appraisal process for TAVI

Full size image

First, appropriateness indexes for the TAVI procedure in operable and inoperable patients were calculated through MCDA using 15 quantitative criteria by appraisal of committee members (N = 21). The analysis of qualitative judgements expressed by the same members as negative, no or positive impact and related comments for six qualitative criteria helped to prepare a set of final recommendations for TAVI; which in this case were: audit of every case, revised criteria for authorisation of centres, team evaluation of frailty and comorbidities, clinical registry of pathology linked to the financing procedure and conditional repayment-payback if no positive outcome at 2 years (this policy act was issued in 2013).

2.2.4 Latest Developments in Lombardy and Future in Italy

During 2015 the Lombardy HTA-MCDA application ontology was revised to incorporate the content of the EUnetHTA Core Model version 3.0 (draft version as of September 2015) and most of the changes made in version 3.0 of the EVIDEM framework. The resulting updated list of domains and criteria in use is reported in Table 8.3.

Table 8.3 List of domains and criteria
Full size table

The pioneering work of the Lombardy region is now explored by other regions in Italy as well as at the national level. Recently, a prescription for MCDA use was added to a national law for priority setting in the medical devices area highlighting the real-life value of this approach to support HTA, decision-making and communication with stakeholders.

2.3 Developing an MCDA Approach for Coverage Decision-Making in Belgium

Coverage decisions in healthcare in Belgium are taken by the Minister of Health, after advice from the National Institute for health and Disability Insurance (INAMI/RIZIV). This case study focuses solely on drug reimbursement decisions.

The Belgian drug reimbursement procedure underwent notable changes in 2001. The Drug Reimbursement Committee (DRC) was established to appraise the reimbursement requests from pharmaceutical companies and formulate advice to the minister of health.

The DRC consists of different stakeholders in the Belgian healthcare sector, including representatives from academia, physicians, pharmacists and sickness funds (voting members) and representatives from the ministries, pharmaceutical industry and INAMI/RIZIV (consultative members). Voting is done by a show of hands in the presence of consultative members. Individual representatives do not have to justify their vote.

The criteria that need to be taken into account during the appraisal process are defined by law. They include added therapeutic value, drug price and reimbursement basis, clinical effectiveness and likely impact of the product given the therapeutic and social needs, budget impact and cost-effectiveness. Criteria for assessing therapeutic value are also defined by law and include efficacy, safety, effectiveness, applicability and comfort. Added therapeutic value is recognised if the drug use in a given treatment demonstrates an impact on mortality, morbidity and/or quality of life. There is no explicit hierarchy in the criteria.

2.3.1 Transparency

The changes in the drug reimbursement procedures introduced in 2001 substantially enhanced the transparency and use of objective criteria compared to the period before the establishment of the DRC (Cleemput and Van Wilder 2009). However, issues of transparency remained. The appraisal phase remains a deliberation process in which formal as well as informal criteria are used. Moreover, the distinction between the assessment phase and the appraisal phase is not always very clear.

Primary assessment reports and decisions of the minister are published on the website of the INAMI/RIZIV, but it is not always clear which elements eventually led to the advice/decision as the main discussion points and arguments are not reported.

2.3.2 A Belgian MCDA Framework

In 2010, the Belgian Healthcare Knowledge Centre (KCE), an independent publicly financed policy research agency in Belgium, examined ways to improve the accountability for reasonableness of the drug reimbursement system (le Polain et al. 2010). First, it was recommended to make a stricter distinction between the assessment and the appraisal process. Assessment implies the collection of the evidence regarding the technology under consideration. Appraisal implies value judgements, e.g. related to the relative importance of each of the assessment elements. These value judgements should, in a democratic system, ideally reflect societal values and preferences. Second, KCE also presented a possible MCDA framework for making health technology appraisal processes more transparent (Table 8.4). The framework is meant to support decision-making regarding new interventions for different indications.

Table 8.4 Key questions and possibly relevant criteria for a healthcare reimbursement appraisal process (MCDA framework)
Full size table

The framework consists of five questions, corresponding to five intermediate decisions. Each question needs to be answered using explicit decision criteria. The criteria must be (1) relevant and (2) weighted in accordance with the relative importance attached by the general public. The advantage of splitting up the decision process in intermediate questions is that it is cognitively easier for people to consider fewer criteria at once than to consider more criteria at the same time when making a choice (Ryan et al. 2001).

The questions are structured hierarchically, presuming that a new intervention can only be worthwhile reimbursing if there is a need for a better intervention, and the added value of the intervention is sufficient. However, it is not enough that there is a perceived need. Even if there is a need, the new intervention still needs to be better on other criteria considered as important. At a higher need, the better the intervention, meaning a higher propensity to pay for the new treatment with public resources (this on aspects that matter to patients).

The Belgian approach foresees the application of MCDA to each intermediate decision in the framework. In contrast to the examples described in the literature, the Belgian MCDA framework prescribes that criteria weights should come from the general public, because legitimacy in healthcare reimbursement decision-making presumes that societal preferences are taken into account. Because the public preferences for the reimbursement criteria were unknown, KCE performed a large population survey in 2014 to derive these weights. The remainder of this case report will discuss the methods and results of this survey and the application of its results into MCDA.

2.3.3 Deriving Preferences for Healthcare Reimbursement Criteria from the General Public

A random sample of 20,000 people, stratified by age and sex, was selected from the National Registry of all residents. People were invited to either fill out the web survey or request a paper version of the questionnaire.

The survey consisted of nine discrete choice questions, one moral reasoning exercise and a number of demographic questions. The part with the discrete choice questions was structured in three blocks:

  1. 1.

    Discrete choice questions for defining the relative importance of criteria for assessing therapeutic need, i.e. the need for a better treatment in a particular disease given the treatment already available, as determined by the quality of life under current treatment, the impact of the disease on life expectancy despite current treatment and the current treatment’s inconvenience

  2. 2.

    Discrete choice questions for defining the relative importance of criteria for assessing societal need, as determined by the prevalence of the disease and the public expenditures per patient with that disease

  3. 3.

    Discrete choice questions for defining the relative importance of criteria for assessing added value of a new intervention relative to the best alternative intervention, as determined by the impact of that new intervention on all previous criteria

The criteria included in each block have been determined through literature review and expert workshops. With the objective of developing a generic MCDA in mind, the criteria were phrased to be relevant for all health conditions, i.e. not disease specific, hence allowing comparison across indications and potentially leading to optimal resource allocation. The criteria included in each block are presented in Table 8.5.

Table 8.5 Criteria included in the survey
Full size table

Responses of 4288 participants from the general public between 20 and 89 years of age (21.4 % out of 20,000 people invited, 89.2 % of respondents) were used for analysis. A multinomial logistic regression analysis was performed to analyse the data and in order to obtain level-independent but criterion-specific weights; a method based on log-likelihood differences between model specifications was used.

Depending on the block, respondents were asked to choose between two different patient groups (block on therapeutic need), two different diseases (block on societal need) or two different health interventions for the same disease (block on added value). With 24 different versions of the questionnaire, differing in the description of the scenarios between which to choose, and three choice sets for therapeutic need, 1 for societal need and 4 for added value, it was possible to obtain weights for each criterion included in a specific block.

The weights were calculated using the following algorithm:

  1. 1.

    Estimation of a multinomial logit regression – also referred to in literature as conditional logit – model for each block.

  2. 2.

    For each block, relative preference weights using the log-likelihood method were calculated:

    1. (a)

      Calculate the log-likelihood for the model.

    2. (b)

      Calculate the log-likelihood for the model minus one of the criteria, which represents the criterion of interest (=the reduced model).

    3. (c)

      Test if the reduced model is statistically equal to the full model with the likelihood ratio test. If the test rejects the equality hypothesis, consider the relative importance of the removed attribute to be different from zero.

    4. (d)

      Calculate the difference in log-likelihood between the full and each reduced model as a measure of relative importance of the attribute, and convert to a proportion.

This results in three sets of weights, one set for each block. The blocks are not combined in the Belgian model. The assessment of “overall need”, encompassing therapeutic and societal need, remains a matter of judgement. If a disease scores high on both types of needs, it will represent as having higher needs than a disease which scores only high on one of the two. No attempts are made, however, to weight the societal needs (societal perspective) against the therapeutic needs (individual perspective) in this model.

2.3.4 Belgian Weights for Reimbursement Criteria

2.3.4.1 Therapeutic Need

The implicit weights given to the criteria included in the therapeutic needs domain are presented in Table 8.6. The general public gave the highest weight to the quality of life under current treatment. Therapeutic need is considered to be the lowest in people with a good quality of life given current treatment that do not die from their disease and experience little discomfort from their current treatment.

Table 8.6 Weights for criteria in the therapeutic need domain
Full size table
2.3.4.2 Societal Need

In the appraisal of societal need, people give more weight to the impact of a disease on public expenditures (0.65) than to the prevalence of the disease (0.35). They consider the need to be highest in very frequent diseases that cost a lot to society per patient.

2.3.4.3 Added Value

During the appraisal of the added value of new interventions, the citizens gave the highest weight to the intervention’s impact on quality of life, followed by its impact on the prevalence of the disease and on life expectancy.

A general observation is that the value loss associated with something negative (higher expenditures, higher treatment discomfort, less patients cured) is higher than the value gain associated with something positive (lower expenditures, lower treatment discomfort, more patients cured). For example, the negative effect on the perceived added value of increasing public expenditures is higher (−0.43) than the positive impact of decreasing public expenditures (+0.23). Table 8.7 presents the weights for criteria determining the added value of new treatments.

Table 8.7 Weights for criteria determining the added value of new treatments
Full size table
2.3.4.4 Using the MCDA in Decision-Making

The framework described is not yet being applied in practice, but it is going to be used from 2016 onwards in the context of early temporary reimbursement decisions.

MCDA could be applied every time the reimbursement of a new treatment is requested. This involves (1) scoring diseases and treatments on the selected criteria, (2) weighting the scores and (3) summing the weighted scores. The clinical significance of the impact of a disease or treatment on a criterion is reflected in the scoring, while the extent to which that clinical significant or insignificant effect should matter for the decision is reflected in the weights.

The MCDA is applied as follows:

  • Step 1: Consideration of the condition targeted by the new treatment and the current treatment for the condition.

    The committee members consider the condition targeted by the new treatment and score the criteria relating to therapeutic need and relating to societal need.

    For the scoring, the committee members should have an assessment report describing the existing scientific evidence regarding each criterion, as well as the evidence gaps. The members could consult external experts, e.g. in case of insufficient or inconclusive evidence.

  • Step 2: Consideration of the added value of the new intervention

    The committee members score the criteria for added value for the new intervention for which reimbursement is being considered. The scores should be based on the best available scientific evidence.

  • Step 3: Weighting of scores for therapeutic need, societal need and added value

    The scores are weighted with their respective public preference weights, as derived from the survey. This is done by multiplying the score with the weight. For each domain the weighted scores of the domain-specific criteria are summed.

    This, results in three scores: one for therapeutic need, one for societal need and one for added value of new treatment. Higher scores represent a higher level of priority in terms of therapeutic need, societal need or added value of treatment, depending on the domain considered. By repeating the MCDA for different decisions, a priority ranking of diseases and treatments will eventually be obtained.

  • Step 4: Deliberation about the resulting scores for therapeutic need, societal need and added value

    The three total weighted scores allow the commission to consider in which quadrant of Fig. 8.4 the intervention is located. The higher the need and the higher the added value, the more likely it is that reimbursement can be considered. Whether it will be reimbursed is still a matter of willingness to pay, and this is something to be judged by the decision-makers. The process remains deliberative on this point. However, the number of interventions about which deliberation regarding willingness to pay is needed reduces (for interventions on the left of the Y-axis, no further discussion is needed, unless criteria have been missed in the MCDA).

    Fig. 8.4
    figure 4

    Preparedness to pay (more) for a new intervention

    Full size image

    There might be criteria that are not included in the MCDA that also matter to the decision. Deliberation should include discussions about whether there are other criteria –not yet included in the MCDA – that are important and that would justify a change in the priority ranking in terms of need or added value. For example, it could be that policy-makers wish to give higher priority to prevention than to cure. If that is the case, preventive interventions might be moved up in the priority ranking. If additional criteria are considered important, they should be made explicit, and the committees should explain how these additional criteria modified the ranking of a disease or a treatment.

Unlike many other MCDA approaches, the Belgian approach does not combine appraisal of need for a better treatment and added value of new treatments in one single weighted score. Although this might be considered a weakness, it could also be regarded as strength. The appraisal of need and added value requires a very different viewpoint: the first one encompasses disease-related criteria and the second technology-related criteria. It is hard to understand how such diverse criteria can be weighed against each other.

3 Discussion

Motivations for this chapter coincided with those of Tanios et al. (2013) on decision-makers’ perceptions of the relevance and need of a wider range of criteria to assist decision-making and also Guindo et al. (2012) on the perceived importance of considering both normative and feasibility criteria for fair allocation of resources and optimised decision-making. Literature shows that country-specific HTA organisations and processes for priority setting have emerged globally. This emphasises the need of observing principles such as transparency, robust and appropriate methods for combining costs and benefits, explicit characterisation of uncertainty and active engagement with stakeholders (Drummond et al. 2008; Chalkidou et al. 2009; Pinchon- Riviere et al. 2010); however, there is still little information regarding what shall be considered as “good practice” whenever appraising the evidence to reach a final coverage decision.

All three case studies attempted to fulfil the methodological requirements of MCDA: (1) completeness (all criteria defining the value of interventions are considered), (2) not redundancy (no duplicates are allowed), (3) mutual independency (a criterion’s score is independent from the score given to other criteria) and (4) operationability (criteria are unambiguously defined, assessment data are independently available and directionality of the scoring scale can be universally understood). However, it was very challenging to run an efficient and explicit process to ensure transparency and consistency of relevant factors and also fulfil the methodological requirements of MCDA; thus, limitations were expected to occur.

The methodological challenges posed by the use of MCDA for HTA reported in the literature were coincident with those of the case studies. For instance, in none of these cases, there was explicit account when dealing with uncertainty about MCDA estimates; perhaps, there is need to incorporate additional and more sophisticated statistical methods for dealing with this issue in the near future whenever presenting results to decision-makers. How to estimate opportunity costs remained as a challenge whenever using MCDA to assist coverage decision-making, should cost-effectiveness be kept as a single criterion to be contrasted against empirically estimated ICERs? Or should it be removed from relevant criteria to avoid double counting? In such cases inevitably some methodological trade-offs shall be made in the future. For a more comprehensive discussion of methodological challenges of MCDA, see Chapter 14.

It also emerged that when testing MCDA for cross comparison of interventions, it may not be possible to generate a generic MCDA framework for HTA that fits all needs of decision-makers, since it may be the case in which committee members may prefer a certain set of criteria regarding groups of similar interventions to assure more fair comparison among them. In the specific cases that looked at EVIDEM criteria, some would argue that it does not comply with core principles such as lack of overlapping or preferential independence while others may contend that the importance of this framework is to make explicit account of what is relevant to decision-makers and promote discussion. All these methodological considerations should be borne in mind for the robust incorporation of MCDA into coverage decision-making and for the agendas of future research. Limitations aside, all three case studies were an attempt to assure that after robust HTA has been conducted, transparent and systematic decision-making should be pursued.

Many lessons emerged, for instance, in Colombia and Lombardy on the need to provide more explanation to committee members before piloting; implementation needs some time for familiarising by decision-makers; this is similar to the findings reported by Goetghebeur et al. (2012). Provision of complete information together with homogeneity and coherence of reports could reduce uncertainty among decision-makers and improve consistency across committees and interventions. A final consensus was that a mixed approach including an MCDA evidence matrix completed with a detailed BIA would be ideal for Colombia – this is in line with the recommendations of the EVIDEM collaboration for the operationalisation of such MCDA framework (www.evidem.org/docs/2015/EVIDEM-v3-0-Decision-criteria-conceptual-background-definitions-and-instructions-June-2015b.pdf).

Lombardy government strengthened its HTA programme taking into consideration existing methods developed by EUnetHTA and the EVIDEM collaboration, implying no need to “reinvent the wheel”. Lombardy also utilised web-based tools for quantitative and qualitative stages of the process; this may provide an opportunity to scale up deliberation to wider audiences within the same region or even within the country without representing major costs in the short run. However, commitment of additional resources, a revision of the procedures and a stricter link between HTA and other management programmes (i.e. revision of pathways of care, risk assessment) should be envisioned for the advancement of the forthcoming HTA policy act in this jurisdiction.

In the case of Belgium, it was mainly the objective of transparency and legitimacy of decision-making that triggered work on MCDA. It is only through the use and consideration of the relevant questions with the relevant criteria and their relative weights that the decision-making process can become more legitimate. Decisions about what the budget allows should be in line with what people consider important, both for individual patients as for the society as a whole. The Belgian approach is deliberative once the relevant values have been made explicit, but the deliberation process should be based on more consistent and transparent appraisals of criteria. Therefore, the process in Belgium should not stop at the point where it is at, and the scores calculated should be complemented with a deliberation process to depict potential additional considerations that shall be included. It was a common consideration that more research is still needed on how to deal with missing or low-quality evidence and also whenever there is need of deciding on early temporary coverage of products that have not yet obtained a marketing authorisation.

There was wide context variation; Colombia, for example, just recently incorporated HTA, and there is still need to upscale the use of MCDA during the appraisal stage of the decision-making process; thus, the pilot presented in this chapter is an illustration of the incipient efforts in this context. Belgium on the other hand incorporated robust evidence assessment for decision-making more than a decade ago, but it is the recent work of KCE which portraits the aims for a more explicit and legitimate process. The framework described in this case study is not yet in use, but it is expected to be implemented in 2016; hence although relevant, this could also be considered as a work in progress initiative.

It is the case of Lombardy, the one that probably represents the use of holistic MCDA in a more systematic and advanced stage at the moment, since it has been in place for over 3 years now. The pioneering work of the Lombardy region presented in this chapter is now being considered at the national level and for a broader focus than drugs and procedures. Of worth noting that each case study adopted a very different approach when attempting to merge MCDA and HTA for coverage decision-making, thus making fair comparison among them more complicated. Further comparative research on methods in the near future might be of value in assisting to identify which approach is the most appropriate.

4 Conclusions

All health systems face the challenge of managing finite resources to address unlimited demand for services; hence it is hoped that the content of this chapter could be of significant value to the field of public health and policy since non-explicit priority-setting processes, poor information, lack of policy on HTA, barriers to implementation, political agendas and limited resources are common findings in many countries (Youngkong et al. 2009).

It seems from these case studies and the growing interest on MCDA that structured and objective consideration of the factors that are both measurable and value based in an open and transparent manner may be feasible through the use of these frameworks. According to Miot et al. 2012, systematic and transparent approaches to priority setting are needed to produce decisions that are sound and acceptable to stakeholders. However, justifying advices towards the general public by making transparent what and how criteria is taken into account in the decision-making process is challenging but creates a societal ground for the decisions made; this is crucial for the continuing support of democratic systems with limited resources all around the globe.

The final results from the case studies may be applicable to wider contexts than Colombia, Lombardy and Belgium. MCDA can increase transparency and make value judgements explicit where before they remained implicit; this improves legitimacy and allows more consistency of results. Further research on merging MCDA and HTA to support better informed coverage decision-making, especially on methods, consistency and replicability of MCDA results, may be of value for all countries. Nonetheless, it is worth considering that values and decisions are expected to be dependent on committees’ stability and composition, as well as contexts and competing technologies of interest.