FormalPara Key Points for Decision Makers

• Structural uncertainty may have a significant impact on the outcome of cost-effectiveness models.

• There is an urgent need for guidelines on handling of structural uncertainty in cost-effectiveness analysis.

• Standardized disease-specific models in cost-effectiveness analysis should be developed to improve the consistency and relevance of health economic inferences.

1 Introduction

Decision making for reimbursement of new drugs is being increasingly supported by health economic analyses. In order to derive informed decisions, it is important to quantify the uncertainty associated with model predictions. Recently, recommendations have been published by the Modeling Task Force from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), describing good research practices in handling uncertainty [1]. The main sources of uncertainty include methodological, parameter and structural uncertainty [2].

Methodological uncertainty can be defined as differences between analysis methodologies. To decrease methodological uncertainty, implementation of a reference case has been recommended, which is a set of methodological practices intended to standardize economic evaluations to improve comparability [3].

Parameter uncertainty is related to the precision of model parameter estimation, which in turn depends on the informativeness of the data that are used and the complexity of the model that is being estimated. The impact of parameter uncertainty can be evaluated using stochastic simulations or sensitivity analysis [4, 5].

In this article, we consider structural uncertainty as uncertainty associated with all aspects of model structure, including health states and the specific relationships between these health states, but also the mathematical form of transition rates (e.g. constant, or time-varying according to a specific function). Of note, the specific parameters that are used for any mathematical expression can in turn be associated with parameter uncertainty.

The reasons for differences in model structure, and thus structural uncertainty, may be that some aspects of the process being modelled represent different levels of relevance, thereby justifying model simplifications. Alternatively, in some cases, some data may not be available although their inclusion could potentially still be relevant for the analysis. Structural uncertainty deals specifically with such assumptions or simplifications made in the model structure.

In contrast to methodological uncertainty and parameter uncertainty, structural uncertainty has only been addressed to a limited extent in current health economic guidelines [6–8], although it has been demonstrated that the impact of structural uncertainty on analysis outcome metrics can be of substantial magnitude. For instance, Bojke et al. [6] showed how structural uncertainty induced changes in outcome that could potentially impact reimbursement decisions. Kim and Thompson [9] showed that the impact of structural uncertainty on estimated incremental cost-effectiveness ratios (ICERs) could be of a similar magnitude to the impact of parameter uncertainty. Both examples illustrate the potential impact of structural uncertainty on public funding decisions, thereby justifying the relevance of more research and guidance in this area.

In the therapeutic area of oncology, small differences in overall survival are typically observed between competing treatments. Therefore, structural uncertainty could have a major impact on the outcome of cost-effectiveness analyses (CEAs) and associated decision making. Breast cancer is the most common malignancy in women worldwide [10], and many new drugs are currently in development for treatment of breast cancer. Health economic analyses are therefore of key importance to support selection and decision making with respect to reimbursement decisions on currently used and new therapeutic agents for breast cancer. Endocrine therapy plays a key role in treatment and management of hormone receptor-positive breast cancers [11]. A number of recent reviews [12–16] have identified up to 20 different CEAs comparing endocrine therapeutic strategies, most of which included either tamoxifen or anastrazole for treatment of hormone receptor-positive adjuvant breast cancer (ABC). However, none of these reviews specifically addressed the impact of structural uncertainty for CEAs comparing endocrine breast cancer treatments.

The objectives of the current analysis were (1) to identify differences in reported structural models and model parameterizations for cost-effectiveness analyses comparing tamoxifen and anastrazole; and (2) to evaluate and quantify the impact of identified differences in model components on analysis outcome metrics.

2 Methods

The analysis was performed in four steps:

  1. 1.

    Review of the literature for identification of eligible CEAs;

  2. 2.

    Definition and implementation of a base model structure, which included the apparent core properties present for all identified CEAs;

  3. 3.

    Definition and implementation of changes or additions in the base model structure or parameterization;

  4. 4.

    Quantification of the impact of differences in the model structure or parameterizations on the analysis outcome metrics life-years gained (LYG), incremental costs (IC) and ICER. Optional changes or additional model components that were identified are referred to as M1, M2, … Mn.

All models were implemented as systems of ordinary differential equations, using a previously developed scripting framework for CEAs [17] based on the statistical scripting language R (version 2.10.0) [18]. This framework allowed straightforward and reproducible implementation of different models and model components in order to allow for an unbiased evaluation of the impact of differences in model structure, fully excluding potential influences of other sources of uncertainty. In addition, this framework implements a modern multistep ordinary differential equation solver algorithm, which automatically adjusts the cycle length to adequate step sizes, thereby eliminating the need to specify the cycle length upfront and overcoming cycle length-induced bias [17].

2.1 Literature Review

Eligible CEAs compared anastrazole and tamoxifen for the treatment of early breast cancer and were implemented using Markov models or ordinary differential equation-based approaches. These CEAs were selected on the basis of a previously conducted review [16] investigating other methodological differences between CEAs of anastrazole and tamoxifen, unrelated to structural uncertainty. For each analysis, the structural model components were extracted from the publications. Subsequently, identified model components were categorized into two groups: (1) structural model characteristics, e.g. health states and associated transition rates; and (2) parameterization of transition rates. For Markov models reporting transition probabilities, these were converted into transition rate constants.

2.2 Definition and Implementation of the Base Model Structure

On the basis of the identified model structures, a base model was defined by including the health states that were present in all different published models, thereby representing the core model structure of health economic models for endocrine drug treatment of ABC. The base model was not necessarily intended as a recommendation but only as a reference point for alternative model structures.

Transition rate parameterizations for the base model were selected by using the mathematically simplest possible implementation as was described for the different identified CEAs. For instance, when a certain transition rate was included using a time-varying or a constant rate, the constant rate was included in the base model. The parameter estimates used for the base model were obtained from the most complete report with respect to the availability of parameter estimate values. The year of valuation was 2012, a 25-year time horizon was used with discount rates of 1.5 % for effects and 4 % for costs, and the cycle length varied over time, depending on the transition rate [17].

2.3 Identification and Implementation of Optional Model Components

For each identified CEA in step I, the model structure was compared with the base model and all differences in the model structure (i.e. health states, transitions and transition rate parameterization) were identified as optional model components.

2.4 Quantification of Differences Induced by Different Model Components Identified

To assess the impact of identified optional model components from step III on analysis outcome metrics, each component was evaluated separately and in a combined fashion. The outcome metrics included LYG, IC and ICER. In the separate analysis, we assessed their impact on outcome metrics by varying one model component at a time. In the combined analysis, model components were combined according to their implementation in each of the identified analyses. The impact of model components on outcome metrics was quantified by computing the relative difference from the base model (RDB) estimate as follows:

$$ {\text{RDB}} = \frac{{M_{\text{N}} - M_{\text{B}} }}{{M_{\text{B}} }} \times 100{\kern 1pt} \% $$

where M represents the outcome metrics for model N (e.g. M1, M2, …, Mn) or base model B.

3 Results

3.1 Literature Review

Eleven eligible publications assessing the cost-effectiveness of anastrazole versus tamoxifen [19–29] were included in this analysis. The identified differences related to structural uncertainty are provided in Table 1. All identified publications used the ATAC (Arimidex, Tamoxifen, Alone or in Combination) clinical trial [30] as a basis for implementation of recurrence rates.

Table 1 Overview of health states and adverse events as identified in previously published cost-effectiveness models comparing anastrazole with tamoxifen for adjuvant treatment of breast cancer
Full size table

3.2 Definition and Implementation of the Base Model Structure

3.2.1 Structural Model Characteristics

Health states present across all analyses and included in the base model were (1) on treatment; (2) off treatment; (3) local recurrence; (4) metastatic disease; (5) death due to breast cancer; and (6) death due to other causes. The resulting base model structure is schematically depicted in Fig. 1.

Fig. 1
figure 1

Schematic representation of the base model structure for health economic analysis of endocrine adjuvant breast cancer treatments. F loc fraction of local recurrence from both on treatment and off treatment, F met fraction of metastatic disease from both on treatment and off treatment, k DtO (t) background mortality rate, k Loc→Met metastasis rate following local recurrence, k Met→DtCa death rate after metastatic disease, k Off→Loc local recurrence rate from off treatment, k Off→Met metastatic rate from off treatment, k rec local recurrence rate

Full size image

The following transition rates were included in the base model: (1) incidence of local recurrence from both on treatment and off treatment (k rec  × F loc ); (2) incidence of metastatic disease from both on treatment and off treatment (k rec  × F met ); (3) rate of metastasis following local recurrence (k Loc→Met ); (4) death after metastatic disease (k Met→DtCa ); and (5) a (time-varying) background mortality (k DeathOther (t)) for patients in the health states on treatment, off treatment, local recurrence and metastatic disease. In addition, after 5 years of treatment, the proportion of women present in the on-treatment health state switched to the off-treatment health state according to the implementation in each of the identified models.

3.2.2 Parameterization of Transition Rates

Most commonly, transition rates were parameterized as constants and were implemented as such in the base model structure (Table 2). Only background mortality was implemented as a discretely time-varying constant changing every 5 years [25, 31].

Table 2 Transition rate constants used for the base model
Full size table

The publication by Mansel et al. [25] most transparently reported parameter values and costs, and it was therefore used as a template to obtain transition rates and costs. Because the rates of adverse events were not clearly stated in each of the identified articles, these were directly derived from the 5-year results of the ATAC trial [30].

3.3 Definition and Implementation of Optional Model Extensions

An overview of the identified models and the differences in structure and parameterization is provided in Table 3. In total, nine additions or changes in the model components were identified. Three components were related to the model structure: addition of health states (M1) and two additional transition possibilities between health states (M2 and M3). Six components were related to choice parameterization (M4–M9), which are provided in Table 4. Further details regarding the implementation of these options are provided in the following sections.

Table 3 Combinations of structural and parameterization differences in published articles and base model
Full size table
Table 4 Structural and parameterization differences and implemented rates
Full size table

3.3.1 Structural Model Characteristics: Metastatic Health States (M1)

Karnon et al. [22] described a CEA in which three metastatic health states were included instead of one. This was implemented by separating the metastatic disease health state into soft-tissue metastasis, bone metastasis and visceral metastasis. All different sites of metastatic disease are associated with different death rates—for instance, the chance of dying from visceral metastasis is higher than the death rate for soft-tissue metastasis. To implement the time-dependent death rates, six tunnel states for each metastatic health state were implemented [32]. Tunnel states were defined for each year from 1 to 5 years, and from 5 years onwards.

The fractions for recurrence used by Karnon et al. [22] were based on the BIG (Breast International Group) trial [33]. We implemented these alternative health states by using the fractions derived from the ATAC trial, because these fractions were used in all of the other analyses.

3.3.2 Structural Model Characteristics: Mortality Rates (M2 and M3)

Various authors included the death rate due to adverse events [21, 23–25, 28, 29] in their analyses. For M2, mortality rates for three life-threatening adverse events were included: hip fractures, endometrial cancer and thrombosis [22]. The population at risk was defined as the population on treatment experiencing the life-threatening adverse events.

For M3, an additional rate for breast cancer-related death after having a local recurrence was included, which was identified in three different publications [23–25].

3.3.3 Parameterization of Transition Rates: the Recurrence Rate (M4–M6)

Three model components (M4, M5 and M6) were identified to describe the recurrence rate. In all cases, some level of time dependency in the recurrence rate was used, as compared with the constant recurrence rate implemented in the base model.

In M4, a discretely time-varying parameter with an interval of 1 year in the first 10 years was implemented instead of a constant recurrence rate [27].

In M5, a discretely time-dependent parameter was included, varying the recurrence rate after 5 and 10 years from the start of therapy [22].

In M6, a continuous time-dependent relationship was implemented using a Weibull equation (Eq. 1) to describe the recurrence rate [23–25], where I represents the intercept and S represents the scale factor.

$$ k_{\text{rec}} (t) = \left\{ {\begin{array}{*{20}c} {t < 10\,{\text{year}}} & {\exp \left( { - \frac{{S_{1} }}{{I_{1} }}} \right) \cdot \frac{1}{{I_{1} }} \cdot t^{{\frac{1}{{I_{1} }} - 1}} } \\ {t \ge 10\,{\text{year}}} & {\exp \left( { - \frac{{S_{2} }}{{I_{2} }}} \right) \cdot \frac{1}{{I_{2} }} \cdot t^{{\frac{1}{{I_{2} }} - 1}} } \\ \end{array} } \right. $$
(1)

3.3.4 Parameterization of Transition Rates: Death Rate After Metastatic Disease (M7)

In M7, death rates after metastatic disease were implemented using tunnel states. Metastatic disease and the time previously spent in this state were defined by using the following series of six tunnel states with corresponding death rates: 0–1 years, 1–2 years, 2–3 years, 3–4 years, 4–5 years, and more than 5 years in metastatic disease [19].

3.3.5 Parameterization of Transition Rates: Metastatic Rates (M8–M9)

In component M8, the rate of having metastatic disease after local recurrence varied according to whether a patient was on therapy. Different rates were used for the first 5 years and after 5 years of therapy [26].

For component M9, time-dependent metastatic rates were included by using tunnel states for the first 5 years after having a local recurrence and for years 6–15 after having a local recurrence [22].

3.4 Quantification of Differences Induced by Different Model Components

The base model showed average incremental costs per patient of €3,647 for anastrazole compared with tamoxifen and a 0.263 incremental LYG, leading to an ICER of €13,868. The results of the analyses based on the different components are presented in Table 5.

Table 5 Effects of separate individual model components on incremental outcome metrics in terms of life-years gained (LYG), incremental costs (IC) and the incremental cost-effectiveness ratio (ICER) for anastrazole versus tamoxifen in terms of absolute values and as the relative difference compared with the base model (RDB)
Full size table

3.4.1 Structural Model Characteristics

Inclusion of additional metastatic health states (M1) resulted in a 10.0 % increase in the LYG and a 7.3 % decrease in the ICER.

Inclusion of mortality due to life-threatening adverse events (M2) resulted in a very small decrease in the LYG (0.03 %) and almost no change in the ICER.

Inclusion of death rates after local recurrence (M3) resulted in an increase of 21.7 % in the LYG and a consequent decrease of 16.8 % in the ICER.

3.4.2 Parameterization of Transition Rates

A discrete 1-year interval time-dependent rate of recurrence in the first 10 years (M4) was implemented, resulting in large differences in the LYG (+23.2 %) and ICER (−21.1 %).

A discrete 5-year interval time-dependent rate of recurrence (M5) caused the largest difference in the LYG (+35.4 %) and consequently the ICER (−29.3 %).

A continuous time-dependent recurrence rate parameterized using a Weibull equation (M6) demonstrated a decrease in the ICER of 12.3 %, which was due to the increase in the LYG of 0.032 (12.2 %).

Analyses with the alternative component M7, decrease in the death rate after year of onset metastasis, resulted in small changes in the LYG (<7.0 %) and ICER (<7.0 %).

Inclusion of time-dependent rates of metastatic disease following local recurrence (M8, with rates depending on the time spent in therapy; and M9, with rates depending on the time spent in local recurrence) resulted in large differences in the LYG (−21.3 % for M8 and −21.7 % for M9).

3.4.3 Comparison Between Overall Published Models

The impact of the implementation of combinations of components as presented in the published models (Table 3) is presented in Table 6. Combining components M2, M3, M6 and M9 (as reported in references [23–25]) resulted in a 33.9 % increase in the LYG, ultimately leading to a 23.7 % decrease in the ICER. A combination of M7 and M9, as reported by Fonseca et al. [19], resulted in a 13.3 % decrease in the LYG and consequently a 18.0 % increase in the ICER. Only incorporation of component M8, as reported by Moeremans et al. [26], resulted in a 0.207 incremental LYG, corresponding to a decrease in the LYG of 21.3 % and an increase in the ICER of 28.0 %. Inclusion of component M4 for the incidence of recurrence and component M7 following a distant recurrence, as reported by Rocchi et al. [27], resulted in a 31.6 % increase in the LYG and consequently a decrease in the ICER of 25.9 % to €10,278. A combination of components M1, M2, M5, M7 and M9, as reported by Karnon et al. [22], resulted in an increase in the LYG of 45.6 % to 0.383 and the largest decrease in the ICER of 30.2 % to €9,683.

Table 6 Effects of combined model components identified in the previously published cost-effectiveness analyses (CEAs) on incremental outcome metrics in terms of life-years gained (LYG), incremental costs (IC) and the incremental cost-effectiveness ratio (ICER) for anastrazole versus tamoxifen in terms of absolute values and as the relative difference compared with the base model (RDB)
Full size table

4 Discussion

A wide variation in the choice of model characteristics of CEAs comparing anastrazole and tamoxifen in early breast cancer was identified, which were associated with ICERs varying between €9,804 and €17,966 when assessing the univariate impact, and ICERs between €9,684 and €17,744 when considering the multivariate estimates as implemented in the previously identified analyses. The range of ICERs that were identified did not have direct implications for the reimbursement status, when considering the threshold of £30,000 (approximately €36,000) used by the National Institute of Health and Care Excellence (NICE) [34]. These relatively low ICERs are related to the relatively low incremental costs between both therapies. Nonetheless, the observed differences in the LYG for the multivariate estimates (0.207–0.383 years) could indeed become relevant for decision making when higher treatment costs are involved, which is a realistic scenario, considering the rising intrinsic costs of new therapeutic agents in oncology.

4.1 Model Characteristics

Ultimately, health economic model characteristics should be biologically and clinically plausible. However, several of the identified model assumptions did not adequately reflect disease progression. We now discuss the specific properties that were identified in the different CEAs.

Metastasis ultimately leads to death, and hence describing the processes of metastasis is of key relevance to capture the dynamics of disease progression. It has been established that metastasis of breast cancer occurs in different parts of the body, with variable and time-dependent death rates [35–40]. Therefore, the use of various metastatic sites and time-dependent death rates is an important consideration for description of disease progression, instead of single metastatic health states and constant rates, which were implemented in several of the identified models.

Various clinical trials have demonstrated that the majority of recurrences in early breast cancer occur in the first 2 years after diagnosis [30, 33, 41] while hormone receptor-positive tumour relapses can occur even after a period of 10 years from the end of treatment [42, 43]. When considering models to describe recurrence, a constant recurrence rate (base model), the reported discretely time-dependent rates with a 5-year interval and the reported Weibull model did not specifically account for these characteristics, whereas the 1-year interval time-dependent rate constant did include this property.

The rate of having metastatic disease after experiencing a first local recurrence was demonstrated to be time dependent in several studies [30, 44–46]. Therefore, inclusion of time-dependent parameterization after having a local recurrence resembles natural disease progression best.

Ultimate comparisons of efficacy are based on survival, and therefore death rates are another important characteristic that needs to be carefully considered. Although inclusion of time-dependent death rates, which reflect a decrease in the death rate after the first year of metastasis (M7) only has a limited impact (a maximum relative deviation of 6.84 % in the LYG), various reports have demonstrated that patients have an increased risk of death in the first years after metastasis, thereby supporting the clinical relevance of implementing time-dependent death rates [44, 47].

The use of mortality due to adverse events is scientifically well supported—for instance, after hip fractures [48]—but its impact on the analysis outcome was shown to be of limited magnitude in our analysis. Nonetheless, we do consider adverse event-related mortality as a relevant component to include in future CEAs.

Overall, from the difference in outcome metrics from Tables 5 and 6, it becomes clear that specifically the choice of the functional form (e.g. time-varying or constant) for rate constants is an important factor for ultimate differences in the outcome metrics that are observed.

4.2 Disease-Specific Models

The aforementioned differences in model assumptions, combined with the substantial impact on outcome metrics, clearly illustrate the importance of the implementation of a standardized disease-specific model for comparison of endocrine treatments in early breast cancer. The concept of disease-specific models, or disease-specific reference cases, has been recently outlined by Afzali et al. [49]. Implementation of standardized disease-specific models with adequate reflection of the underlying disease can reduce the magnitude of variation between analyses. This is especially relevant within oncology, given the typically small differences in efficacy and high treatment costs, which translate into a potentially large impact of structural uncertainty. Such disease-specific guidances have been already implemented in other disease areas such as rheumatology [50, 51] and osteoporosis [52]. In the field of quantitative pharmacological analysis, including oncology, disease-specific models [53] and system-specific models [54, 55] have been implemented and used to support development.

On the basis of the evaluation of structural model components and their impact on outcome metrics, we can identify a number of scientifically well supported components related to structural and parameterization components in CEAs, which significantly affect CEA outcome measures. We therefore suggest the following conceptual model characteristics for a standardized model for comparison of endocrine breast cancer treatments: (1) time dependency of recurrence; (2) inclusion of time dependency of having metastatic disease after experiencing a local recurrence; (3) inclusion of soft-tissue, bone and visceral metastasis health states in addition to disease-free, local recurrence, death due to breast cancer and death due to other causes health states; and (4) inclusion of time dependency of death after recurrence. An overview of these properties is provided in Fig. 2. Furthermore, time dependency is still frequently reported as a series of empirical discretized values, potentially leading to a suboptimal description of time-dependent rate constants. We therefore recommend the use of continuous functions to more accurately describe such changes.

Fig. 2
figure 2

Proposal for a standardized cost-effectiveness model for endocrine treatment of adjuvant breast cancer based on adequate reflection of disease progression. F bone fraction of recurrences being bone metastasis, F loc fraction of local recurrence from both on treatment and off treatment, F met fraction of metastatic disease from both on treatment and off treatment, F soft fraction of recurrence being soft-tissue metastasis, F vis fraction of recurrences being visceral metastasis, k Bone→DtCa death rate after bone metastasis, k DtO (t) background mortality rate, k Loc→Met metastasis rate following local recurrence, k Off→Loc local recurrence rate from off treatment, k Off→Met metastatic rate from off treatment, k rec local recurrence rate, k Soft→DtCa death rate after soft-tissue metastasis, k Vis→DtCa death rate after visceral metastasis

Full size image

4.3 Guidance on Structural Uncertainty

Standardization practices, such as development of disease-specific models, will be constantly subject to change, as the understanding of biological and clinical properties of cancer disease progression is constantly developing and should be incorporated into disease-specific models. Therefore, structural uncertainty can never be fully minimized by means of standardization practices only, and its impact should be appropriately considered when conducting CEAs. However, currently, no explicit and clear guidance regarding inclusion of structural uncertainty has been provided—for instance, by national reimbursement bodies such as NICE in the UK [3] and the Pharmaceutical Benefits Advisory Committee in Australia [56]. The lack of such guidance creates potential opportunities for introducing bias that may allow for ‘optimized’ favourable outcomes [57, 58], thereby supporting the importance of developing guidance on evaluation of structural uncertainty.

4.4 Strengths and Limitations of this Analysis

Quantitative comparisons between methodological approaches in health economic analyses are often difficult to perform, because of the different sources of uncertainty that affect outcome metrics. In the current analysis, we carefully reviewed all reported analyses and subsequently re-implemented the identified model components, allowing for a relatively unbiased comparison of the impact of different structural model components on outcome metrics. We consider this approach useful for assessment of the impact of structural uncertainty in other areas of CEA as well.

We were not able to retrieve some of the model assumptions in a limited number of cases, as indicated in the Methods section. This finding supports the need for increased transparency and reproducibility in the reporting of CEAs. The use of a scripting-based frameworks warrants substantially improved reproducibility and transparency, allowing straightforward evaluation and external review, and should therefore be considered for implementation in guidelines related to handling of structural uncertainty.

A consequence of not being able to retrieve a limited number of model assumptions was, however, that our analysis outcome metrics did not exactly match the original estimates—although, in our view, this was not of relevance to our objectives, results or conclusions.

We did specifically choose not to include other types of uncertainty (i.e. parameter and methodological uncertainty). For the current analysis, the impact of methodological uncertainty can be disregarded, because all of the evaluated models were compared in the same computational framework. With respect to parameter uncertainty, it can be expected that this would have potentially inflated all of the ranges in outcome metrics further, but it would have also substantially clouded the specific evaluation of the impact of structural uncertainty. One could, however, imagine a case where some potentially more complex structural model components may be associated with increased parameter uncertainty, compared with simpler model structures.

5 Conclusion

A systematic review of structural model properties for CEAs comparing endocrine treatments for early breast cancer was performed, and the associated impact of differences in model structure and parameterization indicated a substantial impact on outcome metrics. The wide variation in the model structures that were identified supports the need for (1) improved guidance on the handling implications of structural uncertainty; and (2) the need for a standardized disease-specific model for CEA of endocrine treatments in early breast cancer. On the basis of this analysis, we have provided recommendations for a disease-specific model for endocrine treatment comparison in ABC.