Introduction

Worldwide, healthcare funders are increasingly using economic evaluations to inform their decisions related to health care and the adoption of new technologies. There are many multi-attribute utility-based instruments (MAUI) for the assessment of Quality of Life (QoL), such as the Health Utilities Index (HUI) 3, the Finnish 15D, the SF6D and the EQ-5D, the most widely used MAUI in published cost-utility analyses [1, 2]. The National Institute for Health and Care Excellence (NICE) from England and Wales states that health effects in cost-effectiveness analyses should be expressed in quality-adjusted life years (QALYs), utilities should be based on public preferences and use of EQ-5D is recommend [3].

The EQ-5D is a generic instrument commonly used to measure patient-reported QoL. In order to help inform decision-making in economic evaluations, it is used to assign a preference value to the amount of time living on the reported health status. Many countries, some from Latin America (LA), have derived population value sets for the EQ-5D [4–7].

The classic version of the EQ-5D comprises five dimensions with three severity levels and a visual analogue scale (EQ–VAS) [8]. Dimensions are mobility, self-care, usual activities, pain/discomfort and anxiety/depression. Recently, the EuroQol Group developed a new version of the instrument with the same dimensions but five levels of response instead of the original three in the classic version, recently recalled EQ-5D-3L. The new labels for the response levels range from no problems to unable/extreme problems with three intermediate levels (slight, moderate and severe problems). This new version, called EQ-5D-5L, describes 3125 possible health states, and it was designed to improve the measurement properties of the EQ-5D-3L, reducing the ceiling effect and increasing the discriminatory power [9]. Recent studies confirm the higher discriminatory power and lower ceiling effect for the 5L version as compared to the 3L [10–13]. A five-digit number (one per dimension, on the same order than in the instrument) usually codifies the EQ-5 D health states; this code is usually called state profile. Each digit varies from 1 to 5 representing no problems level to unable/extreme problems level, respectively. For example, the worst state (“pits”) is represented by 55555 while being unable to walk but with no problems in the rest of dimensions is represented by 51111. The misery index is a proxy measure of the severity of the state. It is defined as the addition of the five digits of its profile, for example the state 23221 has a misery index equal to 10. The EuroQol have recently developed an EQ-5D-5L valuation study protocol aiming to harmonize 5L valuation studies around the world [14]. To implement and facilitate this standard protocol, the EuroQol group developed specific software, the EuroQol Valuation Technology (EQ-VT) [15].

Uruguay is located in the southern cone of South America and according to the last census has 3,286,314 inhabitants [16]. By the end of 2007, Uruguay started a healthcare reform, encompassing healthcare delivery, financing and management, establishing an Integrated National Health System with the objective of improving the quality, access and efficiency of healthcare services. Since the reform, the Ministry of Health regularly uses Health Technology Assessment (HTA) and economic evaluations in decision-making process involving the financial coverage and provision of high-cost technologies for the whole population, through the National Resource Fund (Fondo Nacional de Recursos) [17, 18]. Uruguay had no previous value set, and it is the first country in LA to undertake a general population valuation study for the EQ-5D-5L instrument. The objectives of our study were to obtain social preferences and derive the value set from Uruguayan general population using the EQ-5D-5L questionnaire, as well as to report the population norms.

Methods

Protocol

We used a standardized interview protocol developed by the EuroQol group based on the obtained evidence from a set of conducted pilot studies [19]. The protocol consists in five different sections: (1) a general welcome, (2) introduction to the research and completion of background information, (3) the set of composite time trade-off (C-TTO) tasks, (4) the set of discrete choice (DC) experiment tasks and (5) general thank you and goodbye. We also collected socio-demographic information and health literacy measurement using the Short Assessment of Health Literacy-Spanish questionnaire (SAHL-S).

Eliciting preferences methods

Traditionally, EQ-5D-3L valuation studies were mainly based on time trade-off (TTO) methods [4]. However, the TTO version used in the “Measurement and Value of Health” protocol [20] had some problems, especially with the transformation of values from states considered to be worse than dead [21, 22]. Lead-Time TTO, Lag-Time TTO [23] and C-TTO [24], more recent TTO versions, were tested on several pilot studies. Based on the results of those studies, the EuroQol Group concluded that the EQ-5D-5L protocol should include the C-TTO version [14].

The conventional TTO approach has two different tasks, one for states considered to be better than death (BTD) and another one for states considered to be worse than death (WTD). For the valuation of states BTD, respondents are required to choose between living 10 years in a specific health state (life B) or X years in full health (life A). The amount of time X in life A is varied between 0 and 10 years. For states WTD, the procedure is conceptually and operationally different, and participants have to choose between dying immediately or live X years in a specific state followed by 10–X years in full health (life B).

Lead-time TTO and lag-time TTO methods add extra time in full health to the health state to be valued (Life B). The lead-time approach used in this protocol places the extra time before, being the options living 10 years in full health and later 10 years in the given health state or X amount of time (between 0 and 20 years) in full health in life A. The main characteristic of these variants is that the iterative trading process allows the participant to move between negative and positive values without explicitly thinking about whether the state is worse or better than being dead.

C-TTO involves the use of traditional TTO approach for states considered to be BTD, and a lead-time TTO for states considered to be WTD, combined in a unique task. So, when the participant exhausts the 10 years in full health in the traditional TTO task and does not want to spend any time in full health as the evaluated state is very bad (WTD) he is switched to the lead-time TTO component.

The interview protocol included 86 EQ-5D-5L health states (selected using Monte Carlo simulation [14] ) to be evaluated using C-TTO divided in ten blocks with similar representation of all severity levels. All the blocks included one very mild state with only one health state with mild problems (i.e. 21111) and the pits state (55555). Respondents were randomly assigned to one of the ten C-TTO blocks. The presentation order of the states within each block was also randomly generated by the EQ-VT.

The protocol included a DC experiment as a secondary valuation technique [14]. A DC experiment is an ordinal elicitation technique that has received recent attention for eliciting EQ-5D-5L values [25, 26]. DC experiments require individuals to make a pairwise comparison between two different scenarios, being in our case two EQ-5D-5L health states. The protocol included 196 DC pairs of EQ-5D-5L health states divided in 28 blocks of seven pairs with similar misery index. Respondents were randomly assigned to one DC block, and the order of each pair and its position on the screen (i.e. left or right) were also randomly generated by the EQ-VT.

Quality control

The initial EQ-5D-5L valuation studies found some interviewer effects that could affect data quality, as it is reported by Ramos-Goñi et al. [27]. For further valuation studies, the EuroQol Group decided to create a quality control tool to monitor interviewer performance. This tool mainly evaluates interviewer protocol compliance through four key parameters of the C-TTO task: the full explanation and the time spent on the wheel chair example, the time used to complete the 10 C-TTO tasks and the presence of large inconsistencies. The DC experiments section was monitored to detect unusual response patterns (i.e. AAAAAAA, ABABABA). Interview quality was checked weekly. Based on weekly results, we decided whether we had to retrain or drop interviewers from the team. After quality control analysis, researchers (FA, UG and LRA) gave feedback and retrained interviewers when necessary.

Sampling and data collection

Uruguay was geographically stratified. The study took place in the following locations: Uruguay capital city, Montevideo; and the departments of Maldonado and Paysandú. Quotas by location, gender, age and socio-economic status replicated the Uruguayan population structure [16].

The EuroQol Group recommends including 10000 C-TTO responses in the valuation studies. Since each participant values 10 health states, the initial sample size was 1000 individuals. The power calculation was based on precision requirements for the estimation of the C-TTO mean [27].

Twenty-one trained interviewers administered the questionnaire using the EQ-VT. During the face-to-face interviews, respondents had the control of the computer most of the time and the interviewers were available to assist and monitor the process.

The valuation exercise started explaining the objectives of the research, then the respondents filled out the EQ-5D-5L and rated their current health state using the EQ–VAS. Additionally, they gave background information (age, sex, educational level and their experience with illness). Prior to completion of the C-TTO tasks, participants received an explanation (using as an example a life living in wheel chair) and completed three mock states of different severity in order to verify their understanding. Later on, they completed 10 C-TTO and seven DC experiments. Upon completion of the tasks, participants answered follow-up questions related to the difficulty and comprehension.

Once both tasks and the follow-up questions were completed, the interviewers asked the respondents to complete the Short Assessment of Health Literacy–Spanish (SAHL-S) instrument [28]. This questionnaire evaluates health literacy through 18 multiple-choice questions combining word recognition and comprehension. Low health literacy is defined by identifying 14 or fewer correct items. Respondents were asked to read aloud 18 medical terms (word recognition), and the interviewer assessed comprehension through the multiple-choice question. Health literacy is a construct that reflects the capacity to obtain, process and understand health information and services needed to make appropriate health decisions [29].

Statistical analysis

We describe the sample characteristics using means and standard deviations for continuous variables and percentages for discrete variables. The EQ-5D-5L descriptive system from the recruited sample is presented by age group.

Valuation data from C-TTO tasks and data from DC experiments were available from the collected data. We initially tried to follow the hybrid approach reported by Ramos-Goñi et al. [27]. However, in our case, the DC models had several logical inconsistencies, leading us to base our analysis on the C-TTO data only. The DC and hybrid analyses are available from the authors upon request.

We started the C-TTO analysis using the classic ordinary least square (OLS) model. However, our data had problems that prevented us from using this approach: heteroscedasticity and significant outliers. Thus, we opted for using robust regression [30]. This regression method basically applies a different weight to each observation based on how far away it is located from the median of the population sample. In this way, the impact of the outliers is reduced and the heteroscedasticity problem is addressed. We used a tuning constant based on the bi-weight function to calculate each weight in the model. A tuning constant of 7 is usually used, in order to confer similar model efficiency as the OLS model, and assuming no heteroscedasticity or outliers [31]. We set this value to 8.5 to include a broader range of values for each respondent, without losing the logical consistency of the model. We analysed the distribution of weights according to the misery index value in order to explore the contribution of different groups of observations to the final model estimations.

For the model specification, we started with a 20-parameter main effects model, using the response values as dependent variables and health states as explanatory variables. We created a dummy variable Dij indicating whether the dimension i is at level j. For example, we created variables MO2, MO3, MO4 and MO5 for mobility dimension, indicating whether the mobility dimension is at level 2 or 3 or 4 or 5, respectively. Similar sets of variables were created for each dimension. In order to explore alternative model specifications and performance, we added interaction terms to the main effects model. We evaluated traditional Nj terms (1 if at least one dimension is at level j) and Nij terms (1 if at least one dimension is at level i or j). We also tested the following interactions terms: (1) D1, number of movements away from full health beyond the first; (2) I2, number of dimensions at level 2 or 3 beyond the first; (3) C3, number of dimensions at level 3, 4 or 5 beyond the first; (4) K45, number of dimensions at level 4 or 5; (5) I45, number of dimensions at level 4 or 5 beyond the first; (6) O2, 1 if all dimensions are at level 1 or 2; (7) Z2, 1 if at least one dimension is at level 2 or 3 and one is at level 4 or 5; and (8) Z3, number of dimensions at level 2 or 3 given that at least one dimension was at level 4 or 5. We used a stepwise approach to decide whether to keep the interaction terms in the model or leave them out.

In this manuscript, we present three models: (1) the OLS model for comparison purpose, (2) the main effects robust model and (3) the robust model including best interaction terms. All statistical analyses were performed on STATA 11 MP [26], using the “regress” command for OLS and “rreg” for the robust regression.

Expanding the modelling exercise, we have performed additional analyses to check differences on preference values by educational level. We have performed an ANOVA test for crude preferences and we have also added dummies by levels of education on the final estimation of the model. None of these results were statistically significant (data not shown).

Exclusion criteria

Interviewers with low-quality performance, fulfilling pre-specified criteria regarding interview quality (i.e. too little time spent explaining the task, no explanation of the lead time section, C-TTO responses with clear inconsistencies or too little time to perform all TTO tasks) were excluded during the data collection process. We also excluded respondents meeting two additional criteria: (1) having a positive slope on the relationship between their values and the misery index of the health states. This means, that the respondent poorly understood the task, as he/she provided higher utility values for worse health states; and (2) respondents who valued all states at the same value, except non-traders (i.e. subjects who value all states as 1).

Model performance

We used four criteria to evaluate the performance of the model: (1) logical consistency of parameters, (2) goodness of fit, (3) prediction accuracy and (4) parsimony. A set of model parameters is said to be logically consistent if predictions for logically better health states (ex: 12111 is logically better than 13111) are higher than the predictions for logically worse health states. In our models, it means that MO2 < MO3 < MO4 < MO5, and so on for SC, UA, AD and PD dimensions. We used Akaike information criterion (AIC) and Bayesian information criterion (BIC) to evaluate goodness of fit, adjusted by the number of model parameters. We calculated the mean square error (MSE) and the mean absolute error (MAE) to evaluate prediction accuracy. The principle of parsimony stated that when two competing models are similar in terms of performance parameters, the simplest model should be selected. These four criteria were used to compare different model specifications using different interaction terms.

Comparison between predicted values from different models and to the Argentinian crosswalk 5L value set

In order to compare predictions form different models, we calculated the estimated values from robust models and compared them with the weighted means of the 86 TTO health states included. Those predictions were also compared to the ones from the crosswalk 5L value set from Argentina, (the 3L value set that uses a mapping function between the 3L and the 5L versions). Because of the EQ-5D-5L crosswalk value set for Argentina has not been previously estimated [32], it was calculated specifically for this study following the methodology proposed by van Hout et al. [33].

Results

Study recruitment took place between October 2013 and June 2014. We started the field work with 21 interviewers, who were trained, and evaluated on a weekly basis at the beginning of the data collection phase. Based on the quality control analysis, we decided to keep 11 interviewers with good performance, which resulted in the exclusion of 220 interviews conducted at this stage by 10 poor-quality interviewers. Excluded participants had similar age, gender and educational level than the remaining sample. We periodically analysed the remaining interviews during data collection and decided to stop data collection when our analysis showed the robustness of the results. The study sample had 805 respondents between 20 and 83 years old. Eleven subjects met exclusion criteria, leaving 794 subjects in the final sample. Sample characteristics were similar to the Uruguayan population in terms of gender. However, younger as well as higher educated categories were slightly over-represented in our sample, though utility values did not significantly differ by educational level. Nearly 44 % of the population had low health literacy despite the fact that more than 80 % had educational attainment of at least some secondary level (see Table 1).

Table 1 Study sample and Uruguayan general population characteristics
Full size table

Forty-four per cent of the sample reported no problems on any dimension of self-reported EQ-5D-5L. Older respondents reported more problems in all dimensions, and the mean self-reported VAS also decreased with increasing age, and was smaller in women (Table 2).

Table 2 Self-reported health using EQ-5D-5L descriptive system and EQ VAS
Full size table

The results from the OLS model showed a logical inconsistency between the coefficients associated with slight and moderate problem in usual activities dimension. This inconsistency was not observed when using the same model specification with a robust regression approach. Both robust model estimations reported that the main effects and the main effects with interactions were logically consistent. The goodness of fit of both robust models was similar, gaining only 0.4 % of relative improvement in AIC or BIC with the interaction terms. Similarly, the prediction accuracy of the main effect robust model was similar to the main effect model with interaction (MSE = 0.002) (Table 3). Based on the similarities of these parameters and taking into account the parsimony criterion, we chose the robust estimation of the main effects model as the most appropriate model for the Uruguayan value set. The technical appendix shows how to estimate an individual state value, as well as the code to do it in Stata. In the electronic supplementary material, we report the utility values for the 3125 different states.

Table 3 Model estimations: comparison of OLS, main effect robust model and main effect with interaction terms
Full size table

The weights applied to the responses in the robust model range from 0 to 1. The distribution of weights, according to different values of the misery index, showed how the respondents’ opinions have more discrepancies as the misery index increases (Table 4). For example, for a misery index = 6, the 10th percentile is 0.992, meaning that 90 % of the responses have almost the same impact in the model estimations (i.e. weight close to 1). However, the 10th percentile for those states with a misery index = 21 is 0.449, meaning that at least 10 % of the responses for these states will be considered as half important in the model estimations (i.e. weight close to 0.5). The selected model predicts index values that range from −0.264 to 1. States with a misery index = 6 had a mean predicted value of 0.965 (Table 4). The predictions from the model with interactions are slightly lower at the top and bottom of the scale than predictions form the main effects model, i.e. it has a lowest prediction of −0.288 for the pits state and has a mean prediction for misery index 6 states of 0.95. When comparing the prediction for a misery index = 6 in the Uruguayan population with the Argentinian EQ-5D-5L crosswalk value set, the Uruguay values are about 0.05 higher, and they are also higher than the Argentinian values across the misery index spectrum.

Table 4 Model predictions comparison, and with the crosswalk 3L Argentinean value set
Full size table

The mean index values, using the selected Uruguayan EQ-5D-5L value set, for the general population in Uruguay is 0.954. Older people have worse health-related quality of life for all paired comparisons (highest P value < 0.001). Males had slightly higher values than females (Table 5), but this difference was not significant.

Table 5 Population norms for EQ-5D-5L in Uruguay
Full size table

Discussion

The EQ-5D-5L is a recently developed instrument. Only a few countries conducted valuation studies, and only the Spanish study was recently published, though authors did not recommend the use of the reported value set [27]. This is to our knowledge the first study that provides a population-based EQ-5D-5L value set in Latin America.

The choice of a robust estimation for modelling the C-TTO data is also a novel approach in EQ-5D valuation exercises. While other studies present models mainly based on OLS or random effects estimations [4], we selected a robust estimation based on the observed between-respondent variability. Some extreme differences in opinions in our sample made some of the OLS or random effects coefficients to be logically inconsistent, due to the significant heteroscedasticity and the presence of outliers. Robust regression tries to solve these issues by weighting opinions less strongly if they are extreme. Extreme opinions have less impact depending on how extreme they are, and the values close to the majority (median) have the greatest weight. In our estimation, we relaxed the robust condition of the estimation as much as possible, stopping when inconsistencies in coefficients were found. We included all responses from the sample, as there is no 0 weight for any response. This was a balance point we chose in order to have logical results but also to incorporate everybody’s opinions.

Based on the parsimony criteria, we think that a more complex model should be preferred only when the improvement compared to a simpler one is large enough to overcome the complexity. In our estimation, the improvement of the best model with interaction terms tested was marginal compared to the main effects model. That led us to prefer the main effects model instead of the model with the interaction terms.

Given the fact that there is not any EQ-5D-5L value set currently recommended in the literature, and no previous EQ-5D-3L value set was available for Uruguay, we decided to compare our results with the EQ-5D-5L crosswalk value set derived from the original 3L set for Argentina [5], being a close country with similar socio-economic characteristics. In our selected model, we had slightly higher values in Uruguay in the entire severity spectrum. Taking into account the changes on the descriptive system of the 3L and 5L versions of EQ-5D, [9] it was something expected. For example, the levels for the misery index 6 states on the 5L version are by definition less severe (“slight”) compared to the same levels in the 3L version (“some”). Also, the higher observed index value for the pits state can be explained, as the level for the mobility dimension has changed from “confined to bed” in the 3L version, to “unable to walk” in the 5L version, making the description of the pits state (55555) in the 5L version better than the corresponding state (33333) in the 3L version. As both anchor values have been moving up in our estimations, it is expected that the whole scale move up according these anchors. Population norms derived for Uruguay showed to be consistent and similar to international population norms previously published [34].

One limitation of this study is the use of a quota (i.e. non-probabilistic) sample. Though our sample was broadly representative of the socio-demographic characteristics of the Uruguayan population, younger and higher educated individuals were slightly over-represented. However, the age difference was small. Additionally, although the proportion of participants with tertiary education was slightly higher in the sample compared with national data, utility values did not significantly differ according to educational level. Another limitation is the fact that we have not used the information from the DC experiment. The recently published study from Spain reported the feasibility of obtaining an EQ-5D-5L value set using a hybrid approach, combining the C-TTO and DC data [27]. However, in our initial analysis, when we tested our DC models, they showed several inconsistencies, and these could not be solved through a hybrid modelling approach. In addition, we have seen in our C-TTO responses some extreme differences in opinions, mainly regarding severe health states. This fact could also explain the inconsistencies found in the DC models. As far as we are aware, there is no available “robust” estimation method for analysing DC data, limiting our capacity to include this information in our estimations. Another limitation of this study is the fact that we have not performed an internal or an external validation of our predictions. Given the final sample size of our study, which was somewhat smaller than originally intended, and that the requirements for internal validation reduce statistical power (i.e. randomly splitting the sample and evaluate how the model derived in the estimation set applies to the validation set), we included all responses in the model estimation.

We obtained the EQ-5D-5L value set that will be implemented in Uruguay, which is the first country in Latin America to undertake such a study. The use of these values will help researchers, in Uruguay and eventually in other similar socio-economic countries, in conducting cost-utility studies based on the specific preferences of the general population to inform decision makers’ resource allocation decisions.