Introduction

Informal care has an important role in a patient’s health and quality of life (QoL). The involvement of unpaid carers in the provision of care is expected to further increase [1], highlighting the importance of considering in economic evaluations the costs and benefits attached to informal care. Evidence suggests that only a few studies have considered the value of costs [2] and benefits [3] of unpaid carers in economic evaluations [4]. A further consideration is that most existing outcome measures are either not focused on carers’ concerns (in the case of preference-based measures of health status) [5] or not preference-based (‘sum score’ measures of carer outcome) [6], and therefore, impacts on carers may not be appropriately captured or valued in economic evaluations. These limitations of conventional instruments highlight the need for preference-based instruments for measuring the caring experience [7, 8].

The Carer Experience Scale (CES) was developed using meta-ethnography and semi-structured interviews with carers with the aim of providing a direct assessment of care-related welfare that could be incorporated in economic evaluations [6]. The CES consists of six conceptual attributes: activities outside caring, support from family and friends (social support), assistance from the government and other organisations (institutional support), fulfilment from caring, control over caring and getting on with the care recipient. Each attribute has three levels ranging from few/little/rarely to some (times) and most/a lot. Although there is some overlap with attributes used in other measures, the CES incorporates the attribute of control which is not included in other measures and appears to be of higher importance in certain carer groups [6]. The valuation of CES was conducted using best–worst scaling to identify index values that would assist the incorporation of the caring experience into economic evaluations [7].

Although content validation was built into the development of the CES [6], other psychometric properties of the measure need to be established. Validity is an important psychometric property that reflects the extent to which instruments measure the theoretical concept they are supposed to measure, determining the level of confidence that can be placed on the results generated. Given that no ‘gold-standard’ measure of carer-related QoL exists, criterion validation of the CES is not possible. Furthermore, assessment of predictive validity is less relevant in this context, as the CES is an outcome measure and not intended as a clinical tool to identify carers at risk of adverse events in the future. Instruments exhibit good construct validity when observations about the existence or not of a relationship between a measure and other factors are in line with those expected a priori. This paper conducts a series of investigations to examine the construct validity of the CES, focusing on associations between the responses to the CES and a number of different carer, care recipient, caring situation and motivation-related variables. This is an initial study examining the construct validity of the CES.

Methods

The assessment of the construct validity of CES used data from the ‘Quality of Life in your Neighbourhood survey 2010’ conducted in Bristol (UK) [9]. This voluntary survey was administered to a sample of adults randomly selected from the electoral register. The survey was postal but could also be completed online. The survey comprised 61 questions capturing information about individuals’ local area, lifestyle, health and well-being and other contextual information. The CES questionnaire and a number of other caring-related questions were also included for carers responding to the survey.

Respondents who were caring for family members, friends, neighbours or others due to long-term physical or mental illness, disability or other problems related to old age and had completed the CES questionnaire were included in the analysis. Questionnaires with incomplete and missing data or with more than one answer given to the questions of interest were omitted from the analysis. Data from survey respondents were double-entered and analysed using Stata version 12MP. Index value scores for the CES response profiles were calculated by summing the utility weights for each level of the six attributes in the relevant profile. The index scores for the 729 caring states defined by the CES range from 0 (worst caring state) to 100 (best caring state) [7].

Validating measurement tools requires hypotheses about the interpretive argument to be stated before the evaluation by taking into consideration all available evidence that could strengthen the development of constructs [10, 11]. Therefore, a set of hypothetical constructs with respect to the anticipated relationship between CES (attributes and score) and contextual variables was developed drawing on a combination of the qualitative research on the development of CES [6, 7, 12] and on the broader literature, both quantitative and qualitative, around the caring experience (although these studies related to other measures of the caring experience, rather than the CES). Where there was no existing evidence, hypotheses were developed based on the authors’ judgement on possible associations.

For the assessment of construct validity, 23 contextual variables were extracted from the survey and grouped into four categories, namely, those that related to the carer, recipient, caring situation and underlying motivation in caring. Associations between the responses on CES and contextual variables were investigated using chi-square tests for categorical variables and one-way analysis of variance for continuous variables. Fisher’s exact tests were used when one or more cells of a contingency table for a categorical variable had an expected frequency of less than five. The levels of statistical significance used in the examination of statistical strength of the associations were 5 and 1 %. In addition to the statistical significance, effect sizes were also calculated using Eta squared.

Hypotheses about whether the overall CES score is associated in the expected manner with the caring context

The CES score is the sum of a set of index values that are applied to individual responses. The literature points towards associations between the caring experience and variables in all four categories selected. More specifically, the literature indicates strong evidence of (a) positive association between caring experience and the carer’s health (overall health, functional state or absence of chronic disorders) [8, 12–19], recipient’s age [12, 15, 17, 20, 21] and health status [8, 12–14, 16–18, 20] and (b) negative association of caring experience with the duration and intensity of caring [8, 12–15, 17, 21–24] as well as with the relationship proximity [8, 12–14, 19].

Intensity of caring comes about from a range of factors (such as the underlying health of the recipient and tasks undertaken), and therefore, it was generated from a combination of three dummy variables—(1) being a main carer (or not); (2) assisting in the provision of personal care (or not); (3) recipient’s health status—which can be assumed to determine the level of effort required by a carer. Caring was considered ‘intense’ when all three categories were fulfilled, ‘relatively intense’ when two categories were fulfilled and ‘not intense’ when one or zero category was fulfilled. Evidence on statistically significant relationships between the CES score and the carer’s age [18, 20], gender [22–24] and employment [17, 18, 23, 25] was limited and with contradicting direction of the association, so they were not included as constructs in this study.

A number of studies have pointed out a strong association between the attachment to the recipient and overall caring experience. However, studies examining associations between the caring experience and other motivation-related variables were limited [16, 19]. In this study, five motivation-related variables were explored. These motivations were driven by: duty, being the closest person to the recipient, absence of someone else to provide care, lack of money for paid care and caring by free choice. The former four were expected to be negatively associated with CES scores in contrast to the later one that was anticipated to be associated with higher CES scores. Finally, it was expected a priori that sleep disruptions, being confined at home by caring and lack of frequent meetings with family and friends would all be negatively associated with caring experience. Overall, from 18 associations hypothesised for this investigation, 17 were based on available evidence from the literature and only the variable ‘being confined at home by caring’ was based on authors’ opinion (given the lack of previous evidence).

Hypotheses about whether responses to the CES items are related in the expected manner with the carer and recipient-related characteristics

Given the more specific nature of the CES attributes, there was less existing evidence upon which to hypothesise relationships between the contextual variables and the six CES items. Of 27 associations between the six CES attributes and carer and recipient characteristics, 16 were based on the available literature.

  • Activities outside caring relate to the ability of a carer to perform other tasks in personal life. Positive associations were anticipated with the carer’s health and employment as well as with the recipient’s health [12, 13]. Negative associations were expected when the carer was older [13], when the carer’s frequency of exercise or meeting with the family or friends was low and when caring confined carers at home.

  • Social support concerns the support (practical and emotional) carers receive from their families and friends. A positive association with the carer’s and recipient’s health was expected [13]. A positive association was also expected when carers were meeting frequently with family or friends and a negative when caring confined carers at home.

  • Institutional support refers to the assistance a carer receives from the government or other organisations. For this attribute, a positive association was expected when carers were receiving a financial support for caring (means-tested benefit).

  • Fulfilment from caring relates to the positive feelings derived from caring. On the basis of available evidence, positive associations were expected with the carer’s health [26, 27], carer’s age [26, 28] and recipient’s health [12, 29]. Negative associations were anticipated for carers with disability [27], female carers [12] and employed carers [26, 28].

  • Control over caring is concerned with the level of independence in decision-making about the care of a person. Control was anticipated to be positively associated with the carer’s and recipient’s health and the age of the carer [12]. In addition, positive associations were expected when carers’ sleep quality was good and when caring confined carers at home.

  • Getting on with the care recipient refers to the quality of the carer–recipient relationship. Positive associations were expected with the recipient’s age and when carers met frequently with their families or friends.

Hypotheses about whether responses to the CES items are related in the expected manner with the caring situation and motivation for caring

The motivation-related variables were used to capture what factors influenced the carer’s decision to provide care. The five factors assessed were the provision of caring because of duty, being the closest to the person needing care, free choice, absence of someone else and the lack of money for paid care.

When there was no one else to provide care and therefore there was an element of ethical obligation, a strong negative association with most CES attributes was hypothesised. The exceptions were the attributes of institutional support, where no association was hypothesised and control, for which a positive association was expected. When caring was perceived as a duty, it was anticipated that fulfilment from caring would be lower and getting on with the care recipient disrupted. If caring was perceived as a free choice, a positive association with fulfilment, control and getting on with the care recipient was expected. Control and getting on were also expected to be positively associated with providing care as the closest person to the recipient. Furthermore, feeling close to the recipient was expected to be negatively associated with the activities outside caring and fulfilment from caring [12, 28]. Finally, if there was no money for paid care, carers were expected to report lower institutional support and fulfilment from caring.

There was strong evidence in support of possible associations between the CES attributes and the caring situation, described by the duration and intensity of caring. It was anticipated a priori that duration and intensity of caring would be negatively associated with the activities outside caring [12, 13, 28, 30], social support [13] and getting on with the care recipient [30]. However, duration and intensity of caring were anticipated to be associated with better control over caring. Finally, the provision of personal care was anticipated to be positively associated with institutional support [31] and negatively associated with fulfilment [26, 30]. For this investigation, 18 (53 %) constructs were based on available evidence, and the rest were hypothesised by the authors.

Hypotheses about whether the CES demonstrates construct validity for both carers of older people (65+) and carers of those aged 18–64

The CES was developed with carers of older people, but the attributes appear to be common to measures developed for carers more generally. In this investigation, the associations between the CES scores and contextual variables in the analysis were examined for those caring for people aged 18–64 and over 65 separately. The results were then compared with the constructs developed in the first investigation, with the purpose of examining whether the CES demonstrates construct validity in carers of both older and younger adults.

Results

The ‘Quality of Life in your Neighbourhood’ survey was administered to 28,105 adults. In total, 5,187 (19.3 %) questionnaires were returned and 984 (19 %) of these were for people who reported being informal carers, 730 (74.2 %) of whom had fully completed the CES questionnaire as intended. The sample consisted mostly of female carers (62.5 %), carers below the age of 65 (74 %), carers in good health (83.4 %), carers without major disability (86.1 %) and carers without long-term illness (63.9 %). Carers were mostly caring for older recipients (75.8 %) and caring for people for whom they described health as at least ‘fair’ (61.4 %). Full descriptive statistics and responses to the CES are provided in Tables 1 and 2.

Table 1 Sample characteristics
Full size table
Table 2 Responses in each of the six attributes of the Carer Experience Scale questionnaire (n = 730)
Full size table

Is the overall CES score associated in the expected manner with the caring context?

Most CES scores were distributed at the higher end of the scale (Fig. 1). Associations and effect sizes between the carer, care recipient, caring situation and motivation-related variables and CES scores are presented in Table 3. Associations that were hypothesised a priori have been highlighted in italics. In total, 17 out of 19 associations were captured by the CES, and all were in the expected direction. In more detail, all strong associations directed by the literature were identified, 7 (87.5 %) of which were significant at the 0.01 significance level. Mean CES scores across different levels of selected variables are presented in Table 4. Apart from the perception of caring as a duty, all motivation-related characteristics were statistically significant at p ≤ 0.01. Further aspects determining the level of caring intensity, such as being a main carer, and assistance in the provision of personal care were examined. The former was associated with CES with p ≤ 0.01, whilst the latter seemed a less significant driver of the caring experience or intensity of caring. As hypothesised, the carer’s age and gender were not associated with CES. Even though no association between employment and CES scores was hypothesised, the two variables were found to be positively associated.

Fig. 1
figure 1

Distribution of the Carer Experience Scale scores (n = 730)

Full size image
Table 3 Univariable associations and effect sizes between the Carer Experience Scale scores and contextual variables, using one-way analysis of variance (n = 730)
Full size table
Table 4 Mean Carer Experience Scale scores and confidence intervals (CI) for different categories of selected variables (n = 730)
Full size table

Are responses to the CES items related in the expected manner with the carer and recipient-related characteristics?

Associations between the CES attributes and carer and recipient characteristics are presented in Table 5, with expected associations highlighted in italics. Regarding the activities and social support attributes, all associations that were hypothesised were significant at p ≤ 0.01. Institutional support and fulfilment were not associated as expected with a number of carer and recipient-related variables. Control over caring was associated with the carer’s age and being confined at home by caring as hypothesised a priori, but not with the carer’s or recipient’s health. Finally, associations expected between getting on with the care recipient and the frequency of meetings with family or friends and recipient’s age were both statistically significant.

Table 5 Univariable associations between the Carer Experience Scale attributes and carer/recipient-related variables, using chi-square test unless otherwise is indicated (n = 730)
Full size table

Are responses to the CES items related in the expected manner with the caring situation and motivation for caring?

Motivation- and situation-related characteristics were expected to be important determinants of the caring experience and significant influences on the responses to the CES questionnaire. This is reflected in the 34 associations hypothesised a priori presented in Table 6. All 11 associations hypothesised between the caring situation and motivation-related variables and the attributes of activities outside caring and social support were statistically significant at p ≤ 0.01. Institutional support was found to be associated with the lack of money for paid care and assistance in the provision of personal care as expected.

Table 6 Univariable associations between the Carer Experience Scale attributes and caring situation/motivation-related variables, using Chi-squared test unless otherwise is indicated (n = 730)
Full size table

Fulfilment from caring and getting on were significantly associated with the provision of care due to free choice, absence of someone else and the lack of money for paid care. Furthermore, evidence suggested that getting on with the care recipient is negatively associated with the duration and intensity of caring [30]. However, a positive and non-significant association appeared in this study. Finally, and as hypothesised, control over caring was significantly associated with all characteristics related to the caring situation and the provision of care due to being the closest person to the care recipient at p ≤ 0.01.

Does the CES demonstrate construct validity for both carers of older people (65+) and carers of those aged 18–64?

Based on the constructs developed for the CES score and the contextual characteristics, 17 associations were hypothesised. In total, 11 associations appeared to be statistically significant in both age groups (Table 7). Generally, associations between CES and the carer’s health (overall health, existence of disability or long-term illness), caring by free choice, caring due to the absence of someone else or due to the lack of money for paid care as well as the intensity of caring, which were perceived as major drivers of caring experience, were all evident in both groups. However, some contextual variables (providing care due to being the closest person, recipient health state and providing personal care) were significantly associated with the CES score in only one of the two groups.

Table 7 Univariable associations between the Carer Experience Scale (attributes and score) and contextual variables for those caring for people aged 18–64 and 65+, using one-way analysis of variance (n = 730)
Full size table

Discussion

This study examined whether the CES captures the caring experience in a valid way in a heterogeneous sample of carers of people over 18 years of age. Positive findings would enhance the case for using the measure in evaluations and resource allocation decisions involving carers. A number of investigations were conducted to examine whether associations between the CES attributes and score and contextual variables would be observed as anticipated. Overall, 73 % of the constructs hypothesised (including constructs related to the carer, recipient, situation and caring motivation) were significant at p ≤ 0.05, indicating that the CES is an instrument that largely measures what it purports to in a valid way.

The findings of this study were in line with the available literature in the field. More specifically, the carer’s health, disability or long-term illness, the provision of care due to being the closest to the person needing care, recipient’s age and health, duration and intensity of caring were significantly associated with the caring experience. Similarly, the carer’s age and gender were not associated with the overall caring experience. Employment was positively associated with CES responses, which is in line with the findings observed in a sample of caregivers of stroke survivors in South Korea [23] despite most studies indicating a non-association [8, 12–15] or a negative association [17, 18, 25]. Employment may create the opportunity for social relationships and to ‘have a life outside’ caring, and these factors may help to alleviate the carer’s sense of burden. Nevertheless, more attention needs to be placed on understanding the inevitably complex relationship between employment and the caring experience. In the present study, the positive association can be partially explained by the larger proportion of employed secondary carers (64 %) in the sample (with the secondary caring role potentially being a lot less stressful than a primary caring role).

In addition to what was already known, this study sheds more light on the factors associated with the caring experience, where comprehensive evidence was lacking. The ability of a carer to perform activities outside caring and the level of social support available within caring were significantly associated with all four categories of caring context (carer and recipient characteristics, motivations and caring situation). However, carer and recipient characteristics were not particularly associated with institutional support, fulfilment, control of caring and getting on with the care recipient. These aspects of the caring experience were more closely related to the caring situation and motivation variables.

More specifically, caring by free choice, caring because of the absence of someone else or the lack of money for paid care were significantly associated with the level of fulfilment derived from caring and the carer–recipient relationship (getting on), indicating that the emotional and interpersonal aspects of the caring experience are potentially driven by the underlying motivation in caring. Likewise, the caring situation was associated with the level of institutional support and control over caring. As has been mentioned earlier, control over caring is an attribute of CES that has not been incorporated in any other carer-specific measure of QoL. Given that most studies focus on the negative implications of the intensity and duration of caring in the carer’s QoL, this study shows that other variables related to the caring situation potentially have a positive impact on caring as they are, on average, associated with better caring experience, capturing the need of carers to have control over the caring process [6].

The work has some limitations worth highlighting. The first potential limitation is that around 25 % of individuals who reported being carers did not fully complete the CES. Future research may examine whether some groups of carers feel certain questions on the CES do not apply to them. Another limitation is the lack of information collected on the relation between carer and recipient. This could have added more context to the data as well as potentially allowing the testing of differential responses between spousal carers and others. Finally, further investigation is required to examine the psychometric properties of the CES. These investigations could include examining the construct validity of the CES in specific clinical subgroups (for example dementia) and in relation to other established measures of caring burden (such as the Zarit burden scale) [32]. Investigation of the responsiveness of the CES in detecting changes in the carer’s outcomes over time would also be valuable.

The key strength of this study is the large sample size and wide range of variables referring to the context of caring. This has enabled a comprehensive examination of construct validity and an examination for the performance of the CES amongst carers of younger (18–64) and older (65+) adults. Although some hypothesised associations were not found to be significant, in general the findings are encouraging with respect to using the CES to measure outcomes for a heterogeneous group of carers in the UK.