Introduction

Historically, the treatment of depression and other mental disorders has focused on reducing symptoms and decreasing the negative experiences of mental illness. However, there is increasing recognition of the importance of incorporating a recovery model of care into both treatment and research 1 , 2. Recovery models focus on increasing positive experiences and helping clients live a meaningful life instead of a sole focus on reducing symptoms 2. However, in order to be more recovery focused, psychometrically sound measures of recovery are required.

One of the most commonly used measures of recovery in mental illness research is the Recovery Assessment Scale 3. The scale consists of 41 statements, each positively worded, and participants rate their agreement with each item. Factor analyses showed a five-factor structure with only 24 of the items loading on the factors 4, so only those 24 items are scored in subscale analysis. The five factors or subscales are personal confidence and hope, willingness to ask for help, having a goal and success orientation, better reliance on others, and not being dominated by the symptoms of the mental illness. Although the five-factor structure of the RAS is well supported, a bifactor model has not been examined. A bifactor model includes a general factor that affects all items and accounts for the correlations between factors 5. As depicted in Fig. 1a, the bifactor model includes a general factor of all scored items. Additional factors are included for each of the subscales, the five RAS subscales in this case. No correlation is included between the subscales because the general factor accounts for any bivariate correlation between subscales. This differs from the five factor model, depicted in Fig. 1b, that models the associations between factors directly without a general factor. An advantage of knowing whether a five-factor or bifactor model fits the RAS is that the five-factor model supports use of the subscales while the bifactor model supports the use of both subscales and total scores. Although total scores that include all the subscales are often used, rarely is a bifactor model investigated to determine if the total score is appropriate to use or if the subscales should be the main focus in research and clinical settings.

Figure 1
figure 1

Diagram of the bifactor (a) and five-factor (b) models The bifactor model includes a general factor along with an additional factor for each subscale. The general factor accounts for the correlation between subscales. The five-factor model only includes the five subscales with correlations between subscales

Full size image

The RAS has shown good reliability and validity further supporting its continued use in mental health research. Fourteen-day test-retest reliability was high (0.88) as was internal consistency (0.93) in one of the original evaluation studies 6. The RAS also correlates positively with other measures of positive outcomes such as self-esteem and quality of life 6 as well as other recovery-based measures 7 supporting its validity. While the RAS has excellent psychometric properties overall, one omission from the literature is the RAS’s sensitivity to change. Sensitivity to change is the ability of a measure to show change after an intervention, across time or between groups that should theoretically differ on the measure 8. Being sensitive to change is particularly important for measures used in clinical trials and treatment settings as they may reflect measurement problems rather than true success or failure of the treatment if the measure is not sensitive to change.

The current study sought to expand our knowledge of the psychometric properties of the RAS. Secondary analysis of data from a randomized controlled trial of a self-management support services intervention for chronic depression was used 9. The randomized trial showed the intervention improved the total RAS score and reduced depressive symptoms and major depressive episodes. For this study, we examined both factor structure and sensitivity to change of the RAS. For factor structure, we tested the five-factor model and the bifactor model. We examined sensitivity to change three ways: comparing the intervention and control group; comparing baseline and follow-up scores from people with a diagnosis of depression at baseline but no diagnosis at follow-up; and comparing baseline and follow-up scores in the intervention group alone. Both the total score and the subscales were analyzed for sensitivity to change.

Methods

Participants and procedures

This study was a secondary analysis of a clinical trial testing a depression self-management support service intervention against a treatment as usual control group 9. To be eligible, potential participants had to be 18 years of age or older, have seen a doctor for antidepressants at least twice in the past year, be diagnosed with recurrent depression or dysthymia, and have residual symptoms of depression. Exclusion criteria were current participation in any other depression treatment trial, plans to move out of the study area, and diagnosis of bipolar disorder or psychotic disorder. Participants were not excluded if they had comorbid anxiety, substance use, or personality disorders. Participants (n = 302) were recruited from four primary care clinics and a local medical center using two recruitment methods. The first method identified potentially eligible participants through electronic health records and sent an invitation letter to the potential participants. The second method gave medical providers information and brochures about the study so they could refer patients. All participants provided informed consent, and the institutional review boards of the study centers approved the procedures before the study was conducted.

After participants were accrued, they completed a baseline assessment. They were then randomized to either the self-management intervention or the treatment-as-usual control group. The intervention lasted for 18 months and consisted of regular contacts with a mental health professional functioning as a care manager and skills groups that included both recovery-oriented and cognitive-behavioral strategies as well as strategies to improve motivation and engagement in treatment. Groups met weekly for the first 2.5 months, twice monthly for the next 2 months, and then monthly during the maintenance phase or the remaining months. A mental health profession and peer specialist led the groups, and the peer specialists also provided brief contacts, as needed, that focused on recovery goals. The care manager also contacted participants at least monthly during the initial year of study participation to support participants’ practice of self-care strategies and assess symptoms and progress in meeting their treatment goals. Contacts after the first year depended on symptom level, and engagement with other providers. Care managers also provided reports to other providers and coordinated follow-up care. Assessments were completed at 3, 6, 12, and 18 months after the baseline assessment.

Measures

Recovery Assessment Scale

The RAS consists of 41 items, each answered on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Only 24 of the items are included in scoring, and averaging the items creates the scores for the total scale and the subscales: confidence and hope; willingness to ask for help; goal and success orientation; reliance on others; and no domination by symptoms.

Structured Clinical Interview for the DSM

The Structured Clinical Interview for the DSM (SCID) 10 is an interviewer-administered measure used to make diagnoses of psychiatric disorders from the Diagnostic and Statistical Manual of Mental Disorders 11. Interviewers completed training in administering the SCID before interviewing participants. For this analysis, only the diagnosis of major depression was used.

Statistical analysis

Factor analysis

Data from the baseline assessment was used for the factor analyses. A confirmatory approach was used for the factor analyses as the factor structure of the RAS has already been investigated. However, to add to the literature on the RAS, we tested a bifactor model as well as the previously supported five-factor model. A bifactor uses a general factor, which affects all items, to explain the correlation between the other factors 5. A bifactor model can support the use of both the total scale score as well as the subscale scores. However, if the five-factor model fits better, this would suggest a focus on the subscales instead of the total score. We therefore tested a bifactor model and the five-factor model using LISREL 8.8. The analyses used a polychoric correlation matrix with diagonally weighted least squares for the estimation method. Models were compared on the tests of perfect and close fit, root mean square error of approximation (RMSEA) 12, comparative fit index (CFI) 13, root mean square of the residuals (RMSR), expected cross validation index (ECVI), and the corrected Akaike information criterion (AIC). For the RMSEA, RMSR, ECVI, and AIC, lower values indicate better fit. For the CFI, higher values indicate better fit. Although the RAS has 41 items, we only used the 24 scored items in the bifactor model and did not include the unscored 17 items in the model.

Sensitivity to change

We examined sensitivity to change in three comparisons. First, the intervention and control groups were compared at each follow-up assessment (3, 6, 12, and 18 months). Second, people who met criteria for major depression on the SCID at baseline were classified as improved at each follow-up if the SCID showed they did not meet criteria anymore. For this comparison, RAS values at the follow-up assessments were compared to baseline. Lastly, the baseline values were compared to the follow-up values for the intervention group only. For these analyses, we used the effect size measure of sensitivity to change 8 , 14. While other measures are available, they could not be conducted for the first comparison (intervention and control groups) as the measures assume a correlation between the groups and the groups were created by random assignment. Although we use Cohen’s 14 original guidelines of interpreting effect size (0.2 = small, 0.5 = moderate, 0.8 = large), these are not strict cutoffs and effect sizes should be considered within the context of the population and study design.

Results

The sample recruited was consistent with the population of the clinics and medical center through which the study was conducted (see Table 1 for descriptive statistics). The sample was predominantly female, unsurprising given the gender difference in depression 15. The average participant was middle-aged and had not completed a bachelor’s degree. Attrition was low as 90% of the sample was retained by the last (18-month) follow-up.

Table 1 Demographic characteristics (n = 302)
Full size table

Factor analysis

The confirmatory factor analyses supported the five-factor model. The fit indices, reported in Table 2, showed that both the five-factor and bifactor models fit the data well. However, the AIC did not substantially improve between the five-factor and bifactor models 16. Also, factor loadings, in Table 3, showed that some loadings on the general factor of the bifactor model were low, particularly for items 30 through 32 and item 40. As the five-factor model is more parsimonious and it did not substantially improve with the addition of the general factor, the original five-factor structure was supported.

Table 2 Fit indices for factor analyses
Full size table
Table 3 Factor loadings from factor analyses
Full size table

Sensitivity to change

Effect sizes are reported in Table 4. The RAS showed moderate to large changes over time or when symptoms improved but only showed a small change between the two treatment groups. When comparing the control and intervention groups, three subscales (confidence and hope, willingness to ask for help, and goal/success orientation) and the total score only showed small differences between the two groups by the 18-month follow-up. No domination by symptoms showed small to medium differences between the control and intervention groups. In contrast to the comparisons between treatment groups, nearly all subscales of the RAS and the total score showed small to moderate changes at 3 months, increasing to moderate and large changes by 18 months when comparing baseline RAS scores to follow-up scores for those whose depression remitted. The one exception was the reliance on others subscale that practically did not change across all three comparisons. The effect sizes comparing baseline to follow-up RAS scores in the intervention group showed a similar pattern to the analyses for improvement by the SCID with the total RAS score and all subscales except Reliance on Others showing small changes at 3 months and moderate to large changes by 18 months.

Table 4 Sensitivity to change results
Full size table

Discussion

The results of this study showed that the RAS is a psychometrically sound measure confirming previous research 4 , 6. The five-factor structure shown in other studies was supported, and a bifactor model did not fit the data better. While the overall RAS score showed sensitivity to change across treatment groups and changes in depression diagnoses and across time in the intervention group, not all subscales showed equal sensitivity to change. The no domination by symptoms subscale showed excellent sensitivity to change while the confidence and hope, goal and success orientation and willingness to ask for help subscales were sensitive to change in depression diagnosis and across time in the intervention group. The reliance on others subscale was not particularly sensitive to change, and most subscales only showed small differences between the control and intervention groups on follow-up assessments.

That the symptom subscale of the RAS showed change particularly well is not surprising as most treatments are focused on alleviating symptoms 2. The intervention in this study partially focused on helping participants engage more in traditional mental health care and on CBT skills 9. Previous studies have highlighted how the RAS correlates with more traditional symptom-based measures 17.

Implications for Behavioral Health

The results imply that the subscales of the RAS should ideally be used to measure response to treatment, whether in research or clinically, instead of just the total score. Results of the factor analysis suggested that each subscale measures a distinct, but related, aspect of recovery rather than just facets of a single construct of recovery. The sensitivity to change results also showed that certain subscales, no domination by symptoms in particular, are more sensitive to change than other subscales. The RAS was particularly sensitive to change over time, as shown by comparisons of changes in depression diagnosis and changes over time in the intervention group. However, effect sizes were smaller when comparing the intervention group to the control group suggesting that the RAS may be more useful for longitudinal comparisons but less likely to show results for symptom-focused treatments.

The implications should be considered within the strengths and limitations of the study. Participants were included regardless of comorbidity status, improving generalizability of the results but possibly reducing effect sizes. Unlike most sensitivity to change studies, we were able to compare an intervention and control group as well as changes over time. This intervention may not have affected reliance on others, possibly explaining why this subscale did not show sensitivity to change from the intervention. Also, the subscales that did show sensitivity to change may not show sensitivity to other interventions. For example, an intervention focused on individual therapy or medication treatment would differ significantly from the intervention described in this paper and the sensitivity to change results may not apply. Further trials of different interventions would be needed to test whether reliance on others is sensitive to other interventions. These results might not translate to populations with other mental illnesses as their primary diagnosis or to populations outside the Northwest region of the USA. Further studies are also needed to determine whether the 17 unscored items of the RAS have to be administered or whether only the 24 scored items can be used. Nevertheless, results support the continued use of the RAS and gives guidance on ways to effectively use the measure in clinical use or future research studies.