Introduction

Standardized measures designed specifically to help assertive community treatment (ACT) teams identify clients who might be ready to transition to less intensive services are needed for the field and are important for mental health practice, policy, and services research. ACT is a widely studied intervention for individuals living with serious and persistent mental illnesses and there is ample evidence that ACT reduces hospital admissions, especially among those who are the highest utilizers of inpatient services (Bond et al. 2001; Dixon 2000). By design, ACT serves the most profoundly ill among those with mental illnesses and was originally conceptualized as a life-long, time-unlimited service (Stein and Test 1980). However, ACT as a lifetime service is contrary to what we know today about mental illnesses and recovery (Anthony 1993; Davidson 2003).

In many communities, more ACT capacity is needed (Cuddeback et al. 2006); however, a key component of ACT team fidelity is the provision f “time-unlimited” services which allows clients receive treatment indefinitely such that many ACT teams have interpreted “time-unlimited” to mean “forever,” potentially creating situations where clients are overserved and are kept on ACT teams longer than necessary (Donahue et al. 2012). A potential solution to this capacity-demand problem is to transition ACT clients to less intensive services.

Findings from studies that have examined the transitions of ACT clients to less intensive services are mixed (Audini et al. 1994; Bromley et al. 2017; Hackman and Stowell 2009; Jones et al. 2003; McRae et al. 1990; Salyers et al. 1998; Stein et al. 1999; Stein and Test 1980; Tomar et al. 2020). For example, in Stein and Test’s original study of ACT transitions, within 14 months of transition from ACT, those who were randomly assigned to transition from ACT regressed to their pre-ACT functioning and hospitalization patterns (Stein and Test 1980).

Additionally, Bromley et al.’s study suggests that ACT clinicians may make mistakes in discharging clients which could be remedied with assessments of clinical status and processes to identify factors that predict successful discharge from ACT. Other studies, however, report no deterioration in outcomes among those transitioned from ACT and suggest that clients can be selectively discharged or transferred without losing gains in housing, employment, or behavioral health functioning (Huz et al. 2017; Rosenheck and Dennis 2001; Salyers et al. 1998).

In light of the mixed findings regarding studies that examine transitions among ACT participants, standardized measures designed specifically to help ACT teams identify transition-ready clients are needed to advance the field. Without such measures, it is possible that ACT teams will be limited with respect to identifying participants who might be ready to transition to less intensive services and this is problematic given the need for increased access to evidence-based practices in the public mental health system (New Freedom Commission on Mental Health 2003).

Measures such as the Tool for Measurement of Assertive Community Treatment (TMACT), the Dartmouth Assertive Community Treatment Scale (DACTS) have been developed to assess ACT fidelity, and, more recently, the Transition Readiness Scale (TRS) has been developed to help ACT teams identify program participants who are potentially ready to transition from ACT to less intensive services (Monroe-DeVita et al. 2011; Winter and Calsyn 2000; Donahue et al. 2012). The TMACT and the DACTS require on-site fidelity reviews and there is limited information about how they perform with respect to predicting who can transition from ACT. The TRS shows great promise as a tool to assess ACT consumers’ transition readiness and uses administrative data and a data algorithm to calculate transition readiness scores for ACT consumers (Donahue et al. 2012). Donahue and colleagues (2012) demonstrated the feasibility and practicality of using data that are routinely gathered through a statewide data reporting system in generating TRS scores for ACT consumers and ACT teams, and suggest, in the absence of a centralized data reporting system, ACT teams could implement the TRS given the availability of the needed data elements at the programmatic level.

For teams that do not collect or are unable to access the data needed to generate TRS scores, paper-and-pencil measures could be a useful strategy to assess transition readiness among ACT consumers, and this remains a gap in the literature. The development of these measures has advanced the field but more research is needed about accessible and practical measures of ACT transition readiness. To address these gaps, we developed and pilot tested the Assertive Community Treatment Transition Readiness Scale (ATR), a paper-and-pencil measure designed to be completed by ACT staff to identify ACT clients who might be ready to transition from ACT to less intensive services. Additionally, the psychometric properties of the ATR are reviewed.

Method

The Assertive Community Treatment Transition Readiness Scale (ATR)

In developing the items for the ATR, several sources for item content were identified: (a) experienced ACT staff, (b) relevant literature and research on ACT transitions, and (c) similar standardized measures. Semi-structured interviews with ACT staff members who had at least one year of experience in transitioning ACT clients from ACT to less intensive services and who were employed at a large community mental health agency in the Midwest were conducted. Interviews were designed to solicit in-depth information about staff members’ experiences with transition from ACT to less intensive services, with emphasis on the characteristics of clients who are ready to transition. For example, ACT staff were asked: “How would you describe a consumer who is ready to be transitioned to less intensive services?” and “How would you describe a consumer who is not ready to be transitioned to less intensive services?” and “Describe your experiences with a successful transition of an ACT consumer to less intensive services.”

Interviews lasted from 60–90 min and were facilitated by the principal investigator of the study. All interviews were audio-taped and transcribed. Participation was voluntary and written consent was obtained. A total of 22 ACT staff members participated in the study. Independent coders used an iterative process to build an understanding of characteristics of clients who were ready to transition from ACT. Open coding analysis techniques (Emerson et al. 1995) were used. Coders coded one interview together to ensure consistency then applied the deductive codes to his or her assigned transcripts and noted any emerging codes.

In total, 20 different themes emerged and the frequency with which these themes were mentioned among respondents were noted. For example, a number of themes were expressed by 100% of the ACT staff members who participated in the interviews: stability, criminal justice contacts, housing, substance use, and hospitalization. Ultimately, ATR items were written to capture the following areas: psychiatric and behavioral stability; hospitalization and incarceration; housing stability; substance use; treatment engagement; medication compliance; independence; dependence on team; complexity of health and behavioral issues; intensity of service need; benefits; social support; resources; insight; daily structure; and employment.

Next, additional sources were reviewed to validate the emergent themes and identify additional content areas. These sources included the Level of Care Utilization System for Psychiatric and Addiction Services (LOCUS) (Sowers et al. 1999) and relevant literature (e.g., Hackman and Stowell 2009; McRae et al. 1990; Salyers et al. 1998; Stein and Test 1980; Stein et al. 1999; Susser et al. 1997). The literature review and the semi-structured interviews were the basis for the ATR items. Care was given to constructing items by using principles of good item construction and to writing clear instructions for completion (e.g., DeVellis 1991; Nunnally and Bernstein 2001). As a result, the initial ATR was a 20-item measure designed to assess clients’ readiness to transition from ACT to less intensive services.

Each ATR item is rated on a 4-point scale (i.e., strongly disagree to strongly agree). Total scores range from 20 to 80, with higher scores on the ATR indicating greater potential to transition successfully. The ATR was written at a fifth grade reading level and items were written to cover the following areas: psychiatric and behavioral stability; hospitalization and incarceration; housing stability; substance use; treatment engagement; medication adherence; independence; dependence on team; complexity of health and behavioral issues; intensity of service need; benefits; social support; resources; insight; daily structure; and employment.

Design

To examine the psychometric properties of the ATR, a retrospective cohort study design was used to collect clinician-report data on ATR items from ACT staff who had at least one year of experience in transitioning ACT clients to less intensive services. ACT staff members who participated in the study were asked to think about an ACT client who in his or her own definition had a successful transition from ACT to less intensive services and a client who in his or her own definition had an unsuccessful transition to less intensive services. Respondents were asked to choose clients for whom they knew recently, well, and had knowledge about post-transition outcomes (i.e., hospitalization, incarceration, continuation with treatment and medication, return to ACT). Then, respondents were asked to think about these clients as they knew them immediately before they were transitioned from ACT, and to complete two copies of the ATR (i.e., one for their successfully transitioned clients and one copy for their unsuccessfully transitioned clients). Respondents were invited to review case notes to help recall information.

Data collection was conducted through SurveyMonkey and study solicitations appeared on two nationally- and internationally-viewed ACT-specific web sites (i.e., The ACT Center of Indiana at the Indiana University-Purdue University Indianapolis and The Assertive Community Treatment Association). This strategy resulted in data on the ATR items from a national sample of 109 ACT staff who reported on a sample of 218 ACT participants who were transitioned from ACT to less intensive services. Data collection occurred between 2009 and 2011.

The study was approved by the Institutional Review Board at the (omitted to preserve the anonymity of the review process).

Sample

There were missing demographic data for 14 (13%) of the 109 respondents such that sample characteristics for 95 ACT staff are described. ACT staff who provided data on the ATR were mostly female (66.3%, n = 63), white (92.6%, n = 88) and had a Master’s degrees in social work or other fields (63%). ACT staff averaged 15.33 (SD = 8.68) years of mental health experience and 6.29 years (SD = 4.93) of ACT experience. Also, on average, ACT staff reported having transitioned 24.02 (SD = 33.71) clients from ACT to less intensive services. Most respondents came from outpatient mental health centers (91%, n = 87) in urban areas (72%, n = 68) and most (75.5%, n = 71) were from full fidelity ACT teams. Also, 45% (n = 34) of respondents were ACT team leaders and 20% (n = 15) were case managers. The remaining respondents included psychiatrists, psychiatric nurses, substance abuse specialists, vocational specialists, benefits specialists and peer support specialists. Respondents from 19 states and Canada provided clinician-report data on the ATR for 218 clients who were classified as having either successful or unsuccessful transitions from ACT.

The characteristics of the sample of 218 clients who were transitioned from ACT are shown in Table 1. As shown in Table 1, among those who were transitioned successfully, 54% (n = 61) were male, 63% (n = 70) were white, 34% (n = 38) were African American, 63% (n = 65) had a diagnosis of schizophrenia or schizoaffective disorder, 25% (n = 26) had a diagnosis of bipolar disorder, and 28% (n = 32) had a co-occurring substance use disorder. Among those who were unsuccessfully transitioned, 59% (n = 48) were male, 49% (n = 40) were white, 49% (n = 40) were African American, 69% (n = 50) had a diagnosis of schizophrenia or schizoaffective disorder, 24% (n = 17) had a diagnosis of bipolar disorder, and 45% (n = 37) had a co-occurring substance use disorder.

Table 1 ACT client sample characteristics and post-ACT transition outcomes
Full size table

Measures

Transition Status

As stated previously, each survey respondent was asked to complete one copy of the ATR for his or her successfully transitioned client and one for his or her unsuccessfully transitioned client and to think of these clients as they were immediately before they transitioned from ACT. ACT workers were allowed to define successful and unsuccessful themselves. Transition status was coded 0 = success and 1 = failure in the analyses below.

Hospitalization

Hospitalization after transition from ACT, though not a perfect indicator, is an important one that should be correlated with the ATR. Therefore, ACT staff were asked on the ATR to indicate whether their successful and unsuccessful clients were hospitalized after their transition from ACT.

Incarceration

ACT staff were asked on the ATR to indicate whether their successful and unsuccessful clients were incarcerated after their transition from ACT.

Medication Adherence

ACT staff were asked on the ATR to indicate whether their successful and unsuccessful clients were adherent with medication after their transition from ACT.

Treatment Adherence

ACT staff were asked on the ATR to indicate whether their successful and unsuccessful clients were adherent with treatment after their transition from ACT.

Return to ACT

ACT staff were asked to indicate whether their successful and unsuccessful clients returned to an ACT team at some point after their transition from ACT to less intensive services.

Data Analysis

Exploratory factor analysis (EFA) was conducted to determine the factor structure of the ATR. EFA is most appropriate when there is not enough information to specify the underlying factor structure of a set of variables, and generally it is used when constructs are less well defined (Fabrigar et al. 1999; Gorsuch 1983; Loehlin 1998). Comrey and Lee (2013) suggested the following guidelines for EFA and appropriate sample sizes: 50 cases is very poor, 100 is poor, 200 is fair, 300 is good, 500 is very good, and 1000 or more is excellent; a minimum of 10 observations per variable or items is needed to optimize computation. Here, EFA was conducted with a sample of 218, although it is important to note that 109 respondents contributed the sample of 218 as each respondent was asked to complete two versions of the ATR. Bartlett’s test of sphericity and the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy were used to examine the suitability of ATR items for factor analysis and the scree test and Eigenvalues greater than 1.0 were used to determine the number of factors in the scale.

Unweighted least squares with promax rotation was used to extract factors (Bollen 1989). To enhance simple structure items with high loadings on a given factor (i.e., ≥ .40) and relatively low loadings on other factors (i.e.,<.20 than the loading on the given factor) were selected as indicators of a given factor. Then, empirically derived factors were examined for interpretability (i.e., an ACT client’s general potential to transition from ACT to less intensive services).

Cronbach’s alpha (α) was used to quantify the internal consistency reliability of the ATR (Nunnally and Bernstein 2001) and the following guidelines were used to characterize different values: (a) Poor: < 0.60; (b) Marginal: 0.60–0.69; (c) Good: 0.70–0.79; and (d) Excellent:  ≥ 0.80. In addition, a receiver operating characteristic (ROC) curve fit and area under the curve (AUC) was computed to examine the sensitivity and specificity of the ATR. Generally speaking, an AUC of 0.5 suggests a test has no ability to discriminate (i.e., ability to diagnose patients with and without a disease or condition), an AUC between 0.7 and 0.8 is considered adequate or acceptable, an AUC between 0.8 and 0.9 is considered excellent, and an AUC over 0.9 is considered outstanding. An AUC of 1.0 indicates a perfectly accurate test (Carter et al. 2016).

To assess the validity of the scores on the ATR, predictive criterion validity and known-groups validity of ATR scores were examined. To establish known-groups validity, bivariate inferential tests (i.e., chi-square and t-tests) were used as a preliminary step to establish differences in ATR scores and post-transition outcomes among successfully and unsuccessfully transitioned clients. Then, to examine known-groups validity, logistic regression was used and transition status (i.e., successful vs. unsuccessful) was regressed on gender, race, diagnosis, and ATR scores. To examine predictive criterion validity of ATR scores, logistic regression was used and post-transition hospitalization (i.e., treated as a dichotomous dependent variable) was regressed on gender, race, diagnosis and ATR scores. This was repeated for all post-transition outcomes (i.e., homelessness, incarceration, medication adherence, treatment adherence, and return to ACT).

With respect to missing data on the 20 ATR items, there were no missing data on 12 of the items and just one missing observation for each of the remaining 8 items. Missing data on client variables (gender, race, diagnosis, substance abuse) ranged from 11% (n = 23) to 20% (n = 43). There was less than 1% missing data on each of the post-transition outcome variables (i.e., homelessness, medication discontinuation, treatment discontinuation, incarceration, and hospitalization).

Results

Factor Structure

Clinician self-report data on ATR items for 218 ACT clients who were successfully or unsuccessfully transitioned from ACT were analyzed. In the analysis, the null hypothesis that the correlation matrix was an identity matrix (Bartlett’s test of sphericity) was rejected (X2(190) = 2081.96, p < 0.001), and the Kaiser–Meyer–Olkin (KMO) was 0.92, indicating that the items on the ATR were suitable for factor analysis. The results of the EFA and evaluation of a scree plot suggested a one-factor solution with the single factor having an Eigenvalue of 8.31 and accounting for 41.91% of the total variance in ATR scores. All but two items had factor loadings greater than 0.40. Item 8, “He/she has not had negative consequences from substance use over the last several months,” and item 11, “He/she is too dependent on services,” were dropped from the final measure. The final measure – see Appendix A – contained 18 items.

Reliability and Validity

Cronbach’s alpha (internal consistency reliability) for the 18-item ATR was examined for 218 ACT clients. Cronbach’s alpha for the 18 items on the ATR was 0.92, indicating excellent internal consistency reliability. The ROC analysis produced an area under the curve (AUC) of 0.83 (SE = 0.03, CI 0.78−0.89), indicating above average discrimination for scores on the ATR.

Results from the bivariate analyses are shown in Table 1. As shown in Table 1, there were no differences in the demographic or clinical indicators between those who were successfully vs. unsuccessfully transitioned from ACT, with one exception. Those who were unsuccessfully transitioned were more likely to have a co-occurring substance use disorder, compared to those who were successfully transitioned (44.6% vs. 28.3%, X2(1) = 5.55, p < 0.05).

Also, shown in Table 1, successfully transitioned clients had higher ATR scores, on average, compared to those who were unsuccessfully transitioned (56.09 [SD = 8.48] vs. 44.28 [SD = 9.39], t(209) = 9.57, p < 0.001). With regard to post-transition outcomes, compared to successfully transitioned clients, unsuccessfully transitioned clients were more likely to experience post-transition homelessness (3.4% vs. 31.0%, X2(1) = 30.11, p < 0.001), hospitalization (14.4% vs. 61.4%, X2(1) = 49.21, p < 0.001), incarceration (4.2% vs. 15.7%, X2(1) = 7.91, p < 0.01), discontinuation of treatment (8.0% vs. 64.6%, X2(1) = 70.37, p < 0.001), discontinuation of medication (8.8% vs. 75.9%, X2(1) = 91.03, p < 0.001), and a return to ACT (10.3% vs. 47.1%, X2(1) = 34.88, p < 0.001).

Although not shown, the results of the logistic regression models suggested, when controlling for gender, race, diagnosis, and co-occurring substance use, an increase in ATR scores was associated with a lower probability of post-transition hospitalization (β = − 0.09, p < 0.001), discontinuation of treatment (β = − 0.16, p < 0.001), discontinuation of medication (β = − 0.18, p < 0.001), homelessness (β = − 0.16, p < 0.001), and a return to assertive community treatment (β = − 0.06, p < 0.001).

Discussion

The use of standardized measures in routine mental health settings is not new or particularly innovative, and practitioners in community-based mental health settings have been encouraged to use standardized measures for two decades or longer (Ellwood 1988; Huxley 1998; Jenkins 1990; Slade et al. 1999). Measures such as the ATR and the previously mentioned TRS, which uses administrative data and a data algorithm to calculate transition readiness scores for ACT consumers (Donahue et al. 2012), can provide ACT teams additional guidance with respect to identifying clients who could be ready to transition from ACT.

The ATR is a single measure designed to assess many aspects of transition readiness and meets the characteristics of feasibility of standardized measures for routine mental health settings proposed by Slade et al. (1999). In particular, consistent with Slade and colleague’s tenets of feasible standardized measures, the ATR is brief (i.e., only 18 items), simple to use, relevant, and has the potential to be useful in routine mental health settings.

Clinical judgment should and always will be an important part of the decision-making process of identifying clients who are ready to transition from ACT to less intensive services. However, professional judgment can be used along with high quality standardized measures to improve assessments of an ACT participant’s readiness to transition. A standardized measure unique to identifying transition-ready ACT participants can formalize and codify the transition decision-making process by providing guidance to ACT teams concerning relevant information to consider. This is particularly important for new and/or inexperienced ACT teams and staff members. Moreover, a standardized measure of ACT transition readiness can reduce subjectivity and bias inherent in clinical judgment.

Additionally, measures of transition readiness, such as the TRS and ATR, can facilitate communication and accountability because they can provide quantitative information that can be incorporated into assessments and reports. Furthermore, standardized measures of ACT transition readiness can be used as clinical tools for monitoring a client’s progress and facilitate case planning, and as administrative and/or supervisory tools that can focus treatment goals and monitor team performance. Finally, client functioning on standardized measures can be compared among teams across a state or region and with national norms, which could lead to local, state and national standards about transitioning clients from ACT.

This study has a number of limitations that warrant discussion. ACT staff were asked to recall clients whom they judged to have successful and unsuccessful transitions and to think about these clients immediately before they transitioned from ACT. Thus, recall bias and differences in how respondents defined successful and unsuccessful transitions may have affected ATR scores in unknown ways. Additionally, clinician judgment was based on recall without objective measures or data. Another limitation is missing information about each consumer’s time on ACT prior to transition, which could have implications for the interpretation of ATR scores and transition issues. That is, clients who are identified as ready to transition based on ATR scores after only a short time on ACT may present different post-transition challenges and needs compared to a client with the same score after 3 years on ACT.

Although the sample of ACT staff members represented 19 different states, the extent to which the sample is representative of all ACT staff is not clear. Thus, caution in generalizing the results presented here is warranted. In addition, it is important to note that 92% sample of ACT providers who participated in the study were white and the fact that the sample of providers was not more diverse is a limitation of the study. The findings here should be considered in light of the potential for implicit bias towards clients among ACT providers.

Moreover, the sample of ACT staff was restricted to those who had at least some experience in transitioning ACT clients to less intensive services. This was done to ensure that ACT staff participating in the study had sufficient experience and number of ACT clients from whom to select their successful and unsuccessful transitions. Because of this restriction, it is unclear how well ATR scores would predict post-transition outcomes or transition disposition if completed by ACT staff with less experience. Other limitations of the study include the lack of client self-assessment and evidence of other forms of reliability, such as test–retest reliability and interrater reliability for the individual items on the scale.

Implications and Directions for Future Research

Measures such as the ATR and the TRS, in combination with professional judgment and other assessment methods, can be used to assess the strengths and transition readiness of ACT participants. Despite the limitations noted above, scores on the ATR appear valid toward their intended interpretation and use. Additional research is needed to test the psychometric properties of the ATR and this should be done in real time with clients as they transition from ACT to less intensive services. Prospective testing of the ATR could reveal a different factor structure, and the extent to which the ATR predicts other important outcomes, such as mental health functioning, quality of life, and community engagement should be examined, which warrants further research.

Conclusion

The Assertive Community Treatment Transition Readiness Scale (ATR) was designed to assess ACT transition readiness and demonstrates excellent internal consistency reliability and evidence of known-groups and predictive criterion validity. The ATR is a promising, simple to use measure, however, there is a need for more research on the psychometric properties of the ATR, and reliability and validity of of standardized assessment measurement tools for assessing transitions from ACT to less intensive services.