Effectiveness of assertive community treatment regarding reliable change among adolescents

Assertive community treatment (ACT) for children and adolescents with serious mental illness is an important part of the mental health system. It constitutes a middle term between hospitalization and occasional outpatient treatment, and thus complements both types of treatment whenever emergency hospitalization would be too heavy or when standard outpatient treatment would be too light [1]. ACT is also particularly used to provide care to severely ill patients who tend to refuse care or traditional outpatient treatment. Assertive outreach allows ACT to avoid dropouts from treatment and breaks of contact with the healthcare system. Among adult patients, ACT has proved to be efficient, on the one hand, in reducing psychiatric symptoms and length of inpatient stay, and on the other hand, in improving social functioning, adherence to medication and employability [2, 3].

However, a recent systematic review showed that there are comparatively few studies having examined the efficacy of ACT on adolescent populations [4]. Most of them led to consistent results and showed statistically significant improvements after ACT on social and symptomatic fields. One other study highlighted that a newly implemented ACT program designed to care for youth transitioning from adolescence to adulthood led to significant improvements in both objective and subjective outcomes, such as, respectively, limitation of penal consequences or improvement in overall functioning [5]. A randomized controlled trial on drug- and alcohol-dependent adolescents also showed a significant reduction in symptoms for the patients assigned to ACT [6]. Similar conclusions were reached in a study on adolescents at high psychiatric risk who were followed by ACT teams, where post-treatment assessments showed a significant decrease in aggressive and self-harming behaviors [7]. Finally, a Swiss study compared assessments before and after ACT treatment on adolescents with a wide range of severe psychiatric disorders and showed a significant decrease of their social and symptomatic difficulties [1]. Even though all these results seem rather promising, the improvements after ACT are significant only in the statistical sense. Indeed, the statistical tests only point out whether the observed differences after treatment are due to chance alone, but do not imply that the differences are large enough to be noticed by the patient. In other words, a statistical difference between pre- and post-treatment assessments does not necessary mean that this difference is clinically meaningful for the patient.

Clinical significance differs from statistical significance in the sense that the former is based on expectations set by patients, clinicians, and researchers [8], whereas the latter only refers to the probability of rejecting a null hypothesis and does not implies that this evidence is meaningful. Typical expectations regarding a given type of treatment may be that by its end patients regain normative levels of functioning or that their problems are solved. To determine clinical significance, several methods have been proposed [8,9,10]. One of the most frequently used is the reliable change in dex [RCI; 11]. This index allows the comparison of pre- and post-treatment scores by taking into account the expected scores of a normal population and the measurement error of the instruments used. Originally, two conditions were necessary to consider a treatment as being unequivocally effective [11]: (a) the amount of change in outcome measures was to be significantly reliable, that is, the RCI was to be greater than or equal to 1.96 and (b) patients’ scores were to pass from dysfunctional to normative values after treatment. However, the use of both criteria has recently been considered as very conservative and unrealistic for many clinical contexts [12]. The most recommended approach used in recent research is to only consider the first criterion and to moderate its interpretation, that is, to consider values of the RCI greater or equal to 1.96 as a clinically significant improvement rather than as complete recovery. Conversely, when the RCI is smaller than or equal to –1.96, then the patient is considered as having deteriorated during treatment. Because the RCI is widely used in different areas of psychology [13, 14], and because its performances are similar to other indices, it has been recommended in studies on treatment efficacy [15]. Indeed, the use of a single method that has proved to perform well allows for easier comparisons between studies. Despite this, there is currently no study using the RCI to assess the efficacy of ACT on adolescent populations.

The current study

The general objective of our study is to investigate the efficacy of ACT in a natural setting. The first objective of our study is to assess whether ACT has a statistically significant impact on patients’ symptoms and social functioning. We thus expect patients to present less symptoms and an improved social functioning after treatment. Our second objective is then to extend these results by testing whether the impact of ACT exceeds statistical significance and reaches clinical significance. Therefore, we will focus on the RCI of each treatment outcome to investigate whether ACT has clear clinical benefits.

Methods

Participants

All the adolescents followed by the ACT teams in the Vaud canton in Switzerland between 2010 and 2014 were included in the study; none of them were excluded for any reason, so that the sample most closely resembles the usual patients that the teams meet in “real world” situations. This way, the study remains as naturalistic as possible. Descriptive statistics of our sample are provided in Table 1. In total, 179 participants took part in this study (mean age = 15.76 years, SD = 1.76 year), 77 girls (43.02%) and 102 boys (56.98%). Most of them lived with at least one of their parents (104/179, 58.10%), 56 (31.28%) were placed in foster care or in a social care institution, and 5 (2.79%) lived alone or did not have a home address. Approximately one-third of the participants were school dropouts when the ACT was introduced. Among our respondents, 38.26% received social measures, such as help in schooling, curatorship, withdrawal of child custody or of parental authority, or any other type of social help. Eighty-five participants suffered from a previous traumatic experience (such as migration, adoption, physical abuse, parent separation), with most of them (75/85, 88.24%) having had more than one type. Almost half of our participants (89/179, 49.72%) used legal or illegal psychoactive substances, among which 39 (43.82%) used more than one. Due to the naturalistic setting of the study, a set duration of treatment was not fixed in advance. Thus, the mean duration of the treatment was 185.46 days, with a large variation depending on the patients (SD = 136.85). Among the participants with a known history of psychiatric care, about 20 suffered from psychiatric symptoms for less than 1 year (11.7%), 42 of them for 1–5 years (23.46%), and 35 for more than 5 years (19.55%). Twenty-two patients did not have a psychiatric diagnosis because they have been addressed to the ACT teams by non-medical structures (e.g., schools, youth protection services) and because a psychiatrist of the ACT team has not been involved. Patients were assessed at the beginning of the ACT and at discharge by a member of the ACT team. There were no dropouts, but 31 participants did not have complete data at discharge, because they did not want to meet ACT teams again after discharge, end of care was decided by phone, or end of care was not clearly formalized.

Table 1 Descriptive characteristics of participants (N = 179)
Full size table

Intervention

ACT is provided by a multidisciplinary team (composed of psychiatrists, nurses, and social workers) working together and sharing responsibility for each patient (for a detailed description, see [16]). According to the ACT model, the interventions are flexible and follow a “no drop-out” policy, meaning that reluctant patients are still contacted and offered services. The interventions are provided in the usual social environment of the patients (at home, at school, in cafés), with a high frequency of meetings (up to 10 times a week), and involve their families directly in the treatment. Five main types of interventions are provided by the teams: (1) early intervention aims at reducing the duration of untreated disorder and at avoiding its aggravation by promoting a child and adolescent psychiatric monitoring, by setting up medications, and by sustaining professional or school integration (nine patients; 5%); (2) transition case management is provided near the end of an hospitalization and lasts during a couple of weeks after leaving the hospital to make the transition to the return home easier and to avoid a relapse (37 patients; 20.7%); (3) care provided to hardly accessible patients or in refusal of care proceeds from the no drop-out policy of the model and aims at (re)establishing a therapeutic alliance between adolescents at high psychopathological risk and the health care professionals involved in the situation (81 patients; 45.3%); (4) psychiatric assessment in the community consists in assessing the care the patients need in the very structures that host them (e.g., schools, juvenile prisons, social care institutions) and in supporting the expertise of the care network (41 patients; 22.9%); (5) support in socio-educational institutions is provided to overcome the absence of psychiatric homes for adolescents in Switzerland and permits a therapeutic follow-up for the patient as well as supervisions/intervisions for the socio-educational staff (11 patients; 6.1%).

Instruments

Difficulties related to mental illness

The French version of the Health of the Nation Outcome Scales for Children and Adolescents [HoNOSCA, 17, 18] is a short, reliable, and sensitive instrument used internationally to assess a broad spectrum of difficulties associated with mental illness [19,20,21]. More specifically, the HoNOSCA items require the clinician to evaluate the following dimensions: aggression, concentration, self-injury, substance misuse, school difficulties, physical illness, hallucinations, somatic disturbance, emotional issues, difficulties in peer relationships, independence, family relationships, and school attendance. It is composed of 15 items rated on a 5-point Likert scale, ranging from 0 (no problem) to 4 (severe problem). A higher score refers to more severe difficulties. In addition to the Total score, which we calculated by averaging Items 1 to 13 (Gowers et al., 1999), recent research on the factor structure of the HoNOSCA [22] suggested the use of two additional subscores: an Externalizing symptoms subscore (Ext; mean of Items 1, 2, 5, and 11) and an Emotional Problems subscore (Emo; mean of Items 3, 4, 6, 7, 8, 9, 10, 12, and 13). Complementary studies confirmed that the two additional subscores were sensitive measures of clinical improvement regardless of patients’ mental illnesses [23]. As Items 14 and 15 did not assess youths’ functioning, they have not been taken into account in our analyses.

Overall level of functioning

The Global Assessment of Functioning Scale [GAF, 24] is one of the most widely used measures among psychiatric services to evaluate a global level of current functioning, including the severity of psychological symptoms [25]. It was developed as a single-item scale for clinical evaluation of the Axis V of the DSM-IV, with a score ranging from 0 (important functional impairment) to 100 (superior functioning). A score above 80 represents a good to superior functioning and lower scores represent a poorer functioning. It is inexpensive, easy to administer, and applies to all ages, making it an instrument of choice for comparing different populations. Moreover, this scale has an excellent inter-rater reliability and nomothetic validity [26, 27].

Procedure

The instruments were rated at admission and at discharge by the clinician who was in charge of the patient (and who knows him best). The clinician who rated the instruments could be a nurse, a social worker, or a psychiatrist. Each one of them was previously trained to use the instruments.

Statistical analyses

A Multivariate Analysis of Variance (MANOVA) was used to examine the five types of treatment provided by the ACT teams as the independent variable (IV) and four outcomes as dependent variables (DVs): the total score of the HoNOSCA (Items 1–13), its Externalizing and Emotional problems subscores, and the GAF score. We then used linear mixed-effects models (LME) to model the effect of ACT on the four outcome measures. We followed these analyses by calculating a reliable change index (RCI; 11) for each outcome measure and by evaluating how many patients recovered, versus deteriorated, after ACT. The RCI is an index calculated using a statistical procedure to determine a degree of clinical significance that meets the standards of efficacy set by consumers and clinicians. According to Jacobson and colleagues [11], clinical significance is attained during the course of therapy when a patient moves from the dysfunctional towards the functional range. Accordingly, the RCI allows determining whether a given patient has crossed a clinical significance threshold (e.g., a meaningful reduction of symptoms), while simultaneously accounting for measurement error. It is calculated by subtracting the post-treatment score from the pre-treatment score and by dividing the result by the standard error of the differences. An RCI score ≥ 1.96—that is a z-score level of significance of p < 0.05—is then considered as a reliable change. Finally, we used multivariate regressions to check whether treatment duration (IV) was a significant predictor of reliable change for each outcome (DVs). The data were analyzed using R software v3.1.2 [28].

Results

As the ACT teams provide five different types of interventions, we tested whether they had different effects on the outcomes. Because the assumptions of multivariate normality for a classical MANOVA were violated (Shapiro–Wilk tests were all significant, all ps < 0.001), we resorted to robust methods for calculating the MANOVA. Using Todorov and Filzmoser’s [29] method on the ranked data, the difference scores of the outcome variables were not affected by the type of treatment, λ(28) = 0.73, p = 0.48. We therefore conducted all our subsequent analyses without distinction between the types of treatment received by the participants.

Linear mixed-effects models (LME) on random intercepts were used with restricted maximum likelihood to model the evolution over time of each one of the four outcome measures (HoNOSCA-Total, HoNOSCA-Ext, HoNOSCA-Emo, and GAF). Time was considered as a fixed effect in our model, and because a given measure was supposed to be correlated for each patient across time, subjects were considered as random effects. Table 2 shows that all three HoNOSCA scores significantly diminished (i.e., reduction of problems) and that the GAF score increased (i.e., improvement of functioning) across time.

Table 2 Random intercept mixed-effects models of four outcome measures
Full size table

We then calculated a RCI [11] for each outcome measure. Figure 1 shows the amount of change for each patient after treatment. For each outcome, more patients improved than deteriorated. More specifically, the ratio of improvement on deterioration was the best for internalizing symptoms (HoNOSCA-Emo: 30/5 = 6), immediately followed by the overall level of functioning (GAF: 23/4 = 5.75) and general difficulties related to mental illness (HoNOSCA-Tot: 30/6 = 5). The worse ratio was for externalizing symptoms (HoNOSCA-Ext: 20/11 = 1.82). On the whole, 47 (26.27%) patients improved, whereas only 14 (7.82%) deteriorated in at least one of the four outcome measures.

Fig. 1
figure 1

Evolution of each patient after treatment for four outcome variables. Shaded areas represent reliable change index intervals. Triangles represent patients with a clinical improvement after treatment; squares represent patients with clinical deterioration; points represent participants below the critical values of reliable change. Axes of the GAF plot are shifted to keep the recovered patients on top of the plot

Full size image

Finally, as treatment duration varied considerably between patients, we tested whether it was a significant predictor of RCI scores for each outcome. As illustrated by Fig. 2, multivariate regressions showed that treatment duration does not affect clinical change in any of the four outcomes, all bs < |0.001|, all ps > 0.53.

Fig. 2
figure 2

Relationship between duration of ACT and four outcome variables. All regression lines are non-significant (all ps > 0.05). Shaded areas represent Wald confidence intervals

Full size image

Discussion

Despite the proven efficacy of ACT on adult populations in the United States, research in Europe on both adult and adolescent populations has not come to definite conclusions [30]. The few European studies [1, 31, 32] concluding on a therapeutic efficacy of ACT only reported statistically significant changes. Although statistical significance is a good hint of the efficacy of ACT, it does not necessary imply that the treatment results in clinical improvements felt by patients, nor that the considerable financial investment made in ACT leads to tangible results. Our study aimed at evaluating, in a naturalistic setting, both statistical and clinical efficacy of ACT among adolescent populations in Switzerland.

First, our results compared the outcomes of the five different types of interventions proposed by ACT teams in Switzerland (i.e., early intervention, transition case management, care for hardly accessible patients or in refusal of care, psychiatric assessment in the community and support in socio-educational institutions). Indeed, because of different health care systems across Europe, the implementation and structure of ACT teams does not always correspond to the original ACT model. This point has often been raised as a possible explanation for the differences between European and American studies [33]. Because the types of interventions proposed by ACT teams in Switzerland is broader than the those of the original ACT model, it was necessary to test whether any, among their five types of interventions, was significantly more—or less—effective compared to the others. Our results showed no difference in outcomes between the five types of ACT interventions. Thus, even by adding interventions to the original model, their efficacy was similar as long as they were made by ACT teams.

Second, we assessed the improvement in the overall level of functioning (i.e., GAF) and the reduction of difficulties related to mental illness (i.e., HoNOSCA) between admission and discharge from the ACT. Our results showed that both emotional and externalizing difficulties were reduced and that the psychological and social functioning of patients was improved after ACT. Unlike studies in the United Kingdom and in the Netherlands [34,35,36], our results are more in line with previous American and Australian studies on adolescent patients [5, 7]. They also complement and extend the previous results obtained by other Swiss studies using the same type of ACT structure on both adult [37] and adolescent patients [1].

Third, after observing a statistically significant reduction of difficulties and an improvement in overall functioning, we confronted these statistical results to standards of efficacy set by patients and clinicians, by calculating a RCI [8]. To our knowledge, this is the first study on the efficacy of ACT for adolescents that assesses more than statistical significance and that extends its results to clinical significance. It thus constitutes an important and innovative point of our study, all the more so as the outcomes are positive. Indeed, among the patients followed by ACT teams, we observed encouraging recovery rates after treatment. More than a quarter of them (26.3%) passed from dysfunctional to functional levels on one of the outcome measures: 21.4% recovered from general difficulties related to their mental illness (HoNOSCA-Total), 21.4% recovered from emotional difficulties (HoNOSCA-Emo) and 14.1% recovered from externalizing problems (HoNOSCA-Ext). ACT also allowed 17.7% of patients to improve their overall level of functioning (GAF) until attaining levels closer to the functional population. The recovery rates of our sample may seem quite low compared to those of other psychotherapy studies focusing on improvement of very specific traits or symptoms and using the RCI for outcome measurement. For instance, it has been shown that the recovery ratio of callous–unemotional traits of delinquent adolescents following family therapy was between 12 and 48% [38]. In the same way, studies on both adult [39] and adolescent outpatients [40] highlighted that between 20 and 41% of them recovered from their symptoms after psychotherapy. However, as mentioned before, these studies were focusing on very specific outcomes and/or excluded patients with severe mental illness. In contrast to those studies, ACT treated hardly accessible patients, patients in refusal of classical treatment, and patients with severe mental illnesses. Studies including adolescent outpatients with more severe disorders (e.g., conduct disorders, autism spectrum disorders) report symptoms recovery rates among 15–26% of them [41], which is more in the range of what we observed. Moreover, in our study, very few patients clinically deteriorated following ACT, between 3.08% and 7.75%, which is still much lower than the 14–24% of clinical deterioration that have been observed among outpatients in other studies [41]. In our sample, clinical improvement after ACT was between five and six times more frequent than deterioration. Taking into account that adolescent patients followed by ACT teams have often severe and chronic psychiatric symptoms, these clinical recovery rates are rather promising. Our results also point out that ACT teams are more efficient in dealing with emotional symptoms, such as self-injury, addictive behaviors, hallucinations, or anxious and depressive symptoms, and in improving the overall social and psychological functioning levels, than they are regarding externalizing problems, such as disruptive and antisocial behaviors or scholastic skills (e.g., reading or counting). Conversely, the lower impact of ACT on externalizing symptoms may be explained by the fact that disruptive and antisocial behaviors generally need treatments that last much longer than 6 months, which corresponds to the mean duration of ACT. Moreover, according to ACT guidelines, teams are not supposed to try and treat these disorders, but to reorient patients to more containing structures, such as socio-educational institutions or child rehabilitation homes.

Despite these strengths, our study has some methodological limitations. For instance, our results consider ACT as a single, uniform treatment and do not differentiate between the five types of interventions provided (which are not always mutually exclusive). Even if the effectiveness of these five interventions did not differ significantly with regards to our outcome measures, it would be interesting for future studies to identify more precisely which one is most effective, particularly in terms of clinical significance. Another limitation would be that we did not formally test inter-rater agreement between clinicians in this study. Because they were trained to use the HoNOSCA and the GAF, we inferred that their ratings would be accurate. Finally, as the time of post-treatment assessment was set at discharge from the program, one may expect that patients are discharged only when they have recovered. From this point of view, the efficacy of ACT teams in Switzerland may seem low as recovery rates much higher than 25% should be expected. However, the main objective of ACT is not to help patients recover from mental illness, but rather to help them to get proper treatment and to prevent deterioration. Thus, an improvement of 25% of patients’ functioning when only stabilization is expected may be considered as a pretty positive outcome.

Conclusions

Contrary to other European countries, ACT teams in Switzerland proved some efficacy in dealing with severely ill patients or with patients in refusal of care. They also enable a substantial percentage of patients to recover from their difficulties, while also enabling those who do not respond well to their treatment to be reoriented to appropriate care facilities. The specificities of their interventions—in particular the fact that they propose a broader range of interventions than what is proposed in the original ACT model—may be the factor explaining why their therapeutic success is higher than in other countries. Moreover, ACT teams in Switzerland seem to complement quite well other care structures, with which they pursue positive collaborations. This may create a synergetic effect that could be a possible explanation of their successful results in Switzerland. Future studies should probably investigate the interrelations between different care structures to identify the factors that make ACT successful depending on different countries.