Introduction

Children and adolescents with autism spectrum disorder (ASD) are psychiatrically hospitalized at significantly higher rates than children with other developmental or psychiatric disorders and have longer lengths of stay than patients without ASD (Croen et al. 2006; Kalb et al. 2012; Siegel and Gabriels 2014). Caring for youth with ASD in an inpatient setting is challenging as the inpatient milieu and brief interventions are frequently inappropriate for patients with significant communication and/or social challenges (McGuire et al. 2015). The majority of research on interventions to improve outcomes in inpatient psychiatric care for this population occurs in specialized settings, which are designed solely for inpatients with ASD and related challenges (Siegel 2018).

Previously, an evaluation of a care pathway for youth with ASD (Autism Spectrum Disorder Care Pathway; ASD-CP) who were hospitalized in a general psychiatric inpatient unit at a public hospital demonstrated that staff training and specialized treatment tools can be successfully implemented to improve patient care (Kuriakose et al. 2018). The initial evaluation of the ASD-CP was associated with a clinically important but statistically nonsignificant 40% decrease in average length of hospital stay as well as a clinically and statistically significant decrease in use of crisis interventions (i.e., holds and restraints). The initial evaluation showed promise in improving outcomes for this vulnerable population within non-specialized settings. Given the very limited access to specialized inpatient units across the nation, more work in this area, and specifically on the sustainability of such outcomes, is needed (Siegel 2018).

The literature on inpatient psychiatric interventions shows that many have initial success (Lean et al. 2015; Stange et al. 2003). However, in comparison to the evaluation of initial implementation, the sustainability of such programs is evaluated far less frequently (McHugh and Barlow 2010; Novins et al. 2013; Wiltsey Stirman et al. 2012). It is possible that sustainability of outcomes may be impacted when external trainers or consultants on the initial implementation team have withdrawn and enthusiasm surrounding intervention launch may have waned. Overall, data regarding the sustainability of inpatient mental health interventions for children and adolescents is lacking, and little is known about the long-term stability of interventions with initial promising results.

Some research on milieu interventions has shown that sustainability of outcomes within inpatient psychiatric settings is possible. For example, a 10-year evaluation of an inpatient intervention consisting of trauma-informed training of staff, changes to the therapeutic milieu, and hospital-wide policy changes, showed sustained improvement in patient/staff communication and reduced holds and restraints (Madan et al. 2014). The six-year evaluation of a partnership to improve quality of care and rehabilitation in psychiatric hospital settings showed that hospitals had maintained programming and continued progress in patient care and staff engagement (Birkmann et al. 2006). However, in another intervention targeting patient engagement in unit activities, outcomes were not sustained after the initial implementation, and the benefits and progress initially made by these interventions fell back to baseline (Lean et al. 2015). Overall, information regarding the sustainability of evidence-based inpatient mental health treatments is scarce, and data regarding the sustainability of generalized inpatient care systems specific to the autism population are not available.

Therefore, it is important to examine whether the initial positive results of the ASD-CP were sustained. The current study replicated analyses conducted in the initial paper (Kuriakose et al. 2018) and compared the outcomes of patients receiving the ASD-CP during an 18-month follow-up period to the outcomes of patients from the 18-month pre-implementation and 18-month initial evaluation periods, to assess maintenance of positive effects.

Methods

Methods have been previously described in detail in the initial evaluation of the initiative (Kuriakose et al. 2018).

Study Participants

Participants for this study were sampled from a larger population of children presenting to the pediatric psychiatric emergency department at a public hospital in a major metropolitan city in Northeastern United States. Of note, a majority of the patients ( ~ 75%) served at this hospital receive services funded through Medicaid. Participants for the current study included patients 4–17 years old who met inclusion criteria across three 18-month time periods: January 2014 to June 2015 (before the implementation of the ASD-CP) and July 2015 to December 2016 (the initial evaluation period) and January 2017 to June 2018 (the follow-up period).

Inclusion criteria for the study were as follows: (1) Discharge diagnosis of ASD; (2) nonverbal or minimally verbal expressive communication level (i.e., only using single words or short phrases); (3) 1:1 staffing assigned by the admitting physician due to low levels of adaptive functioning and/or high levels of disruptive behavior; (4) first-time inclusion in the ASD-CP (patients who had previously presented to the emergency department and who had received the ASD-CP were excluded from the current study). These inclusion criteria were identified for several reasons. First, patients with no-to-minimal verbal language and severe challenging behavior typically have greater autism symptom severity making adapting to the non-specialized inpatient environment more difficult. These youth who are more likely to exhibit severe challenging behavior are also at increased risk for receiving crisis interventions (Matson and Boisjoli 2009). Next, the ASD-CP involves several layers of evidence-informed intervention; therefore, feasibility of implementation is improved when patients are assigned to receive 1:1 care. Lastly, this subpopulation is aligned with those receiving care at specialized units and research attention through the Autism Inpatient Collection (Siegel 2018), thus allowing for comparison of results across studies.

Participants were grouped according to the period of service. The pre-implementation (PRE) group consisted of those who received services in the pre-implementation period. The initial evaluation (IE) group were youth who received the ASD-CP during initial evaluation period; and, the follow-up (FU) group were youth who received the ASD-CP following the completion of the initial evaluation period. The PRE and IE groups were previously described (Kuriakose et al. 2018); sociodemographic information for these groups are presented in Table 2. In the follow-up period, there were 2006 youth who presented to the pediatric psychiatric emergency service (the Children’s Comprehensive Psychiatric Emergency Program [CCPEP]). About seven percent (n = 140) had a diagnosis of ASD; 71.43% (n = 100) of those patients did not require admission and were released. Of the remaining 40 patients, 62.50% (n = 25) did not meet inclusion criteria for the study (e.g., were readmissions, did not require 1:1 support, were fluently verbal). Therefore, of the children with ASD who were admitted to the unit, 37.50% (n = 15) received the ASD-CP and were included in the FU group. The FU group was entirely male; approximately 67% were between the ages of 4–12 years old and 33% were between 13 and 17 years old. No participant in this group had two or more comorbid psychiatric diagnoses; about 47% had no other diagnosis and 53% had one comorbid diagnosis. Approximately 87% of participants spent 1 day or less in the brief-stabilization unit and all participants were admitted to the inpatient unit following observation (see Table 2).

Care Pathway Setting and Components

The ASD-CP was implemented within the pediatric psychiatric acute care program of a metropolitan public hospital. The acute care program includes a CCPEP and three child and adolescent psychiatric inpatient units with a total of 45 beds. After patients are evaluated in the CCPEP upon presenting to the program, they may be discharged if they do not require further observation and/or stabilization, admitted for observation to a six-bed brief-stabilization unit that is part of the CCPEP, or directly admitted to one of the three psychiatric inpatient units. Regulation permits patients who are admitted to the brief-stabilization unit to be held for up to 72 h, after which time, the patients are either discharged to outpatient care or admitted to one of the psychiatric inpatient units.

The ASD-CP consisted of a four-module staff training, toolkit to aid in implementation, and specified strategies to be utilized with the patient. The staff training was structured as four 45-min modules. Instructional strategies included a variety of approaches, including lecture, video examples, interactive exercises, and role-plays. The training was provided by supervising psychologists or nurses across care settings (i.e., CCPEP and inpatient units) to staff of varying disciplines (e.g., technicians, nurses, attendings). The training was also provided for all new staff across both the initial evaluation and follow-up periods. Content included an overview of ASD and preventative strategies such as establishing predictability, appropriate activities, multi-modal communication, and positive reinforcement, as well as agitation management. Tools including a brief specialized assessment checklist and visual supports to increase functional communication between staff and patients are also part of the intervention package. For a detailed list of tools, see Table 1 or refer to Kuriakose et al. (2018). During both the initial evaluation period and the follow-up period, a clinical psychologist and a trained research assistant were available part-time in the CCPEP and inpatient units to address questions regarding the implementation of the ASD-CP as they arose.

Table 1 Components of the Toolkit
Full size table

Procedures and Measures

A trained research assistant abstracted sociodemographic and care utilization data from patient medical records; 55% of the records were reviewed by an independent rater in order to ensure accuracy. Study variables included age, gender, presence of comorbid diagnoses, amount of time spent in the brief-stabilization unit, brief-stabilization discharge disposition (i.e., discharged to outpatient care or admitted to psychiatric inpatient unit), inpatient length of stay, total length of stay (i.e., brief-stabilization unit stay + inpatient stay), number of holds/restraints in brief-stabilization and/or inpatient units, and number of intramuscular (IM) medication administrations.

Statistical Analyses

Statistical analyses were performed using SPSS Statistics Software (Version 25). First, apriori chi-square tests were conducted to examine potential differences among PRE, IE, and FU groups on demographic (i.e., gender, age group, presence of comorbid conditions) and clinical factors (i.e., disposition following brief-stabilization unit observation, length of brief-stabilization unit stay).

Next, several clinical outcome variables were compared across the three groups to examine whether improvement was sustained. To replicate the analyses from the initial study (Kuriakose et al. 2018), variables of interest included inpatient and total length of stay, number of holds/restraints used within the inpatient and brief-stabilization unit settings, total number of holds/restraints, and total number of IM medication administrations; tests of normality revealed non-normal distributions across groups for each of these variables. Notably, the violation of the normality assumption appeared to be more severe from the initial study with the addition of the data from the FU group. Therefore, nonparametric analyses were selected and a series of Kruskal–Wallis tests were conducted. The Kruskal–Wallis test is a rank-based test for comparing more than two independent variables and, while less powerful than the corresponding parametric one-way ANOVA, does not make assumptions about normality (Ostertagova et al. 2014). Within a rank-based test, rank positions are calculated from scores without consideration of which group that score belongs (i.e., the lowest score across the sample is assigned a rank of 1, the next highest a rank of 2, etc.); these ranks are then used as comparison values in the analyses (Field 2009). Therefore, because the total sample size is 52, scores on continuous outcome variables will be provided ranks ranging from 1 to 52.

Following the Kruskal–Wallis tests, to further understand any significant differences found between groups, post-hoc Mann–Whitney tests were performed; in these cases, a Bonferroni correction was applied. The Bonferroni-adjusted significance level was set at 0.017 (i.e., p of 0.05 divided by a total of three comparisons). Finally, differences in the presence of any holds/restraints across inpatient and brief-stabilization unit settings among groups were evaluated with a chi-square test. Again, to account for the familywise error rate associated with pairwise comparisons, a Bonferroni correction was applied to post-hoc chi-square tests (Field 2009). Because there are again three comparisons, the Bonferroni-adjusted significance level was set at 0.017.

Results

In regard to apriori analyses, chi-square tests revealed no significant differences across groups in regard to gender, age, presence of comorbid psychiatric diagnoses, disposition following observation in the brief-stabilization unit, or days spent in the brief-stabilization unit (all p > 0.05). Refer to Table 2 for sociodemographic information.

Table 2 Sociodemographic characteristics
Full size table

A series of Kruskal–Wallis tests were then performed to identify group differences in inpatient and total length of stay, number of holds and restraints used within the inpatient and brief-stabilization unit settings, total number of holds/restraints, and total number of IM medication administrations. There were no statistically significant differences in the number of holds/restraints during the inpatient stay or in inpatient length of stay or total length of stay among groups (all p > 0.05). Further, in contrast to the clinically important but statistically nonsignificant 40% decrease in length of stay found in the initial study (Kuriakose et al. 2018), clinically relevant reductions in length of stay were not observed in the follow-up period. However, significant differences among groups were found in number of holds/restraints in the brief stabilization unit, H(2) = 8.35, p = 0.02, total number of holds/restraints, H(2) = 7.35, p = 0.03, and total number of IM medication administrations, H(2) = 8.91, p = 0.01 (Table 3).

Table 3 Comparison of length of stay, holds/restraints, and IM medication administration
Full size table

To further evaluate these findings, post-hoc Mann–Whitney tests were conducted. Following Bonferroni correction, where the adjusted significance level was 0.017, statistically significant differences were found between the PRE and FU groups in total number of holds/restraints across settings (U = 69.50, p = 0.012, r =  − 0.45), number of holds/restraints in the brief-stabilization unit (U = 82.50, p = 0.013, r =  − 0.44), and total IM medication administrations (U = 63.50, p = 0.004, r =  − 0.51). Group differences were not found between the PRE and IE groups or the IE and FU groups (all p > 0.05). See Table 4.

Table 4 Follow-up Mann–Whitney test results
Full size table

Lastly, there were significant differences in the proportion of children experiencing any hold/restraint across brief-stabilization unit and inpatient settings, X2 (2) = 8.37, p = 0.02. Follow-up pairwise comparisons, again using the adjusted significance level of 0.017, revealed that a smaller proportion of participants in the FU group experienced a hold/restraint than in the PRE group, X2 (1) = 7.04, p = 0.01 (Table 5). The reduction in youth experiencing holds/restraints approached significance when comparing the IE group to the PRE group, X2 (1) = 4.36, p = 0.04. Proportion of youth experiencing any holds/restraints did not differ between the IE and FU groups (p = 0.39).

Table 5 Number of youth with any holds/restraints across brief-stabilization and inpatient units
Full size table

Discussion

Few data are available concerning the sustainability of inpatient mental health treatments; this is particularly true for the autism population, which has long been frequently yet inadequately served within general psychiatric units. The initial study of the ASD-CP revealed promising preliminary results; the ASD-CP was implemented within the general psychiatric service of a public hospital with limited resources and was associated with clinically important but statistically nonsignificant reductions in length of hospital stay and statistically significant reductions in presence of holds/restraints (Kuriakose et al. 2018). Results from this follow-up study suggest that improvements in use of crisis interventions were sustained in the 18 months following the initial evaluation period while the nonstatistically significant trend towards decreased length of stay did not.

Continued decreases in use of crisis interventions, such as holds/restraints and IM medication administrations, were observed across initial evaluation and follow-up periods. Statistically significant reductions in the total number of holds/restraints used in the brief-stabilization unit and across settings and in IM medication use were found only in the FU group compared to the PRE group, with moderate to large effect sizes. These results are encouraging, as use of both physical and pharmacological restraint often carries risk for significant negative consequences. First, there are substantial safety risks associated with the use of crisis interventions for the patient and the staff. Due to the chance for resulting injuries, these interventions can also be costly. Further, reduction in the use of restraint is a priority area for hospital quality management and of even greater concern in the vulnerable population with developmental disabilities. Most notably, the use of crisis interventions limits patient learning; when patients are denied the opportunity to develop replacement behaviors, maintenance and generalization of reductions in challenging behavior from the inpatient unit to less restrictive settings are unlikely (Kuriakose et al. 2018; Sturmey 2018). Therefore, by reducing crisis interventions, patients may be better able to develop adaptive replacement behaviors and transition more smoothly out of inpatient care. Ideally, if the adaptations generalized to community settings, this leads to lower recidivism rates.

Unfortunately, reductions in length of stay were not apparent across initial evaluation and follow-up periods. In fact, length of stay in the FU group approached that of the PRE group. The mechanisms underlying these length of stay changes across the three time periods (i.e., the decrease in the initial evaluation period and subsequent increase in the follow-up period) are unknown and may be in part due to external factors that were beyond the control of this real-world effectiveness study, such as number of beds available, staffing shortages/turnover, hospital bed closures, and complexity of patients who did not receive the ASD-CP on the unit at any given time. Additionally, length of stay in this population is likely connected to disposition options; patients may be clinically stable on the unit but without appropriate settings to return to. Limited data were available to determine the influence of these possible confounding variables; however, from those data that were accessible (i.e., length of stay for patients outside the ASD-CP sample, substantial hospital bed closures), we could identify no plausible explanation for the changes observed in length of stay across periods. It is also important not to rule out the role that chance may have played in the fluctuations in length of stay, particularly given the study’s small sample size. Further work is required to identify alterations to the ASD-CP that may more effectively and consistently reduce length of hospital stay. In addition, systematic collection of data related to these external factors would be beneficial.

Given the current results, we hypothesize that the impact of the ASD-CP is likely due to a milieu change, in which the self-efficacy and attitudes of the staff toward patients with ASD changed, subsequently improving patient care and outcome. This is further supported by an initial look at fidelity data, which will be examined formally in the future. Specifically, while it is unclear how the tools and strategies that did not require formal documentation were implemented or maintained over time (e.g. coping strategies, motor activity, use of simplified language), the process of data abstraction from medical records revealed inconsistent use of ASD-CP tools that required staff documentation (e.g., staff schedules, patient reward system). Further, while new staff were trained in the ASD-CP across initial evaluation and follow-up periods, there were no organized booster trainings for or formal fidelity checks on retained staff. However, despite decreased documented use of tools and lack of booster training, clinically and statistically significant maintenance in the reduction of crisis interventions is evident across time. Therefore, while decreasing reliance on and use of crisis interventions should remain at the forefront of our efforts, we look forward to future work to better understand the reasons underlying the changes observed in the initial and current studies.

In regard to future directions, following comprehensive assessment and evaluation of fidelity and the impact of fidelity on patient outcome, research should and will continue to improve supports available for staff to increase implementation rates of essential care pathway strategies. As discussed in the previous study (Kuriakose et al. 2018), long-term outcomes for individual patients who receive the ASD-CP also need to be studied. Assessment of if and how the care pathway improves utilization trajectories including rate of readmission and success in their transition to less restrictive outpatient care settings would provide strong indication for the utility of the ASD-CP within general psychiatric inpatient units. Further, upon identification of active ingredients within the ASD-CP Toolkit, adaptations could also be made to increase feasibility and encourage implementation with youth with ASD who do not meet criteria for the ASD-CP but still require modification to treatment as usual. This would allow for improvements to ASD inpatient care across a broader range of the ASD population.

This study is not without limitations. As mentioned, there were no systematic data collected on fidelity or technical assistance to aid in staff implementation. Therefore, our ability to make data-informed hypotheses about why improvements in the use of crisis interventions were sustained and the clinically significant but statistically nonsignificant reductions in length of stay were not sustained is limited. We are also limited by the breadth of data that can be captured given the nature of this real-world effectiveness study. Specifically, as in the length of stay analyses, there may also be outside variables that were not accounted for contributing to the crisis intervention results observed, including level of staff experience, availability of oral PRN medications, dose and dose changes of standing and IM medications, and duration of hold/restraints. However, provided that the general patient population experienced a less dramatic reduction in the presence of crisis interventions (i.e., 24.20% in PRE, 21.90% in IE, and 12.27% in FU for the general population compared to 58.82% in PRE, 25.00% in IE, and 13.33% in FU for the ASD-CP sample) and would have experienced similar fluctuations in these uncontrolled variables, observed changes are likely attributable to the ASD-CP. Moreover, although there may be outside efforts contributing to the general downward trend in use of crisis interventions, given that all staff were trained on the ASD-CP, the potential milieu change following initiation of the ASD-CP may have influenced the trend in the general population. Of course, greater precision regarding the contribution of the ASD-CP to study results can only be obtained through fidelity data on staff adherence to the intervention package and comparisons across patients receiving inpatient care that have ASD and are on the ASD-CP, have ASD and are not eligible for the ASD-CP, and do not have ASD. Our current efforts are focused on gathering these data and elucidating these relationships.

We also continue to be restricted by a small and extremely heterogeneous sample, evidenced by the sizeable standard deviations found across outcome variables. Control for this heterogeneity is difficult to achieve because data are not available on participant functioning (e.g., intelligence quotient [IQ], adaptive functioning, operational definitions of intensity and frequency of presenting problems). Further work on how these data can be collected while maintaining feasibility for settings with limited resources is required, as understanding how youth who present differently perform on the ASD-CP would allow for further personalization of treatment. Lastly, to increase confidence in the improvements associated with the ASD-CP, a multi-site study is necessary.

Despite the work that remains to be done, data continue to support the implementation of a specialized care pathway for youth with ASD served in generalized psychiatric settings. The ASD-CP was implemented across a 3-year period using limited resources, and the sustained reductions observed in use of crisis interventions are encouraging and important. However, following a formal evaluation of fidelity and patient outcomes, changes may be necessary to increase utility, particularly in regards to decreasing length of hospital stay. It is clear that youth with ASD have significantly increased risk for psychiatric hospitalization. Given the limited number of specialized units available, clinical and research attention should continue to focus on the development of strategies for more appropriate care in generalized settings. These strategies should emphasize not only efficacy but feasibility and cost-effectiveness.