Introduction

Post-school outcomes for students with autism spectrum disorder (ASD) are typically limited (Certo et al. 2003; Friedman et al. 2013). Employment rates range between 4 and 12% (Taylor and Seltzer 2011) and are lower than adults with other disabilities. Community living and social outcomes are also poor. Compared to students representing 11 other disability categories, individuals with ASD had the second lowest percentage of living independently (12%), were more likely to be living with parents and had the lowest percentage of friendships (48%) (Anderson et al. 2014), even after controlling for level of functioning. Those who did achieve independent living tended to be White, higher functioning, and from higher household income.

To help address concerns about post-school outcomes, in the U.S., federal law mandates that students with disabilities who attend school receive specialized instruction and related services, including transition services as outlined by the Individualized Education Program (IEP) for students with disabilities (Individuals with Disabilities Education Act 1990). A transition IEP is meant to be a results-oriented plan for obtaining post-secondary goals. The planning process should begin by the student’s 16th birthday (and even earlier in some states) and allows students with disabilities to stay in school until they earn a regular diploma or until their 22nd birthday.

Despite this federal law mandating transition planning, of the limited research on transition planning, Cameto et al. (2004) reported that one-third of students with ASD had no transition plan and one-third of parents wanted more involvement in the IEP and transition planning process. Cameto also reported that only about one quarter of parents felt that the transition planning process was very useful. Emblematic of the problem, nearly one-quarter of students with ASD did not attend their own IEP meetings (23%), which is considerably poorer compared to students with a learning disability (4%), speech/language impairment (7%), or intellectual disability (11%). Lipscomb et al. (2017) confirmed results that transition-age students with ASD were less likely to have participated in the development of their IEPs than students with other disabilities; they also found that students with ASD were rated as less likely to be able to perform activities of daily living well (17%) in comparison to all students with disabilities (46%), and experienced lower parental expectations for post-secondary education compared to other students with disabilities.

One reason for these unsatisfactory results is the lack of evidence-based transition interventions to improve post-school outcomes. Moreover, there is no transition planning process with empirical support (Wehman et al. 2014)—a research gap consistent with conclusions from a recent congressional report that only 2% of ASD research focused on transition and adult issues (U.S. Department of Health and Human Services 2017). Thus, as noted by the Interagency Autism Coordinating Committee (2012) studies of empirically-supported approaches for improving transition planning and outcomes for students with ASD are critically needed.

Despite these limited outcomes and gaps in research, data suggest that individuals with ASD can lead productive, satisfying, and meaningful lives, integrated into their communities and working competitively (Ruble and Dalrymple 1996). However, because competence looks different across the spectrum of autism, transition planning needs to be individualized, context-specific and grounded in an Evidence-Based Practice in Psychology (EBPP) approach (McGrew et al. 2016). The EBPP approach avoids sole reliance on the evidence based practice (EBP) and explicitly accounts for consideration of the student and family characteristics as well as the instructional context and teacher resources and skills necessary for effective intervention planning and implementation.

To help improve transition planning and outcomes, we applied an implementation science framework to adapt a consultation intervention (Ruble et al. 2018; Ruble and McGrew 2015) called the Collaborative Model for Promoting Competence and Success (COMPASS; Ruble et al. 2012b) that has been successful in improving attainment of IEP goals in pre-school and elementary school settings. COMPASS has been tested in two randomized controlled trials (RCTs) of young children with ASD, with large effect sizes (Cohen’s d = 1.5; 1.4; Ruble et al. 2010, 2013). Importantly, COMPASS also was effective in extending services to geographically isolated areas through videoconferencing (d = 1.1), directly addressing the need for improved service access and delivery for those in underserved areas.

We used implementation science to guide our process of adapting COMPASS for the transition age population (see Ruble et al. 2018; Snell-Rood et al. 2018). Implementation science concerns the study of methods to promote the use of EBPs into routine care settings (Proctor et al. 2009). As noted earlier, EBPs are infrequently used in special education. However, consultation is an ideal strategy for embedding evidence-based decision-making in selecting and implementing evidence-based teacher interventions in the classroom (McGrew et al. 2016). When adapting a consultation practice, it is important to realize that it is a multilevel EBP such that its impacts on the student are indirect, as illustrated by the three interconnected levels of the Framework for Evidence Based Implementation and Intervention Practices (FEBIIP; Dunst and Trivette 2012; Fig. 1). Specifically, the implementation practice (i.e., consultation quality), should impact the intervention practice (i.e., teaching quality), which should then impact the practice outcome (i.e., student goal attainment). This multilevel model and the indirect impact of COMPASS on student outcomes was confirmed using serial mediation showing that teaching quality and student engagement sequentially mediated the relationship between consultant quality and student IEP outcomes (Wong et al. 2017).

Fig. 1
figure 1

Framework for evidence based implementation and intervention practices (Dunst and Trivette 2012)

Full size image

Accordingly, we used the FEBIIP framework as our guide in adapting COMPASS, and conducted a series of focus groups with stakeholders who identified areas important for good transition planning and implementation for students with ASD (Snell-Rood et al. 2018) and applied this information to our adapted protocol for COMPASS for transition. As in the original version of COMPASS, the adapted version begins with an initial parent-teacher consultation session that lasts 3 h. As is true for young children, parent participation for transition is critical in COMPASS. For many individuals with ASD, parents and caregivers are lifelong advocates, know the individual with ASD better than anyone else, and serve as key supports for maximizing competence across life skills that continue long past school age. Parents participation was especially important for the targeted adaptation of COMPASS for transition youth because they often are the sole implementers of plans for obtainment of postsecondary goals. For example, student postsecondary goals of driving, working, attending college, accessing transportation for people with disabilities, using Medicaid Waiver services, getting Social Security Disability Income, etc. necessitated parent support. Unlike COMPASS for young children, students with ASD were invited to participate in this targeted adaptation of COMPASS.

Prior to the consultation, all participants complete a COMPASS profile that includes questions about the student’s self-management, adaptive, communication, social, and learning/work behavior skills as well as sensory preferences and avoidances. When possible, students with ASD completed a first-person version of the COMPASS profile questionnaire (36% of students in this study). The information shared during the discussion of the profile supported the identification of a social, communication, and work skill goal as well as the student’s personal and environmental challenges and supports that relate to the attainment of the goal. This information was then used to develop a personalized teaching plan for each goal. Because the targeted adaptation of COMPASS considered post-secondary outcomes, parents and the student, when possible, answered additional questions about post high school goals such as where the student will be living, how she/he would spend her/his day, how the student will move about in the community, friendships, and other activities. Thus, following the development of the teaching plans, plans were also generated for the accomplishment of post-school goals. After this initial consultation, the consultant met with the teacher, the student when possible, and the caregiver for four coaching sessions that each lasted between 60 and 90 min. During the coaching sessions, we reviewed data on the student’s progress toward the goals as well as the strategies to meet the goals. We problem-solved any issues related to accomplishment of the goals. See Ruble et al. (2018) for more detail about the adaptation process applied to COMPASS.

The purpose of this paper is to report our empirical findings from our RCT of the targeted adaptation of COMPASS for transition age students with ASD. We had one primary and two secondary research questions: (a) Does COMPASS for transition planning and implementation improve IEP goal attainment outcomes for students with ASD?; (b) What degree of fidelity did consultants achieve in delivering the modified version of COMPASS?; and (c) How well did teachers adhere to evidence-based instruction in COMPASS? Based on prior RCTs (Ruble et al. 2010, 2013), we hypothesized that the COMPASS group would have improved IEP outcomes compared to a comparison group. We also hypothesized that COMPASS could be implemented with high fidelity and that teacher adherence would improve over time.

Method

Participants and Sampling

Teachers

Twenty special education teachers were randomized into COMPASS or a comparison group. All teachers were certified, indicating that they were credentialed by the state department of education as special educators. All but three were female; 10% had a BA, 85% had an MA, and 5% had a doctorate.

Students with Autism

All students received special services under the educational category of autism (Individuals with Disabilities Education Act 2004) and met the Diagnostic and Statistical Manual IV-TR/5 definition of Autistic Disorder (American Psychiatric Association 2004, 2014) as confirmed by the Autism Diagnostic Observation Schedule—second edition (Lord et al. 2012). Depending on the overall language, two students were administered Module 1 of the ADOS, one received Module 2; one received Module 3; and the remaining 16 received Module 4. Those who were administered Modules 1, 2 or 3 had overall scores ranging from 13 to 29. Overall scores for those who were administered Module 4 ranged from 6 to 23. Higher scores indicate higher autism symptomatology. Students’ ages ranged between 17 and 20 years, with a mean of 18.2 years (SD = 1.1). Ninety percent of the students were male, 70% White, 15% Black, 5% Asian, and 10% multi-racial. For families, 15% had incomes less than $10,000; 10% between $10,000 and $25,000; 35% between $25,000 and $49,999; 25% between $50,000 and $100,000; and 15% more than $100,000. The study was IRB approved.

Sampling

Teachers were recruited from public schools from one Midwestern and one Southern state. After permission was granted at the district level, the researchers contacted teachers directly via email or phone. One student participant with ASD was randomly selected from each teacher’s class or caseload. Between August 2015 and November 2016, a total of 150 teacher-child pairs were assessed for eligibility (see Fig. 1). Of those teacher-student dyads who met inclusion criteria, 18.1% participated, 60.9% did not respond, and 20.9% refused. Overall, 20 teachers and 20 students and their parents participated. Following a baseline Time 1 assessment, teacher-child dyads were randomized into groups; 11 were randomized into the experimental condition. Seven teacher-student pairs were recruited in Year 1 and 13 in Year 2. The comparison group of teachers (n = 9) received online training on three evidence-based practices in autism of their choosing from the National Technical Assistance Center on Transition website (Fig. 2).

Fig. 2
figure 2

Consort flowchart

Full size image

Student Measures to Establish Sample Equivalency

Measures using multiple informants were administered at Time 1 to establish group equivalency. Cognitive level was evaluated directly using the Kaufman Brief Intelligence Test, Second Edition (Kaufman and Kaufman 2004). Adaptive behavior was assessed with the survey interview form (parent report) and the teacher rating form of the Vineland Adaptive Behavior Scales, Second Edition (VABS II; Sparrow et al. 2005, 2006). Social, emotional, and behavioral functioning based on both parent and teacher report was evaluated with the Behavior Assessment System for Children, Second Edition BASC-2; (Reynolds and Kamphaus 2004). Autism severity was assessed by the researchers using the standard or high-functioning versions of the Childhood Autism Rating Scale, Second Edition (Schopler et al. 2010).

Assessment of Treatment Fidelity

To ensure treatment adherence three measures were administered and evaluated.

Consultant Fidelity to the Initial COMPASS Consultation Protocol

Consultant adherence to the initial consultation protocol was assessed using a 35-item close-ended (yes/no) checklist completed by parents (KR20 = .95), teachers (KR20 = .85), and an independent rater (researcher) (KR20 = .74). Scores from all three informants were used in the analyses.

Consultant Fidelity to the Follow-Up Coaching Protocol

The second fidelity measure assessed consultant adherence to the coaching protocol. We applied a 17-item checklist (yes/no) completed by teachers and independent raters (KR20 = 1.0) unaware of intervention goals. Scores from both sources were used in analyses.

Teacher Adherence

Immediately following each coaching session, two raters (the primary consultant and the secondary consultant) assessed the degree to which the teacher followed the teaching plan recommendations using a 5-point Likert-type scale item ranging from 1 (none of elements implemented) to 5 (about 100% of elements implemented). Estimated percent agreement and ICC (2-way random effects model, single rater agreement) were .91 and .97, respectively. Scores from the primary informant were used in the analyses.

Psychometrically Equivalence Tested Goal Attainment Scaling (PET-GAS)

Because each student had different goals, baseline skill levels associated with the goals, and teaching plans, idiographic assessment using PET-GAS (Ruble et al. 2012b) was used to evaluate IEP progress. PET-GAS incorporates several procedures to ensure high quality, comparable, and objective goal attainment assessment. Each goal attainment scale was created using the following 5-point rating scale: − 2 = student’s present levels of performance, − 1 = progress, 0 = expected level of outcome by the end of the school year, + 1 = somewhat more than expected, + 2 = much more than expected. Half-scores were allowed when raters observed skill levels between two benchmarks. A score of zero represented improvement consistent with the actual description of the written IEP objective. PET-GAS pre- and post-treatment ratings were based on video demonstrations, work samples, and/or data collected by the teacher. Two coders independently coded 65% of the goals at baseline and three coders independently rated 35% at final evaluation. Interrater agreement (two-way Random) as measured using the sample ICC for single measures was .94 at baseline and .86 at final evaluation. The primary rater scores were used for analyses.

In a second validation step, we assessed between group goal comparability by applying a 3-point ordinal scale to code three features of each goal: (a) goal measurability (i.e., the degree to which descriptions include prompt level, criterion for success, and an observable skill); (b) goal interval equidistance (i.e., the degree to which benchmark descriptions are equilibrated and scaled appropriately); and (c) goal difficulty (i.e., the degree to which the present levels of performance indicate that the child is completely unable to perform skill with anyone, anywhere, or with any prompts compared to what is written in the objective) (see Ruble et al. 2012b). Two raters independently coded 30% of the baseline GAS forms for the three features. Rater percent agreement was .94 for difficulty, measurability, and equidistance. Because raters matched perfectly on 15 of 16 ratings, ICCs (2-way random effects model, single rater agreement) were not computed. Two raters also independently coded 20% of the final GAS forms. Estimated sample percent agreement and ICC (2-way random effects model, single rater agreement) were .90 and .88, respectively. A between-group analysis using independent t tests revealed no differences between control and experimental condition goals on the three psychometric features of the GAS quality ratings at either baseline or final assessment.

Intervention

The intervention consisted of a 3-h parent-teacher consultation and four 1–1.5 h coaching sessions. All consultations were conducted in-person at the school within the first semester of the school year. The COMPASS consultation intervention is manualized (Ruble et al. 2012b). A shared decision-making approach was used for goal selection and intervention planning. Prior to consultation, students when available (n = 4), parents, and teachers completed a COMPASS profile assessment questionnaire, which was collected and summarized into a joint form used for discussion of the student’s personal and environmental challenges and supports associated with social, communication, and independent skills at school and home. Teachers were asked to update the student’s IEP with the new goals identified in the consultation so that they would be reflected in the student’s educational program. Of 11 teachers in the experimental group, seven complied. After the initial consultation, the consultant created the PET-GAS for each skill. PET-GAS was used for progress monitoring at the four teacher coaching sessions and at the final outcome assessment. Coaching sessions took place about every 5 weeks. Table 1 provides an example of IEP and postsecondary goals.

Table 1 Examples of COMPASS IEP and post-secondary goals
Full size table

Results

An intent-to-treat perspective was used for all analyses. Maximum likelihood estimation using Mplus 8.0 (Muthén and Muthén 2017) with auxiliary correlates (Graham 2003) was used because research (Hayes and McArdle 2017; Shin et al. 2017; Yuan et al. 2012) has shown multiple imputation does not properly handle missing data with smaller samples like that observed in our study, which was limited to two cases on the VABS II. Age and income were used as auxiliary correlates because they correlated with the missingness indicator on VABS II and resulted in smaller standard errors than when they were excluded from the analysis. No baseline differences between groups were observed on parent, teacher, or student variables (Table 2).

Table 2 Comparison of conditions on student, teacher, and family characteristics at baseline
Full size table

Does COMPASS Improve Transition IEP Goal Attainment Outcomes for Students with ASD?

Table 2 summarizes the results of the tests between experimental and control conditions on GAS final scores controlling (adjusting) for baseline GAS scores (i.e., relative change) and GAS final scores without adjustment. Mean GAS scores controlling for baseline GAS score were significantly higher in the COMPASS condition than the control condition. Given that all COMPASS participants had a baseline GAS score of − 2 but two in the control condition had baseline GAS scores above − 2, analyses were also conducted on group differences in GAS final scores not adjusting for baseline scores; these results were still significant, but the effect size (Cohen’s d) was more conservative than when adjusting for GAS baseline scores (see Table 3). Given the lack of variability on GAS baseline scores for the COMPASS group and minimal variability occurring on this same variable in the comparison group (see Table 2), emphasis should be placed on the analyses not adjusting for GAS baseline scores to provide a more conservative estimate of the effect.

Table 3 Comparison of conditions on GAS change scores and GAS final scores
Full size table

What is the Consultant Fidelity of the Modified Version of COMPASS?

Overall mean adherence for the initial COMPASS consultation was 31.45 out of a possible 35 (SD = 6.28) as rated by parents (i.e., 90% of the components were implemented), 32.09 (SD = 3.51) as rated by teachers, and 32.82 (SD = 2.27) as rated by researchers. Repeated-measures ANOVA showed no differences across informants in mean consultant adherence ratings, F(2, 20) = 0.33, p = .72. For the coaching sessions, over 98% of fidelity ratings were rated “yes” across both teacher and independent raters. Teachers rated 98.3% of coaching sessions 1 and 4 as compliant (368 of 374 sessions) and independent raters rated 99.7% of all four coaching sessions compliant (746 out of 748 sessions).

How Well Did Teachers Adhere to Evidence-Based Instruction in COMPASS?

Mean teacher adherence scores for coaching sessions 1, 2, 3, and 4 were 2.09 (SD = 1.04), 2.82 (SD = 1.08), 3.82 (SD = 1.08), and 3.45 (SD = 1.04), respectively. A Friedman test indicated a difference in consultant ratings of teacher adherence to teaching plans over coaching sessions 1–4 (Mean ranks = 1.77, 2.18, 3.18, and 2.86, respectively), Χ2(3) = 9.28, p = .03. Post hoc analyses using a Wilcoxon sign rank test show lower ratings for session 1 versus sessions 3, z = − 2.32, p = .02, and session 1 versus session 4, z = − 2.16, p = .03.

Discussion

In their critical review of the key elements of the transition process for students with ASD, Wehman et al. (2014) identified policy recommendations for school curriculum, employment development, postsecondary education, inclusion, and instruction. However, none of the articles reviewed actually tested a transition planning intervention using experimental methods. Thus, to our knowledge, this is the first experimental trial of a transition planning assessment and monitoring intervention for students with ASD.

To help ensure that our intervention for transition youth was deemed usable, feasible, and acceptable, we applied an implementation science framework to guide our adaptation of COMPASS (Ruble et al. 2018). As our results show, COMPASS as adapted for the transition period was effective. Specifically, IEP goal attainment was much higher for students in the COMPASS group, indicating that COMPASS was able to support teachers and students to achieve transition goals. It is worth emphasizing that the size of the obtained effect was quite large (Cohen’s d = 2.0). Importantly, this finding replicates those from two prior RCTs of younger children with ASD and helps to establish the effectiveness and usefulness of COMPASS across the age span, illustrating the flexibility and general applicability of COMPASS consulting.

The results also demonstrate clinical effectiveness; students in the experimental group were much more likely to achieve their goals. That is 67% of students who received COMPASS met their stated goal (i.e., goals were met at the 0 level or higher) compared to only 18% of those in the control group. Moreover, students in the COMPASS group on average were more likely to exceed expected performance levels, 24% met the 1 or + 1.5 level and 27% met the + 2 level compared to less than 1% of the control who met the + 1 or + 2 levels. Thus, we have evidence of both statistical and clinical effectiveness. Of note, all of this success came at a relatively modest cost of less than 10 h of time across a school year.

COMPASS for transition youth was implemented with high consultant fidelity as rated by parent, teacher, and independent research ratings, and these results were consistent with our prior RCTs. Fidelity was very robust for both the initial consultation (exceeding 90%) and was even higher for coaching (exceeding 98%). One of the barriers to dissemination and implementation of EBPs is low fidelity (Allen et al. 2012). Importantly, these results confirm that COMPASS can be reliably and faithfully implemented.

One problem with establishing fidelity is that it is often difficult to measure. Barriers to more consistent use of fidelity often revolve around validity of available and practical measures. Assessing research fidelity is often time consuming and typically requires objective trained raters. Our findings of a lack of differences in fidelity ratings between the three sources suggest that teachers and parents are able to discern fidelity as well as trained researchers. This implies that COMPASS fidelity can be rather easily and flexibly measured with equal confidence from a variety of sources. Further, this finding suggests that implementers can be flexible in choice of fidelity source. Thus, in terms of one of the key barriers to implementation, fidelity to the model, COMPASS has an established set of measures to support implementation.

Although consultant fidelity was consistently high from the initial contact and throughout the coaching process, teaching fidelity varied across time. Consistent with our studies involving young children and their teachers, teacher adherence was lowest for the early coaching sessions, and improved significantly over time. These findings not only replicate our earlier studies, but continue to support the need for ongoing coaching sessions when consulting with teachers of older students. Like other studies, our findings demonstrate the necessity of coaching for improved teaching quality. Coaching is a distinct activity that is interactive and includes the key evidence-based elements of shared planning, observation, practice, reflection, and feedback (Flaherty 1999; Rush and Shelden 2011). These evidence-based coaching activities allow the teacher to develop competence for obtaining new information, integrating this information into teaching plans, testing the effectiveness of the teaching plans, and adjusting as necessary based on the child’s goal attainment progress. This iterative process results in learning for the teacher that is incremental- building over time with practice, feedback, and ongoing assessment. The initial consultation is the conceptual stage that allows for planning; while the first coaching session is the first observation and practice attempt, followed by reflection, feedback, and another practice attempt. The bottomline is that our finding of incremental improvement is not only empirical but expected based on conceptual models of evidence-based coaching (Dunst and Trivette 2012). This also illustrates that conceptual understanding or planning alone (initial consultation) is insufficient without practice and feedback.

Of note, we found lower overall mean quality ratings of teacher adherence compared to our other studies with young children (Ruble et al. 2010, 2013). That is, for studies with younger students, adherence increased steadily over time with highest quality measured at the final coaching session. But for the transition age students, quality peaked at the third session, then decreased slightly. Possible reasons for the dip are likely context and teacher related. During the last coaching session that occurred 3–6 weeks before graduation, it was not unusual for special education teachers to switch attention from teaching IEP goals to conducting school-wide standardized testing. Thus, the structure of the last coaching session was not robust enough to maintain teaching quality. However, IEP goal achievement continued to increase across coaching sessions. In fact, there was an approximate + 0.5 change in PET-GAS between each coaching session suggesting a dose effect. Moreover, these findings indicate that for some students expected performance levels could be achieved within a shorter time frame, after perhaps 2 (64%) or 3 (73%) coaching sessions. The suggests that schools can be flexible in using COMPASS if only expected performance levels were the target, in that they could chose to stop coaching sessions once a student meets the expected levels of performance.

There were some methodological challenges when comparing this study and the prior RCTs with preschool and elementary age students that are worth noting, specifically, recruitment and the delivery of special education services at the secondary level. For example, we had very low recruitment, in part due to timing. We recruited students in their final year of school, but did not anticipate that teachers often did not know which students were in their last year of school, making it challenging to identify participants. Moreover, although students with disabilities have the right to remain in school until their 22nd birthday (IDEA 2004), many students did not take advantage of this right. In fact, during the first year of our study, every participant left at 18 regardless of intellectual ability. When we consulted with parents to make sure they understood their child’s rights to stay in school, all said they were aware, but were ready for their child to leave school. Differences in how special education was delivered also had an impact. We had difficulties identifying the special educator who actually intervenes with the student. Many high schools organized special education by separating out the teacher of record (often called the case manager) from the actual teacher responsible for implementing the IEP. As a result, the teacher of record often had minimal or no interactions with the student. This situation was most apparent for students who were in general education. Thus, for the transition group, the organization of special education for students was markedly different from the pre-K and elementary settings, with less fixed and direct supervision or support exclusively by the teacher of record. Future research needs to consider these barriers, reasons why students leave school early, and solutions for maintaining students so that the promise of a seamless transition is realized. Despite these lessons learned and barriers faced, COMPASS still worked and was able to adjust for these barriers through evidence-based consultation activities.