Introduction

Children with autism spectrum disorder (ASD) can experience significant challenges in the development of critical communication and social skills such that intensive interventions are required. Access to high-quality intervention has the potential to reduce estimated lifetime care costs for an individual by over a million dollars per child (Jacobson and Mulick 2000). Therefore, timely access to intervention is essential. However, national surveys of providers in the USA have demonstrated the persistent challenge to train a sufficient number of early intervention providers to meet the growing demand for services. For example, in a survey conducted by the National Association of State Directors of Special Education regarding Part C services, 40 of 41 states responding reported significant shortages in special educators (Sopko 2010), a finding echoed by an American Association for Employment in Education national survey (AAEE 2016). For rural and remote communities, even fewer specialists and large geographic service areas magnify these shortages (Center for Disease Control 2016). This urbanistic policy bias extends beyond the USA to Canada as well. Although the Canada Health Act dictates that all children are entitled to equitable access to health services regardless of geographic location, there are four key barriers that limit service in rural areas including travel time, travel modalities, jurisdictional divides and per capita funding models (Vilches et al. 2017).

The use of telehealth service modalities has the potential to get in front of these key barriers. In technology-enabled services, the intervention is not the technology itself, but rather, technology provides access to interventions with demonstrated positive effects (e.g., Mohr et al. 2017). For example, video conference can reduce time lost to travel and provide greater access in remote communities to specialists who are often few in numbers and typically located in urban settings. Via such services, specialists can deliver well-established strategies (e.g., real-time coaching) which are described in the mental health literature (Mohr et al. 2013). In addition to video conference, technology-enabled protocols have been used successfully in numerous formats to deliver services directly to clients. For example, a recent review of over 120 studies examining technology-enabled interventions for medication adherences in mental health and substance use disorders developed a taxonomy of types and modalities of technology-enabled interventions (Steinkamp et al. 2019). The review demonstrated that across the protocols, multiple technology-enabled components were used including social support, data feedback, medication management, psychoeducation, remote care delivery care team contact, and contingency management (Steinkamp et al. 2019). In education, technology-enabled service delivery is on the rise but the scale and rigor of trials used to examine participant outcomes are more limited. For example, preliminary data support the feasibility and acceptability of technology to assist a variety of stakeholders including home-based parenting programs delivered through video conferencing (Traube et al. 2019) and telehealth consultation with classroom teachers to reduce challenging behavior (e.g., Knowles et al. 2017).

Supporting Children with ASD Through Technology-Enabled Intervention in the Community

Due to the high-intensity needs of many children with ASD, it is critical to expand equitable access to quality services. Treatment packages for which efficacy has been demonstrated to target the core challenges in social communication experienced by children with ASD are typically delivered by experienced professionals (e.g., speech-language pathologists, board-certified behavior analysts, teachers, psychologists, etc.), in one-on-one settings, can take place over several months or years, and require either the interventionist travel to the child (e.g., family home or child’s school) or the family to travel to a clinic setting. Efficacy trials have demonstrated that a range of behavioral and Naturalistic Developmental Behavioral Interventions (delivering intervention in natural settings, sharing control between child and therapist while utilizing behavioral strategies to teach developmentally informed targets: Schreibman et al. 2015), can improve children’s communicative and cognitive outcomes (e.g., Fuller and Kaiser 2019; Smith and Iadarola 2015). Efficacy trials conducted in highly controlled clinic settings with university employed clinicians have tested a range of programs through randomized trials such as Early Social Interaction (e.g., Wetherby et al. 2014), Preschool Autism Communication Trial (Green et al. 2010), Pivotal Response Teaching (PRT: Hardan et al. 2015), and Joint Attention, Symbolic Play, Engagement, and Regulation (JASPER: Kasari et al. 2006, 2008). Systematic reviews and meta-analyses indicate that such early intervention programs can lead to gains in core social communication challenges for children with ASD (e.g., Murza et al. 2016).

A significant advance in intervention science is the increase in testing these interventions under real-world conditions through effectiveness trials (Smith and Iadarola 2015). Fewer intervention packages have been tested under these conditions where stakeholders from the community setting (e.g., agency, school, etc.) are trained to deliver the intervention with children in their own setting. In the past several years, randomized effectiveness trials have demonstrated gains in children’s communication skills when coaching parents to deliver the intervention both in North America (e.g., Kasari et al. 2014; Stadnick et al. 2015) and in low resource settings abroad (e.g., Rahman et al. 2016). Yet, fewer of these models have been tested in public agencies or educational settings. However, Joint Attention, Symbolic Play, Engagement, and Regulation (JASPER: Kasari et al. 2006) is an intervention that has been tested in a number of effectiveness trials. This includes effective delivery of the intervention by teachers in preschool classrooms (e.g., Chang et al. 2016; Kaale et al. 2012), and paraprofessionals in center-based toddler programs (e.g., Shire et al. 2017) leading to gains in children’s social communication and play skills. By examining these interventions in the context within which they will be applied, with the community providers who will deliver the service, and with a naturally heterogeneous group of children, the timeline to transfer research-backed practices into community settings is reduced (Weisz et al. 2015). However, it can be more challenging to see an effect of intervention under these conditions when controls are reduced and variability increases. The use of technology adds another dimension to implementation by changing the modality of delivery of the intervention. Complex and dynamic social communication interventions take place within interaction with the child and those in their community. While technology-enabled services may increase the availability and equitable distribution of services to children in need, mediation of services through a virtual platform creates a separation between the specialist and those who are live on-site with the child even though the specialist is present in real time. To date, research examining technology-enabled direct services to children with ASD and their families has largely included single case designs and case reports (Knutsen et al. 2016). Recent reviews examining studies including children with ASD report preliminary support to demonstrate that technology has been harnessed to provide self-directed online materials and consultation to families (e.g., Pickard et al. 2016) as well as video-conferenced coaching with families (e.g., Suess et al. 2014). Altogether, reviews indicate that preliminary positive effects with a focus on reducing challenging behavior and increasing communication skills have been demonstrated (Knutsen et al. 2016; Sutherland et al. 2018).

Training Professionals with Mixed Remote and Face-to-Face Methods

Live coaching is a common tool to support interventionists who are learning to deliver behavioral interventions with young children with ASD (Schreibman et al. 2015). Due to the complex, individualized, and dynamic nature of these interventions, training professionals to learn to implement the strategies with fidelity requires significant time and resources. Recent studies highlight the feasibility and preliminary efficacy of remote instructional support to train new providers to implement intervention strategies with children with ASD. For example, Neely et al. (2016) taught interventionists to increase opportunities for children to request which led to increases in the frequency of children’s requesting behavior. Web-based training has also been used to teach community providers to deliver Project ImPACT (Improving Parents as Communication Teachers: Ingersoll and Wainer 2013), a parent-mediated naturalistic developmental behavioral intervention. Participants’ self-reported satisfaction and self-efficacy significantly increased after receiving the training (Wainer et al. 2017). Although similar examples are emerging in the literature, recent reviews indicate that interventionists’ fidelity following telemediated training has been variable (Tomlinson et al. 2018; Neely et al. 2017). Some studies demonstrate high fidelity across trainees while others show mixed results or no data for implementation fidelity. Therefore, further study of the influence of remote training on interventionists’ fidelity and children’s corresponding outcomes is needed.

An alternative to fully online training may be a mix of remote and face-to-face supports. Mixed remote and face-to-face training supports have been examined in research–practice partnerships. For example, in a randomized controlled intervention trial, the implementation of JASPER, a comprehensive social communication intervention, by early intervention teaching assistants (TAs) with toddlers with ASD was tested in comparison to the music and movement usual care program. To develop and execute this study, the university-based research team worked closely in collaboration with the agency staff including a local supervisor who worked on-site with the TAs (Shire et al. 2017). Over the course of 12 weeks, the research team provided the TAs with a 5-day introductory training followed by weekly remote support including video review and written feedback. Concurrently, the local supervisor provided additional live on-site coaching throughout each week for the more than 20 TAs across two physical locations. Together, this combination of support led to an average of 80% fidelity for the TAs after 11 weeks and significant gains in children’s joint engagement, joint attention, language, and play skills (Shire et al. 2017). The TAs sustained the majority of their skills into the subsequent school year with a new cohort of children (M = 70.71% fidelity; Shire et al. 2019). Although these interventionists made gains with this combination of tools, to date, no comparative examination of remote versus face-to-face training for new interventionists and the corresponding influence on children’s outcomes has been conducted.

Context for the Current Study

The current study is a secondary data analysis from a randomized controlled JASPER intervention trial. The trial was a partnership between the university-led research team and the public regional health authorities in the Canadian Province of Newfoundland and Labrador. This research–practice collaboration was developed at the request of the health sector leadership in the province who were looking for additional evidence-based behavioral intervention programs to add to their available services for children with ASD, which included early intensive behavioral intervention and eclectic therapies within speech-language pathology and occupational therapy. The growing number of randomized effectiveness trials where JASPER intervention was tested when delivered by community providers was an important consideration for health sector leadership.

The larger randomized intervention trial, from which the data analyzed in this study were collected, focused on the development of a province-wide network of home and clinic-based interventions who provide direct services to children with ASD and their families. Due to the real-world geographic and service context of the province, the community trainers were not always able to provide face-to-face training for their interventionists. Therefore, a natural opportunity occurred to explore a quasi-experimental contrast of training supports provide face-to-face versus those provided remotely including video conferencing and video review on (a) interventionists’ growth in intervention implementation fidelity as well as (b) children’s growth in social communication and play skills. We hypothesize that trainees receiving the traditional face-to-face supports where coaches may model and provide live scaffolding as well as environmental supports will demonstrate greater growth in implementation fidelity than those receiving remote support. In addition, we hypothesize that children paired with interventionists receiving face-to-face support will make greater growth in their social communication and play skills compared to those paired with interventionists receiving remote training.

Methods

Participants

Senior Trainers

Five senior trainers (STs) were included in this project spanning four regional health authorities across the province. Three STs covered the densely populated urban center of the province while two STs covered more rural sections of the province. The two STs in the rural locations utilized remote strategies to support their interventionists. (Face-to-face and remote training strategies will be described alongside the intervention section below.) STs were female and 40.4 years of age (SD = 5.90 years). The STs came to the study with Master’s degrees in related fields (e.g., psychology, speech-language pathology, teaching license). They had been engaged in their current positions with the health authority for 9.2 years (SD = 3.42 years). Four of the five STs had engaged in staff management and supervisory support for the existing behavioral intervention programming prior to becoming a JASPER ST.

Interventionists

Twenty-seven interventionists participated in the study with 16 receiving face-to-face training from their STs and 11 received remote support. Demographic data from 21 of 27 interventionists were included in this study (Table 1). The interventionists were primarily female (n = 19), Caucasian (n = 20), and had completed professional training (n = 15), college degrees (n = 6), or a graduate degree (n = 1). Eighteen interventionists reported areas of specialty including autism-specific intervention or assessment training (n = 8, e.g., regional training in applied behavior analysis (ABA), Picture Exchange Communication System, AFIRM online publicly accessible modules about various intervention), speech and language (n = 5), general education (n = 1), special education (n = 1), and occupational therapy (n = 1). The interventionists had been employed in their current positions for an average of 8.12 years (SD = 6.43 years).

Table 1 Senior trainer and interventionists characteristics
Full size table

Interventionists were assigned an ST based on who the ST was in their health region. Interventionists remained with the same ST over the course of the study. Aside from age, the interventionists enrolled in face-to-face versus remote support were not significantly different in demographic characteristics (Table 1). However, on average, interventionists in the remote support group appeared to have more experience. This mean difference is driven by two interventionists who had significantly more years of experience (24 years).

Children with ASD

The 50 children were recruited from those who were eligible for services based on ABA. The provincial ABA Program is currently available to children up to Grade 4 entry who have received a diagnosis of ASD from a qualified professional (e.g., registered psychologist, psychiatrist, or pediatrician). Preschool children are eligible for up to 30 h of service per week, while children in Kindergarten are eligible for 15 h a week and those in Grades 1–4 are eligible for 10 h a week. There are no financial eligibility requirements for availing of these services and all program costs are government-funded. Referrals to the program are often directed from the clinicians who complete the diagnostic assessments with the child.

Fifteen girls and thirty-five boys were included. With 38 of 50 families reporting demographic information, children ranged in age from 2.46 years to 8.51 years (M = 4.66 years; SD = 1.51). The children received a diagnosis of autism spectrum disorder through assessments conducted by the regional health authority including the Autism Diagnostic Observation Schedule-2 (ADOS2: Lord et al. 2012). Parents reported the children’s ethnicity as primarily Caucasian (n = 35) and mixed race (n = 3). Eighteen children were supported by two interventionists who received remote training, while 32 children were paired with interventionists receiving face-to-face support.

Setting and Context

Due to the nature of the geography of the province, the STs were often housed in urban centers within their region and were separated by both distance and by geographic obstacles (e.g., bodies of water, snow/ice, limited roadways, etc.) from their interventionists. Census data from 2016 indicates that nearly 40% of the population (over 200,000 people) of the province live in rural rather than urban centers (Statistics Canada 2016). This challenging terrain, large rural land mass, the financial resources and time required for travel, and the limited number of specialists were significant barriers that prohibited the ability of two of five senior trainers to make weekly visits to provide face-to-face in-person coaching with their interventionists. Therefore, these two STs provided technology-enabled supports including coaching as well as video review and discussion through video conferencing. This real-world context provided a naturally occurring opportunity to explore face-to-face in comparison with remote interventionist training.

Intervention

JASPER, a comprehensive social communication intervention designed to facilitate children’s social engagement, nonverbal and spoken communication, and play skills, was delivered to the children enrolled in this study. JASPER provides systematic support to facilitate children’s spontaneous initiations both in play and in communication. The intervention is made of seven subscales including setting up the environment, balancing imitation and modeling, establishing play routines, expanding play routines, programming for joint attention and requesting, language strategies, and supports for engagement and regulation (Table 2). New interventionists are introduced to the strategies over the course of 5 days as well as the developmental sequence of children’s engagement, social communication, and play skills.

Table 2 JASPER intervention strategy subscales [from Chang et al. (2016)]
Full size table

Intervention Session Logistics

JASPER intervention sessions were 60 min long, delivered one-on-one (interventionist–child), and took place in the child’s home or at a local clinic depending on the family’s preference and proximity to the clinic. In either case, the interventionist had access to toys and materials through their local health authority that were either brought into homes or set up for use in the clinic.

Training for Senior Trainers

All five STs demonstrated excellence in the intervention model. After participating in a 5-day introductory training with the research team, the STs practiced for 16 weeks with two child cases until reaching 90% implementation fidelity (Fig. 1). The STs were identified by the team leadership as potential trainers. They were offered the roles by their local management and all five accepted. The STs were then provided with a three-day intensive workshop to learn to coach other professionals to deliver the intervention. This workshop was followed by 12 weeks to reach coaching fidelity. JASPER coaching fidelity was rated on a 13-item scale where each item is rated from 0 (strategy is not present) to 5 (appropriate, timely, and quality implementation of coaching strategies). STs learned to use the coaching strategies which are presented in a least-most prompting hierarchy. The ST is responsible for helping the child stay regulated, engaged, and advance their play and social communication skills through developmentally appropriate play routines, while also providing the interventionist with timely support that balances correction with reinforcement and is responsive to the needs of both child and interventionist. All STs demonstrated at least 90% coaching fidelity (as rated by members of the research team) with practice cases immediately following the training. Finally, the STs also learned to reliably provide written feedback and score the 31-item intervention fidelity form (see Measures: Interventionist–Child Interaction for reliability scores).

Fig. 1
figure 1

ST and interventionist training timeline diagram

Full size image

Training for Interventionists

All interventionists began with a 5-day intensive introduction delivered by a pair of STs. The interventionists were gathered in groups of 2–8 at one central location. Over the course of 5 days, the interventionists were first introduced to the seven JASPER strategy subscales beginning with identifying developmentally appropriate intervention targets and setting up the environment. Second, the interventionists learned to imitate and model in order to establish play routines. Third, they applied language strategies and began to both expand the routines and program for joint attention and requesting gestures. The 5 days included didactic teaching, discussion, live coaching, and video review. Each interventionist then practiced with a child case for 12 weeks with weekly feedback prior to being assigned the two child cases which are represented in the data in the current study.

Study Phase

Once assigned to two children, each interventionist received weekly support through either the face-to-face or the remote training modality for 12 weeks (Fig. 1). In either case, the interventionist was asked to video record his/her intervention sessions with the child. Real-time support was provided once a week for each interventionist. Therefore, real-time support was provided on alternating weeks for each child case (intervention with Child A in week 1, Child B in week 2, Child A in week 3, and so forth). The interventionist selected which of the two child cases they would prioritize for feedback in week 1 based on the needs of the child and the comfort level of the interventionist to meet those needs. Coaching supports in week 1 were directed to this child case. Data presented in the results are for children enrolled in the study phase.

Face-to-Face Support

The ST attended the intervention session with the interventionist–child dyad in person. The ST provided live coaching which includes a range of strategies including modeling, paired implementation, verbal feedback, and environmental supports (see Table 3 for details) as needed by the interventionist. The ST is expected to follow a least to most support hierarchy and therefore, provide the lowest level of support necessary for the interventionist to be successful. The type of support provided by the ST is expected to shift over time from intensive supports (modeling, paired implementation) to verbal reminders/feedback and environmental supports (e.g., organizing materials, timely handoff of materials for play expansions) as the interventionist gains more skills and confidence. The session wraps up with a discussion to debrief and reflect together.

Table 3 Coaching strategies
Full size table

Remote Support

STs provided real-time coaching via video conference for the interventionist while he/she worked directly with the child. Access to a variety of video conference platforms was provided through the secure provincial telehealth system including Skype of Business, Cisco Jabber, and GoToMeeting using both laptops and tablets. Video recordings of these sessions were obtained by recording within the teleconference platform or with an additional software packaged (Snagit). Unlike face-to-face coaching, the video conference coaching protocol relied more heavily upon verbal feedback and discussion because the ST was not physically present to model strategies with the child, physically arrange the environment or pair in implementation with the interventionist.

Study Design

Data for this study were taken from assessments collected as part of a larger randomized adaptive intervention trial examining the development of new interventionists in a provincial intervention deployment trial. The intervention applied in this trial is Joint Attention, Symbolic Play, Engagement and Regulation (JASPER: Kasari et al. 2006). Five senior JASPER trainers in the local community were each responsible for training 3–10 new interventionists who provided one-on-one clinician–child JASPER services to children with ASD through home- or clinic-based sessions. The data in the current study (see Measures for details) include data taken pre- and post-24 one-hour JASPER intervention sessions. The data were coded from video recorded: (a) child assessments administered by independent assessors and (b) intervention sessions including the trainee interventionist and the child.

The data were collected during the second phase of the larger trial. Interventionists’ implementation fidelity was measured after a 3-month-long practice phase. Interventionists who demonstrated less than 90% implementation fidelity with their practice child cases were randomized to one of two support conditions: (a) intensive refresher (12 h of support across a 5-day period from the assigned senior trainers including coaching, targeted discussion, lesson planning, and video review/feedback) and (b) peer support (12 h of support across a 90-day period from a peer interventionist who had achieved fidelity including discussion and video review). Seventy percent of all interventionists (19/27) received support during this period. The two support conditions (intensive refresher and peer coaching) took place in even proportions among interventionists receiving remote support (50%/50%) and those receiving face-to-face support (53%/47%).

Measures

All measures were video-recorded. Similarly to the intervention, assessments were conducted in the family’s home or clinic. The videos were randomized such that behavioral outcome coders were blinded to condition (remote versus face-to-face training support), time point, and health region. Coders for the child outcome assessments included members of the study team (e.g., graduate students), while the STs coded the interventionists’ fidelity.

Trainee Interventionist–Child Interaction (TCX: Chang et al. 2016)

The TCX is a video recording of the 60-min intervention session including the trainee interventionist and the child when the ST was not present to provide coaching. The TCX was recorded by the interventionist by placing a camera on a tripod or elevated on furniture. The TCX was recorded weekly. The first-week video was used as “entry,” and a video from the final week of intervention was used as “exit” and coded for the dependent variable below.

Dependent Variable: Implementation Fidelity

The TCX video was scored for interventionists’ JASPER implementation fidelity. The rating system included 31 items capturing the seven main intervention components including basic strategies, environment, following the child’s lead, establishing play routines, expanding play routines, programming for joint attention and requesting skills, and language strategies (Table 2). The same rating system applied to research clinicians is applied to community interventionists. Each item is rated from 0 to 5 where “0” reflects incorrect or lack of strategy implementation, a “3” describes mixed implementation where up to 50% of opportunities to use a strategy are missed, and a “5” represented accurate and developmentally appropriate strategy implementation at least 80% of the time. Missed opportunities to deliver a strategy such as respond and expand a child’s communicative bid or to imitate an appropriate playact would lead to lower scores. Unnecessary application of strategies (e.g., modeling when the interventionist could provide space for the child to initiate the next step) or inappropriate application would also lead to lower scores. Application of the array of strategies that matches the pace, goals, and needs of the child over the course of the session will lead to high scores. Item scores were summed and divided by the total number of possible points to obtain a percentage score for implementation. STs established reliability with the first author to score JASPER session fidelity.

Inter-observer Agreement

Intra-class correlations (ICCs) were examined for subscale score totals between each rater and the research team (gold standard). Conventional standards were followed where ICCs at .8 or above were considered acceptable (Shrout and Fleiss 1979). ICCs ranged among raters: setting up the environment (α = 0.838–1.0), imitation/modeling (α = 0.916–0.986), routines (α = 0.887–0.992), expand routines (α = 0.873–0.992), programming for social communication (α = 0.818–1.0), language (α = 0.875–0.933), and supports for engagement (α = 0.830–0.996). The first author then double scored 20% of the videos including major time points (e.g., last session of phase) to check for drift. All STs maintained reliable scoring. STs were trained to conduct the fidelity ratings because this would continue to be an essential tool for their sustained community service protocol beyond the research trial.

Early Social Communication Scales (ESCS: Mundy et al. 2003)

The ESCS is a semi-structured play-based assessment administered at baseline (prior to intervention start) and within 2 weeks of 12-week intervention exit. This 20-min assessment is designed to capture spontaneous initiations of joint attention (JA) and requesting (behavior regulation: BR). The materials include windup toys, a ball, comb, hat, book, and pictures on the walls. Each material is paired with a systematic set of opportunities for children to demonstrate gaze, gestures, and/or language for the purpose of JA or BR. Local assessors who were also employees of the regional health authority and familiar with assessing young children with special needs were trained to deliver the ESCS with practice administrations until they demonstrated at least 80% administration fidelity. The fidelity scoring included items for how each of the different toys presented to ensure that an equal number and type of opportunities were provided to each child. For example, each windup toy is to be presented three times out of the child’s reach and the assessor must pause post-activation to give an opportunity for the child to share (initiate JA) and/or request the toy (initiate BR). Fidelity was scored by the research team for all administrations providing data for this study. Across local assessors, the ESCS was delivered with an average of 81.88% fidelity (SD = 12.43%).

Dependent Variables: Joint Attention and Behavior Regulation

The ESCS video was coded for the frequency of both verbal and nonverbal initiations of: (a) joint attention (IJA) including triadic gaze, point to share, show to share give to share and spoken comments, and (b) initiations of behavior regulation (IBR) including eye contact, reach to request, point to request, give to request, and spoken words to request. Construct validity has been reported for the ESCS (e.g., Mundy et al. 2003) as well as high reliability of the coding schema (e.g., Mundy et al. 1987; Paparella et al. 2011).

Inter-observer Agreement

Intra-class correlations (ICCs) were calculated for both independent raters IJA (α = 0.89–0.95), and IBR (α = 0.96–0.99).

Structured Play Assessment (SPA: Adapted Kasari et al. 2006)

Adapted from Ungerer and Sigman (1981), the SPA is a 15-min assessment designed to capture children’s spontaneous playacts across a range of developmental play levels. The SPA was administered at baseline (prior to intervention start) and within 2 weeks of intervention exit. The child is presented with five sets of toys including: (a) puzzle, shape-sorter, and nesting cups; (b) dolls and a tea set; (c) dolls, phone, mirror, and comb; (d) dolls and furniture, and (e) barn, blocks, truck, and animals. No prompting is permitted. When the child no longer displays novel playacts, the next toy set is presented. The SPA was also delivered by the local assessment team. Assessors trained with practice cases until reaching 80% or greater administration fidelity. Fidelity ratings focused on critical elements of the assessment including the provision of sufficient time for the child to engage with all materials in each set, use of behavioral strategies to support the child’s engagement in the protocol (e.g., keeps the child seated, arranges the environment to ensure the child can reach all the materials), and creation of a responsive and social atmosphere by responding to the child’s communication. All administrations were scored for fidelity. The SPA was administered by local clinicians at an average 93.05% administration fidelity (SD = 10.27%).

Dependent Variable: Playacts

The SPA video was examined for discrete spontaneous play behaviors. Each playact (e.g., put cup to doll’s mouth) was categorized into one of 16 play levels spanning from simple through symbolic play levels (Lifter et al. 1993). Demonstrations of high reliability of this coding have been published (e.g., Kasari et al. 2010; Ungerer and Sigman 1981).

Inter-observer Agreement

ICCs were calculated for independent raters for play type identification for each of the play levels (α = 0.897–0.988).

Statistical Analyses

Linear mixed models (LMMs) were used to evaluate the longitudinal trajectories of continuous outcomes from entry to exit of the 3 months of JASPER intervention (24 sessions) with subject-level random intercepts. Likelihood ratio tests and Akaike information criterion (i.e., lower values) were both used to determine the best covariance structure for each model (i.e., clustering within individual repeated measures). All model assumptions were assessed (i.e., normality of residuals, equal variance, etc.). The group effect (remote support for interventionists versus face-to-face support for interventionists) was defined as a significant interaction between the type of support and time (group by time interaction). Separate models were fit for each longitudinal outcome. Effect sizes (ES) were reported for significant findings using Cohen’s f where effect sizes of 0.10, 0.25, and 0.40 were generally regarded as small, moderate, and large. SAS 9.4 (SAS Institute, Cary NC) and R Cran version 3.4 (R Core Team 2019) were used for all analyses.

Results

Results for interventionists’ fidelity, children’s social communication (initiations of joint attention and requesting), and children’s play types are described in Table 4.

Table 4 Outcomes by types of support
Full size table

TCX Interventionists’ Fidelity

On average, interventionists from both remote and face-to-face training groups entered with high fidelity (M = 0.84, M = 0.90, respectively) but continued to make significant improvement by exit (F(1,139) = 4.11, p = 0.04; ES = 0.17). There was no significant difference in improvement between interventionists receiving remote versus those receiving face-to-face support (F(1,139) = 0.001, p = 0.98).

ESCS: IJA/IBR

All children demonstrated significant improvements from entry to exit in initiations of joint attention skills (F(1,25) = 10.1, p = 0.004; ES = 0.64) and modest improvement for initiations of behavioral regulation (requesting: F(1,25) = 3.08, p = 0.09). There were no significant differences in improvement between children paired with interventionists receiving remote support or those paired with interventionists receiving face-to-face support for either IJA (F(1,25) = 0.88, p = 0.358) or IBR (F(1,25) = 0.09, p = 0.765).

SPA: Play Types

Improvement in children’s play diversity was examined first as total play types (across all play levels) and then specifically symbolic play types (highest play levels). Children paired with interventionists receiving remote support and those paired with interventionists receiving face-to-face support made significant gains in total play types from entry to exit (F(1,23) = 4.42, p = 0.047; ES = 0.44) and symbolic play types from entry to exit (F(1,23) = 11.21, p = 0.002; ES = 0.69). Between groups, a modest difference in improvements in total play types was found where children paired with interventionists receiving face-to-face support demonstrated greater improvements compared to children paired with interventionists receiving remote supports (F(1,23) = 2.98, p = 0.098). There was no significant difference between remote and face-to-face groups in improvements in symbolic play (F(1,23) = 0.61, p = 0.444).

Discussion

For interventionists who are working in community intervention programs, it can be challenging to apply the package of intervention strategies in ways that best match the developmental profile of strengths and challenges which can vary widely across the spectrum of children they are expected to serve. By nature of the autism spectrum, the range of children who require service in the community is wide. Within this sample of children with ASD who qualify for publicly funded intervention services, are children who have few or no words, who demonstrate significant global developmental delays, and those who are working on developing more complex language and symbolic abstraction in their play. Therefore, for these interventionists to achieve JASPER implementation fidelity, they must develop a sophisticated understanding not only of the mechanics of the strategies but also the complexities that come with individualizing the package to address the individual child. Interventionists in this study reached an average of 93% fidelity in the remote condition and 87% fidelity with face-to-face support. To achieve this high level of fidelity by treatment exit, the interventionist must set developmentally appropriate communication and play goals for each unique child and then builds play routines that included the child’s interests. Further, these routines must flexibly expand to balance reaching for the next target play level while supporting the child’s ongoing engagement and regulation in the activity as well as providing appropriate space and time for the child to communicate. This package led to significant change in children’s initiations of joint attention and requesting skills as well as play diversity (number of unique play types).

Coaching of new JASPER interventionist is typically completed face-to-face or as a combined package of face-to-face and remote training. Our clinical hypothesis was that remote support alone may not produce the same effects on interventionists’ development as face-to-face training because key coaching strategies are not physically available to the trainers such as paired implementation (where the interventionist and coachwork as a team to deliver the strategies with the child), modeling strategies directly with the child, and environmental arrangement (e.g., removing clutter, providing timely access to materials to expand the routine). Instead, the coach must rely on specific and targeted verbal feedback to help the interventionist take these actions on their own during the session. Yet, no significant differences were found in interventionists’ implementation fidelity or children’s social communication or play skills between dyads receiving remote support and those receiving face-to-face support. These findings speak to the strength of the local trainers to provide timely, targeted, and explicit feedback throughout the session, a skill that may need explicit support to curate in new trainers. This small group of trainers provides preliminary evidence that within the context of provincial deployment of a dynamic social communication intervention that must be individualized to the needs of each child’s needs and goals, trainers can extend their reach to support new interventionists through technology-mediated supports without compromising the quality of the service or the potential impact of that service on the children’s outcomes.

Feasibility and Acceptability of Technology-Enabled Supports

This study was conducted in a province where the regional health authorities had existing telehealth initiatives in place for both training supports and for the provision of intervention services. Therefore, the study team had the benefit to use available equipment and work with information technology staff who were familiar with this process. Initially, there was a significant amount of consultation required to ensure the video conferencing technology and data transfer methods were secure and confidential. Once the tools were in place, the STs and interventionists anecdotally reported that it took some time to adjust to the new method of coaching as well as layering in the understanding of the technology required to smoothly navigate their sessions.

When considering the acceptability of the video-conferenced remote support, both STs providing these supports informally reported that many of their new interventionists reported a preference for post-session video review and discussion rather than real-time video-conferenced coaching when the child was present. Reflecting upon this preference, there are potential advantages to video review and discussion. For example, review of the video after the session allows for thorough discussion with clear visual reference to specific aspects of the session. JASPER sessions are fast-paced and dynamic, decisions made at the moment with the child must be made quickly and there is little time for the trainer and interventionist in training to reflect upon those choices in the moment. Discussions based on video feedback provide the space and time to reflect on those decisions as well as the consequences of those choices that followed in terms of the child’s behavior.

Increase Accessibility with Lower Cost Delivery Models

The province of Newfoundland and Labrador is physically divided into two major units of unequal area: the much larger mainland territory of Labrador to the north; and to the south is the island of Newfoundland surrounded by the North Atlantic, making it the easternmost region of Canada. In 2016, 58 percent of the population was urban, with the remaining 42 percent settling in the many rural communities among the hundreds of bays, coves, islands and small inlets (Statistics Canada 2016). Various modes of transportation such as planes, ferries, boats, and even skidoos are required to access some of the remote communities. Although formal cost analyses were not conducted, there are notable financial and human resources that were not required for a remote training visit that is required for a face-to-face training visit. For example, the Labrador–Grenfell health region constitutes 71% of the province’s geographic area but only 6.5% of the total population (Statistics Canada 2016). Therefore, the ST managing this region is required to cover substantial distance in order to conduct a single 60-min home visit with an interventionist–child dyad who are living outside the same urban area. To reach a single family’s home, a ST may be required to engage in a full day of travel including multiple modes of transportation (e.g., airplane, jet ski, vehicle, on foot). Not only does the use of remote training support eliminate the travel costs associated with this visit, but frees the block of working hours the interventionist would have spent in travel for additional remote service delivery directly to the interventionists.

Limitations and Future Studies

This study took place within the service delivery model of the province’s health system. A limitation of the study is that the research team was not able to confirm the diagnoses of autism that were provided by local clinicians. Due to the challenging geographic context of the province, a natural opportunity arose to compare the development of new JASPER interventionists and children’s outcomes between remote and face-to-face training methods. However, this comparison was not the original question for which the study was designed. The randomization for this study was built to test two types of supports for interventionists working to achieve a standard of 90% implementation fidelity. Future work could test this comparison with greater rigor by randomizing to remote and face-to-face training groups. Future work may also examine the role of buy-into training modality both with the interventionists as well as with the families. Under current conditions, it is not possible for interventionists and families living in rural and remote areas of the province to access the same density or dose of services with face-to-face methods alone. However, for interventionists and families living in urban areas where equal dose could be provided by either mode, other factors such as interventionist or family preference could influence both the acceptability and the effectiveness of remote supports.

Conclusions

The successful use of telehealth supports by local STs to teach and supervise new interventionists is promising. This work has laid the foundation to establish the use of technology within the province. Based upon this work, the province is also testing a technology-enabled video-conferenced coaching protocol to support caregivers’ ability to apply the strategies with their children at home. Altogether, this study indicates that high-quality training which advances critical skills for interventionists delivering a social communication intervention that is typically provided through face-to-face delivery can also be provided remotely through video conferencing services.