Introduction

With an estimated prevalence of 1 in 68 (Christensen et al. 2016), early accurate identification and treatment of young children with Autism spectrum disorder (ASD) represent a pressing public health and clinical care challenge (Interagency Autism Coordinating Committee 2013). There is growing evidence that (1) accurate, stable diagnosis of ASD is possible during the second year of life (Chawarska et al. 2014; Corsello et al. 2013; Guthrie et al. 2013) and (2) very young children with ASD receiving early behavioral intervention services demonstrate substantial gains in functioning (Dawson et al. 2012, 2010; Warren et al. 2011; Zwaigenbaum et al. 2015). American Academy of Pediatrics (AAP) guidelines endorse universal screening for ASD at 18 and 24 months of age, and at any point when caregivers express concerns (Johnson and Myers 2007). Ideally, all children would be screened for ASD in the second year of life and those at-risk would promptly receive comprehensive evaluations. Unfortunately, for a variety of reasons large numbers of children are still not screened for ASD, wait times for appropriate diagnostic assessment are tremendous, and the average age of diagnosis in the US is still after four years of age (Christensen et al. 2016; Hyman and Johnson 2012; Wiggins et al. 2006). Moreover, groups from traditionally underserved communities are much less likely to be diagnosed at young ages, including children from families of lower socioeconomic status, children whose parents report lower levels of educational attainment, and children from racial and ethnic minority groups (Christensen et al. 2016; Durkin et al. 2010; Fountain et al. 2011; Liptak et al. 2008; Zuckerman et al. 2013). Such diagnostic delays and health disparities contribute to substantial deleterious family stress (Warren and Stone 2011) and restrict access to ASD intervention services (Pierce et al. 2011; Stahmer and Mandell 2007).

Screening in isolation, particularly when relying upon paper-and-pencil parent report measures, may fail to detect many children with ASD or, conversely, over-identify children with other developmental concerns as being at risk (Chlebowski et al. 2013; McPheeters et al. 2016; Pierce et al. 2011; Robins et al. 2014; Stenberg et al. 2014). When children screen positive on ASD risk instruments, providers are most commonly making referrals to (a) tertiary diagnostic centers with tremendous waits for evaluation and/or (b) early intervention systems that, in absence of a diagnosis, typically provide very low levels of non-specific developmental service (Pierce et al. 2011; Stahmer and Mandell 2007). These issues present significant challenges for those providing care to multi-stressed, linguistically diverse, rural, and traditionally underserved families, where accessing such tertiary care services may represent an insurmountable barrier. Up to 40% of children and families from low-resource communities struggle to access these types of evaluation services, even when universal screening programs are implemented (Chlebowski et al. 2013; Robins et al. 2014). These struggles highlight the need for novel approaches to ASD screening, evaluation, and diagnostic decision-making that will realistically advance early detection and intervention, particularly for families facing geographical and resource barriers to comprehensive evaluation.

Despite inherent complexities in understanding service delivery, clinical impact, and financial reimbursement, numerous ‘successful’ telemedicine models have been reported across a variety of medical and psychiatric conditions (Lindgren et al. 2016). Regarding ASD telepractice, there is a small, but growing literature base indicating the feasibility and potential value of telepractice-supported and delivered behavioral intervention services (Lindgren et al. 2016; Simacek et al. 2017; Suess et al. 2014, 2016; Vismara et al. 2009; Wacker et al. 2013; Wainer and Ingersoll 2015). However, to date no study has systematically evaluated the accuracy nor potential perceived clinical value of diagnosing ASD in young children via telemedicine from either a clinician or family perspective.

The present work evaluates the potential value of ASD telemedicine-based diagnostic consultation services across two separate studies. First, we compared telemedicine consultation to gold standard, in-person diagnostic evaluations to assess its diagnostic accuracy and validity. We then implemented telemedicine diagnostic consultation with families in a rural health clinic in conjunction with our state’s early intervention system. These families, as well as professionals involved in the consultation, provided feedback regarding their perceptions of this method of service delivery. We explicitly designed these as two separate studies, as families participating in the initial in-person accuracy study could not provide feedback on what it was like to undergo the telemedicine procedure without the subsequent comprehensive assessment they received, nor could families in the rural health clinic study provide diagnostic accuracy data in absence of a comprehensive assessment.

Methods

Overview of Telemedicine Assessment Procedure

Certain aspects of procedure and set-up were consistent across both studies and are described as follows. Tele-evaluation rooms in both settings were equipped with a wall-mounted Cisco SX10 pan-tilt-zoom camera that allowed for audiovisual communication and far-end control by the remote assessor. Remote assessors used a HIPPA compliant platform (Cisco Systems 2015) to observe and communicate via video, and control the movement of the camera.

The remote assessors (n = 5), who were the same across both studies, were licensed clinical psychologists with specific expertise in ASD in young children. Each psychologist had previously established research reliability on the Autism Diagnostic Observation Schedule—2 (ADOS-2) (Lord et al. 2012) with mean of 6.2 years (3.84 SD; 2–11 range) in independent practice.

In both study 1 and study 2, children and their caregivers participated in a standardized ASD assessment procedure. This procedure has previously been utilized by the investigative team to train primary care physicians to identify ASD within medical homes via rapid in-person assessment (Swanson et al. 2014; Warren et al. 2009) and establish early intervention system criteria for intensive ASD supports (Rotholz et al. 2017). The procedure involves (1) a brief medical and psychosocial interview; (2) observation of administration of the Screening Tool for Autism in Toddlers and Young Children (STAT), a 15–20 min standardized interactive assessment of social communication skills that provides a risk score (0–4, with ≥ 2 considered “at risk”) to differentiate 12- to 36-month-old children with ASD from those with other developmental concerns (Stone et al. 2000, 2004, 2008); and (3) a focused DSM-5 ASD diagnostic interview for toddlers. Psychologists interviewed caregivers directly via the telemedicine technology and observed the STAT as conducted by a trained bachelor’s level research assistant (study 1) or early intervention system provider (study 2). The entire telemedicine diagnostic procedure was scheduled to last no more than 1-h.

Study 1: Telemedicine Diagnostic Accuracy and Validity

Participants

Participants were 20 children (16 boys, 4 girls) between 20 and 34 months of age (mean = 26.65, SD = 4.49) and their caregivers (see Table 1). All children were referred for evaluation to a diagnostic clinic at a university-affiliated medical center due to early concerns about ASD. Families were pulled from the clinic waitlist across a 7 month period based upon time of referral and availability of research evaluation slots. One invited family declined participation.

Table 1 Study 1 participant characteristics by telediagnostic—in person diagnostic outcome
Full size table

Measures

DSM-5 Clinical Interview

All caregivers responded to a DSM-5 Clinical Interview administered by a licensed psychological provider (i.e., licensed clinical psychologist, licensed senior psychological examiner). This interview, consisting of questions across the social, communication, and behavioral domains, probes for ASD symptoms commonly observed in young children. Importantly, this interview procedure has been utilized in other training programs and large clinical trials (see https://clinicaltrials.gov/ct2/show/NCT02272192) for diagnosing ASD in toddler samples (Swanson et al. 2014; Warren et al. 2009). Psychological providers then completed a DSM-5 Symptom Checklist to indicate the presence or absence of core symptoms consistent with established ASD diagnostic criteria.

Screening tool for Autism in Toddlers and Young Children (STAT)

The STAT is an interactive, play-based Level 2 autism screening tool validated for use with children 14–47 months of age (Stone et al. 2000, 2004, 2008). It takes approximately 20 min to administer and consists of easily obtainable items such as bubbles, a car, pretend play toys, and snacks. Examiners administer 12 standardized activities in order to elicit child behaviors in four core categories: play, imitation, directing attention, and requesting. A “pass” (meaning that the play or social-communication skill was observed) is scored as a 0. Scores on each activity are tallied to provide a total score of 0–4, with scores greater than 2 indicative of autism risk (i.e., higher scores reflect more autism symptoms).

Mullen Scales of Early Learning (MSEL)

The MSEL (Mullen 1995) is a standardized cognitive test for children ages birth through 60 months. It consists of four subscales: visual reception, fine motor, receptive language, and expressive language, as well as an overall early learning composite.

Vineland Adaptive Behavior Scales—Second Edition (VABS-2)

The VABS-II (Sparrow et al. 2005) is a semistructured caregiver interview of adaptive behaviors observed within the home environment. It yields four subdomain scores (language, social, daily living, motor) as well as an overall adaptive behavior composite (ABC).

Autism Diagnostic Observation Schedule—Second Edition (ADOS-2)

The ADOS-2 (Lord et al. 2012) is considered the “gold standard” diagnostic tool for assessing for the presence of Autism spectrum disorder. It takes approximately 45–60 min to administer and involves a series of semistructured, interactive, largely play-based tasks. Administration and scoring of the ADOS-2 requires specialized clinical training. All administrators for this study had obtained both clinical and site research reliability. For this study, Module One (ages 31 months and above with single or no words) and the Toddler Module (children 12–30 months without flexible phrase speech) were administered. The ADOS-2 yields three scores, with higher scores indicative of more autism symptoms: social affect (SA), restricted and repetitive behavior (RRB), and a total score.

Procedure

Caregivers completed informed consent procedures in compliance with institutional review board standards. Children were assessed with the STAT while the remote psychologist observed. After the STAT concluded, the psychologist interviewed the caregivers using a DSM-5 clinical interview template. At the end of the procedure, the psychologist completed ratings of clinical best estimate (CBE) diagnosis and diagnostic certainty (Yes / No ASD classification, Likert rating of certainty ranging from 1 = ‘uncertain’ to 5 = ‘very certain’). However, caregivers were not provided with diagnostic information directly from the remote psychologist. Instead, upon completion of telemedicine assessment, families immediately received a blinded in-person comprehensive assessment conducted by a different licensed psychologist. This assessment included administration of measures of cognitive functioning (MSEL), adaptive behavior (VABS-II), autism symptoms (ADOS-2), as well as use of the same DSM-5 clinical interview template. The blinded provider then assigned CBE and certainty ratings, and provided information to the family.

Results

We first examined diagnostic classifications and certainty ratings for remote psychologists. When forced to make a binary choice (ASD: Yes/No), psychologists classified 75% of children (n = 15) as having ASD based on the telemedicine diagnostic procedure. Of the five children rated as “No,” 2 were classified with global developmental delay, 1 with language delay, and for two cases the clinician did not feel they had enough information to make a diagnosis. Telemedicine psychologists rated themselves as “certain” or “very certain” about classifications for 75% of cases. All of the children who were classified with ASD in the telehealth procedure scored above the recommended ASD risk cutoffs (> 2) on the STAT (mean = 2.59, SD = 0.96; range 2–3.75). Only one child in the non-ASD classified group scored in the ASD risk range.

Nineteen out of 20 children evaluated by the in-person psychologist were diagnosed with ASD. All of the children (100%, n = 15) classified with ASD based on the telemedicine procedure were also rated as having ASD by the blinded in-person psychologist conducting the comprehensive assessment. All of these children either scored above ADOS-2 Module 1 cutoffs for autism (n = 4) or within the ‘Moderate to Severe’ range on the ADOS-2 Toddler Module (n = 11). ADOS-2 Comparison (mean = 6.33, SD = 1.23), MSEL Early Learning Composite (mean = 58.33, SD = 10.22), and VABS-2 adaptive behavior composite scores (mean = 70.80, SD = 5.58) suggest these children exhibited fairly severe profiles of developmental impairment and ASD symptoms. In 20% of cases (n = 4), remote psychologists did not classify children as having ASD based on the forced yes/no choice, but children went on to receive CBE ASD diagnoses based on comprehensive evaluation.

For the diagnostically discrepant group, non-parametric comparisons of median values (Mann–Whitney U) found a significant difference in distribution of STAT scores (p < .05), with 3 of the 4 scoring below ASD risk cut-offs (as compared to only 1 out of 16 of children in the consistent ASD diagnostic group). Although not statistically significant, these 4 children tended to evidence lower levels of ASD symptoms and higher cognitive and adaptive behavior skills, including lower ADOS-2 Comparison Scores (mean = 4.25, SD = 0.50), higher MSEL Early Learning Composites (mean = 66.00, SD = 11.76), and higher VABS-2 adaptive behavior composites (mean = 76.00, SD = 7.48). In addition, lower certainty ratings were provided for these children by psychologists involved in the remote assessment (ASD confirmed cases: mean = 4.13, SD = 0.83; CBE only ASD identification: mean = 3.25, SD = 0.96) as well as comprehensive in-person assessment CBE clinicians (ASD confirmed mean = 4.60, SD = 0.63; CBE only ASD identification mean = 3.50, SD = 0.58). One child (Table 1, final column) was identified with global developmental delay in both the telemedicine and comprehensive evaluation procedure.

Utilization of the telemedicine assessment resulted in sensitivity of 78.95%. We were unable to test the specificity of our approach due to the clinical severity of our sample and lack of a non-ASD control group. Similarly, we did not calculate positive or negative predictive value. Ultimately, this preliminary feasibility study provided support for the use of telemedicine to accurately identify a significant proportion of young children with ASD who had already been flagged as showing developmental concerns. It also highlighted challenges accurately detecting some young children via this procedure.

Study 2: Feasibility, Family Perception, and Clinical Value

Participants

Participants were 45 children (mean age = 26.80 months, SD = 3.12, range 19–32; 35 boys, 10 girls) and their caregivers (see Table 2). Participants were consecutively referred over an 11 month period by a regional early intervention center due to ASD concerns. Two invited families declined participation. We partnered with a regional health center serving a rural 23 county region, geographically distant from the urban diagnostic centers of our state. Specifically, families served within this Part C catchment were on average 144.75 (SD = 24.54) miles from the primary diagnostic center and 37.6 (SD = 23.61) miles from the facility where the telemedicine procedures were conducted. Access to the diagnostic center was not contingent upon participation in this program. According to the most recent U.S. Census estimates, the mean population of the counties represented was 34,408; median household income, $40,541.20 (range $29,893–$55,972), and percentage of persons living in poverty was 20.25 (range 16–42.7).

Table 2 Study 2 key variables by diagnostic outcome
Full size table

Procedure

Psychologists and STAT administrators in study 2 were blind to the results of study 1. The same remote diagnostic procedures in study 1 were repeated, including Likert ratings of ASD diagnostic certainty. In this study, the in-person early interventionist administered the STAT while the remote psychologist provided diagnostic information, support, and resources based solely on the telemedicine consultation. Importantly, these diagnoses were not considered research classifications, but were clinically issued by the evaluating psychologist. Psychologists also reported any technical challenges and rated their satisfaction with the evaluation and disclosure. Immediately following the evaluation, families were asked to complete anonymous ratings of satisfaction with the tele-diagnostic visit.

Results

As seen in Table 2, the remote psychologists provided an ASD diagnosis for 64.44% of children (n = 29), ruled out an ASD diagnosis in 22% (n = 10), and deferred rule-in/out of ASD to a full evaluation in 13.33% of cases (n = 6). Average STAT scores for children diagnosed with ASD were 3.13 (SD = 0.66); for children where ASD was ruled out, 1.81 (0.84); and where diagnosis was deferred, the average STAT score was 1.25 (0.81). A breakdown of participants across final telediagnostic groups as broken down across STAT risk categories is presented in Table 3.

Table 3 Ns of study 2 participants broken down by STAT risk status and final telediagnosis
Full size table

When asked to rate clinical certainty, psychologists noted that they felt ‘certain’ or ‘very certain’ of their decisions in 86.67% of cases (n = 39). The highest certainty ratings were for children who received an ASD diagnosis (mean = 4.79, SD = 0.62), the middle for children where ASD was ruled out (mean = 3.9, SD = 0.88), and the lowest for children where definitive ASD diagnosis was deferred (mean = 3.33, SD = 1.37). Psychologists said they were ‘satisfied’ or ‘very satisfied’ with the remote screening technology in 80% of cases (n = 36) but would have preferred to see a patient in person 24% of the time (n = 11), primarily for children who received diagnoses of delay (n = 6), often due to complicating psychosocial factors (e.g., trauma history), technical challenges with the telemedicine platform, or complex diagnostic profiles.

Regarding technical challenges, concerns about aspects of audio (40%) and visual quality (31%) were rated as frequently present by psychologists in study 1, while such challenges were less common when observing visits at the rural clinic assessment location in study 2 (23.8% audio, 4.8% video quality concerns). Examples of such challenges included low audio volume or video lag, none of which was extensive enough to disrupt the evaluation process.

Most participating families (91%) returned the anonymous questionnaire regarding satisfaction with the telemedicine consultation with very high levels of satisfaction documented (see Table 4). As part of the survey, families were given the chance to provide a free text response to the question, “What suggestions do you have for improving telemedicine visits?” Of the 45 caregivers surveyed, 21 did not answer this question. Of the 24 who did reply, 21 responded with comments such as “None” or “N/A”. Three respondents offered the following suggestions: “Longer child evaluations,” “Sound,” and “Fix video,” without offering additional information.

Table 4 Caregiver tele-diagnostic satisfaction questionnaire
Full size table

The telemedicine evaluation saved families an average of 3.92 h (SD = 1.59, range 1–7 h) of estimated travel time relative to visiting the hospital-based ASD clinic. When asked about what services they would have sought out if the telemedicine evaluation was not available, most families (90.24%) indicated they would have waited to travel to the diagnostic clinic, with only 9.76% indicating they would have attempted to pursue another alternative.

Discussion

Although there is a small and growing literature base evaluating the potential feasibility and value of telepractice for supported behavioral interventions for children with ASD, there is no data regarding telemedicine supported early diagnosis. The two studies presented as part of this pilot work represent the first known examination of the feasibility, acceptability, and validity of telemedicine based diagnosis of ASD in very young children. We explicitly evaluated two critical components relevant to understanding the social validity of future telemedicine applications: (1) whether we could design a telemedicine procedure that could accurately identify children with ASD at young ages, and (2) whether clinicians and families would report that such a procedure was acceptable.

In terms of diagnostic accuracy, remote clinicians accurately identified 78.9% (15 / 19) of all children who ultimately received ASD diagnoses based on a blinded comprehensive assessment with gold-standard identification tools. No children were inaccurately classified as having ASD based on the telemedicine consultation in the small validity sample. Together, these results suggest that within referral populations seeking evaluation due to high levels of developmental concern, ASD can be accurately identified via remote observation and interviewing. Certainly, further evaluation in a broader population of children is necessary to understand the true ability of such an instrument to accurately identify ASD, as well as the percentages of children that might not be picked up by such a procedure or who could be misclassified as evidencing risk. Regarding this later point, our findings are aligned with a larger service system intervention supported by investigators involved in this work (Swanson et al. 2014; Rotholz et al. 2017), documenting very limited false positive identification rates when using a standardized, interactive screening tool such as the STAT in referral populations.

Recent technological advances have provided the opportunity to assess feasibility of telepractice for diagnostic and intervention services. Although the present studies rely on fixed point-tilt-zoom cameras in clinic rooms for remote diagnostic assessment, families in study 2 receiving subsequent tele-intervention services had access to inexpensive and portable “telepractice kits,” which included a low-effort technology suite. This suite consists of a robotic arm (Kubi), tablet (iPad Pro 9/7-in) with a video conferencing platform application (Zoom), Wi-Fi units, which when connected allow far-end control of the devices with full pan-tilt-zoom capability to distal providers. These kits, which can be mailed, dramatically increase families’ access to expert input, behavioral coaching, and diagnostic consultation within their own homes. Although the included studies required families to visit a health care facility, future work will examine ways of reaching families who may have difficulty accessing such sites, with the hope of further demonstrating the utility of telepractice for reaching underserved communities.

Importantly, this preliminary work found that across both studies, further evaluation was necessary or recommended for some children with more complex diagnostic profiles. Administration of the ADOS-2 as well as other adaptive or cognitive measures allowed in-person diagnosticians to better describe some children whose ASD symptoms were less evident on the STAT, which does not directly assess for characteristics such as repetitive play, stereotyped body movements or vocalizations, difficulty with transitions, or other behaviors that are coded as part of an ADOS-2. It is also possible that some children only show their subtle social vulnerabilities when given sufficient time within structured interactions with an examiner. It is very important to emphasize that the developed telemedicine procedure is not offered as a replacement for more in depth evaluation of all children, nor is it likely appropriate for all families seeking answers about their children’s developmental functioning. However, telemedicine diagnostic consultation may represent an accurate and acceptable confirmation procedure wherein a percentage of children, already identified as at risk by a medical or early intervention professional, may be identified without need for immediate extended evaluation—and the often lengthy associated wait times, during which children are missing valuable opportunities for early intervention. A promising compromise may be that diagnostically complex children can be identified via telemedicine evaluation as “at risk” and moved into services while they wait for extended evaluation (Rotholz et al. 2017). For a substantial proportion of our referral population, however, our telemedicine procedure was sufficient for capturing children’s diagnostic profiles in a way reported as highly acceptable by their caregivers.

In terms of feasibility and social validity, embedded use of the telemedicine procedure within a rural health center setting was able to provide diagnoses and access to follow-up care for a large number of children and families. Importantly, these families would have experienced at a minimum geographical and time barriers when accessing traditional hospital-based ASD diagnostic center evaluation. Families also reported very high levels of satisfaction with service. In the current work, identification was taking place within the context of an early intervention center affiliated programs where staff and service coordinators would be assisting children with subsequent early intervention services. It is logical to hypothesize that this could in fact represent a preferred practice for families rather than traveling to a diagnostic center far from their home to meet and receive a diagnosis from a provider they may never encounter again. There is work in other areas of psychological practice highlighting family preference for services received within the medical home (Chomienne et al. 2011; Clatney et al. 2008). Recognizing that participating families had this support may have also increased remote psychologists’ comfort with the diagnostic assessment procedure and willingness to offer a clinical diagnosis remotely. It remains to be seen whether the same procedures would be as widely accepted by clinicians or families if such follow-up was more limited or unspecified.

Interestingly, the time associated with these evaluations (i.e. 1 h of clinician tele-present time scheduled for each session) may represent a potent service system advantage involving scalable diagnostic programs over time. However, any movement toward such pragmatic evaluation would need to incorporate the resources necessary to train/support a teleprovider at the remote site (i.e., technicians were needed to administer the STAT) or demonstrate effective use of a novel method (e.g., clinician guided parent interaction, remote assessment applications). Additionally, psychologists providing remote diagnosis provided ratings of certainty, satisfaction, and desire to see a patient in person, but additional rationale for such was generally not provided. In future iterations of this work, it will be important to gather more feedback from diagnosticians regarding which child, family, and psychosocial factors influenced these ratings.

There are several major limitations of the current study. The included samples were relatively small and, compared to other existing studies, over-represented in terms of ASD and levels of impairment. In particular, the lack of a non-ASD control group for study 1, which reflects the clinical referral population of our study sample, impacted our ability to draw conclusions regarding the sensitivity and predictive values of the telemedicine procedure. In future iterations of this work, it will be crucial to examine how telediagnosis functions for a larger, more diagnostically diverse sample. Additionally, the small sizes of our samples meant that we lacked adequate power for sufficiently for understanding the clinical characteristics of children, families, and providers that may influence accuracy and acceptance of this procedure. Our work provides some indication that more mildly symptomatic children may not be as accurately described by a remote screening procedure as opposed to a full, in-person diagnostic evaluation. Understanding which children can be identified more readily and which may require a visit to a tertiary care facility will be an important goal for future work if the ultimate desire is to triage and efficiently streamline referral pathways for children with developmental concerns. Additionally, families and clinicians were also not randomly assigned or allocated to specific service modes introducing some risk of bias into study. Further, the clinicians accomplishing the telehealth evaluations had expertise and training in ASD assessment that may not be as prominent outside of specialized centers and programs. Evaluating how this model works with clinicians at varying levels of experience, or without the presence of an expert early interventionist for telemedicine modalities, will likely be important when making decisions about its broader implementation. Despite these limits, results of the current study highlight the potential benefit of telemedicine as a tool that may be beneficial in pursuit of systems level processes capable of identifying children with ASD at an early age.

In conclusion, it is becoming abundantly clear, particularly in low resource communities, that novel approaches to ASD diagnostic decision making are necessary to advance early detection and intervention for these vulnerable populations. In the current work, we tested the feasibility, acceptability, and accuracy of telemedicine diagnostic evaluation procedure for early identification of ASD. This pilot work demonstrated that a large percentage of children with ASD may be accurately diagnosed via remote observation of standardized assessment procedures, and many families and providers ascribe clinical value to the procedure. Whether such telemedicine-based diagnostic consultation is realistic across a range of broader range of settings, and whether it contributes to enhanced age of diagnosis and engagement with appropriate treatment on a population level, remain important research questions.