Introduction

Although autism spectrum disorder (ASD) is most often diagnosed in childhood, the diagnosis is frequently missed (Russell et al. 2010). Delayed identification could result from a number of factors, including an under-resourced familial or educational environment, the presence of only subtle ASD symptoms, and high intelligence that compensates for social and communicative difficulties (Wing and Potter 2002). When undiagnosed individuals reach adulthood, clinicians face additional assessment challenges, including limited diagnostic tools designed for this age group. The purpose of this study was to evaluate the diagnostic accuracy of the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2; Lord et al. 2012) with clinically complex adults served in community mental health centers (CMHCs).

Most adults with ASD have at least one co-occurring psychiatric disorder (e.g., Buck et al. 2014; Croen et al. 2015; Lugnegård et al. 2011). ASD symptoms overlap with those of many mood, anxiety, psychotic, and personality disorders, thereby complicating differential diagnosis. In addition, an ASD diagnosis requires that symptoms be present from early childhood, but it is often difficult to find family members or caregivers who can provide details about this early time period for adults. These challenges have resulted in many adults with ASD remaining undiagnosed, experiencing unmet service and treatment needs (Garland et al. 2013; Mandell et al. 2012). In order to provide appropriate services for adults with ASD, clinicians need tools that accurately distinguish ASD from other psychiatric disorders in adulthood.

To this end, the performance of diagnostic instruments like the ADOS-2 must be understood within clinically complex samples. By clinically complex, we mean individuals in a community outpatient setting who commonly have chronic mental health conditions, multiple co-morbidities, and/or complicating medical conditions or substance use histories (Lopez and Basco 2015). The ADOS-2 is a widely used, semi-structured assessment tool that allows systematic and standardized evaluation of the presence of ASD symptoms. It includes five modules (Lord et al. 2012): the Toddler Module for children aged 12–30 months without phrase speech, Module 1 for children aged 31 months and older without phrase speech, Module 2 for children with phrase speech who are not verbally fluent, Module 3 for children and young adolescents with fluent language, and Module 4 for older adolescents and adults with fluent language. While insufficient on its own for a diagnosis, the ADOS-2 is considered the field’s “gold-standard” for collecting standardized and objective information about social communication skills, restricted interests, and repetitive behaviors. Previous studies have found Module 4 of the ADOS and ADOS-2 to be fairly effective at discriminating between adults with and without ASD in university- or lab-based settings, with a sensitivity for ASD of at least 80% and specificity of at least 70% (Hus and Lord 2014; Lord et al. 2000; Pugliese et al. 2015). All three of these studies included participants who were referred to an ASD or developmental disorders clinic, as well as a smaller control group of typically developing individuals and individuals with non-ASD diagnoses (e.g., mood disorders, anxiety disorders, attention-deficit/hyperactivity disorder).

While these studies find that the ADOS-2 is effective in discriminating between individuals with and without ASD in research settings, it is unclear whether the same results generalize to community clinics that serve diagnostically heterogeneous individuals, often with serious mental illness. Studies to date were enriched for adults with ASD, which could reduce the potential to identify false positives, thus inflating the accuracy of the instrument. The characteristics of the non-spectrum group in these studies also make it difficult to generalize to the clinical setting of a CMHC. For example, Lord et al. (2000) only included individuals in the non-spectrum group if they scored below the autism cut-off on the Autism Screening Questionnaire (ASQ; Berument et al. 1999), meaning that individuals with subthreshold ASD symptoms were excluded. In addition, individuals with psychosis were not included in the Lord et al. (2000) or Hus and Lord (2014) studies. Pugliese et al. (2015) excluded adults with intellectual disability and included only three participants with psychotic symptoms. In CMHCs, the prevalence of psychosis is higher, and ruling out psychosis is likely an important component of an ASD evaluation. Although distinct in many ways, there are clear areas of symptom overlap between psychosis and ASD (see Chisholm et al. 2015; King and Lord 2011 for reviews). This is particularly true for the negative symptoms of psychosis (e.g., affective flattening, poverty of speech, social withdrawal), which are similar to the core social communication impairments in ASD. In addition to multiple phenotypic similarities between the two disorders, schizophrenia and ASD share risk factors and co-occur at elevated rates (Chisholm et al. 2015).

Two studies have shown that Module 4 may not perform well in differentiating between ASD and psychosis. Bastiaansen et al. (2011) found that the ADOS domain and total scores did not significantly differ between adults with ASD and adults with schizophrenia using the original algorithm. The Module 4 only correctly classified 74% of cases, with a sensitivity of 61% and specificity of 82%. Using the same sample, de Bildt and colleagues (2016) applied the revised Module 4 algorithm (Hus and Lord 2014) and also found that the ADOS did not discriminate well between ASD and schizophrenia, with a sensitivity of 61% and specificity of 50%. Bastiaansen et al. (2011), however, found three ADOS items on which adults with ASD scored significantly higher than adults with schizophrenia: stereotyped language, quality of social response, and overall quality of rapport.

While these two studies give some insight into the ability of Module 4 to distinguish between individuals with ASD and individuals with clinically complex presentations, they have several important limitations. First, the study protocols did not include a review of developmental history in the schizophrenia group, so they were unable to rule out the possibility that ASD symptoms were present in early childhood before the onset of schizophrenia. Second, the studies relied on a relatively small sample of adults with schizophrenia (n = 18) and only included adult males, which limits the ability to generalize the results. To address these issues, we examined the accuracy of the ADOS-2 Module 4 in a clinically complex community sample, and included a developmental history for all participants. Specifically, we evaluated the ability of the ADOS-2 Module 4 to discriminate between adults with and without ASD.

Methods

Procedures

The study was approved by the institutional review board of the University of Pennsylvania. The study took place in partnership with two CMHCs. Horizon House is a community mental health agency serving individuals in Philadelphia, the surrounding counties, as well as throughout the state of Delaware. It provides integrated psychiatric, medical, and behavioral services to adults. Hall-Mercer Community Behavioral Health Center provides outpatient services to children and adults with psychiatric disorders and developmental disabilities. It is affiliated with the University of Pennsylvania and its teaching hospitals, with a specific mission to care for the underserved in Philadelphia and surrounding areas. As in many CMHCs, assessment and treatment protocols vary considerably depending on the treatment team and the perceived needs of the individual in treatment.

Community clinicians at these two CMHCs asked their clients for permission for research staff to contact them about the study, clarifying that participation would be voluntary and would not affect their care. Staff obtained a signature authorizing or declining contact. Study staff reviewed participants’ charts prior to the in-person evaluation. Abstracted chart information included demographics, previous and current diagnoses, age of onset of psychiatric symptoms, substance use, trauma history, family psychiatric history, psychotic symptoms, current medications, past hospitalizations, and educational history. Any mention of ASD in the chart (e.g., a suspected diagnosis or rule-out) was also noted. In addition, clinicians completed the Social Responsiveness Scale for Adults (SRS-A; Constantino and Gruber 2012) for each client prior to the in-person evaluation.

The clinical evaluation team members (EB, MC, BM, JM) were blinded to the SRS-A results at the time of evaluation, but were provided chart review information. Evaluations took place in one of two CMHCs. They began with the ADOS-2 (Lord et al. 2012). The ADOS-2 examiner completed algorithms for the ADOS-2-WPS version and the ADOS-2-Revised (Hus and Lord 2014). The WPS algorithm provides separate cut-off values for the Communication and Social domains, as well as a cut-off for the sum of the two domains, to assign a classification of autism, autism spectrum, or non-spectrum. The revised algorithm is divided into two domains, consistent with the DSM-5 diagnostic criteria for ASD (APA 2013): Social Affect (which combines communication and social behaviors) and Restricted and Repetitive Behaviors. The revised algorithm has a single cut-off value for the combined domain total.

Following the ADOS-2, the participant was administered the Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Overview and Psychosis modules (SCID-IP; First et al. 2002). The SCID overview includes a series of standardized questions to establish a timeline (i.e., onset and course) of psychiatric symptoms and treatment history. To obtain further information about developmental history, selected social and educational history questions from the Kiddie-Schedule for Affective Disorders and Schizophrenia (K-SADS; Kaufman et al. 1997) were administered (e.g., questions about developmental milestones, friendships during childhood, academic performance in grade school, inclusion in special education). The clinical evaluation ended with several questions to assess any ASD symptoms not observed by the clinicians or reported by the participant (e.g., stereotyped motor movements, restricted interests, hyper- or hypo-reactivity to sensory input), guided by DSM checklists for an ASD diagnosis (including both DSM-IV-TR and DSM-5 diagnostic criteria). We also attempted to obtain developmental history information from a family member of participants, with their permission to complete a phone interview with a selected family member. This family interview included the Social Communication Questionnaire (SCQ; Rutter et al. 2003) and other questions about early development.

Following each evaluation, the clinical team convened to review evaluation results and come to diagnostic consensus. The clinical team members were doctoral level clinical researchers with expertise in research reliability with the ADOS and assessment of ASD and psychosis through the lifespan. Three of the four team members (EB, BM, JM) brought more than 50 years of collective experience with the diagnosis of ASD in adults; EB, MC, and JM also brought more than 50 years of collective experience with the diagnosis of psychosis in adults. The clinical team used all available information (i.e., chart review, ADOS-2, SCID, K-SADS questions, behavioral observations, follow-up questions about ASD symptoms) to complete ASD diagnostic checklists (both DSM-IV-TR and DSM-5). For the purposes of this study, participants who met the DSM-5 diagnostic criteria for ASD received a diagnosis of ASD. The clinical team also determined whether the participant had a lifetime history of psychotic symptoms (specific psychotic disorders were not differentiated), and whether other psychosocial/developmental/neuropsychiatric issues (e.g., possible intellectual impairment, history of trauma, mood, anxiety, or personality disorders) were present. The team paid close attention to developmental history, symptom onset, and symptom course, as well as whether the current presentation (e.g., flat affect, social withdrawal) could possibly be attributed to medications.

Participants

CMHC consumers aged 18 or older who were served through outpatient behavioral health or case management programs were eligible to participate in the study. We excluded individuals served through developmental services because of our focus on Module 4, which can be used with adults with mild to moderate intellectual disability, but requires at least fluent verbal language. A total of 312 eligible clients completed consent-to-contact forms, with 169 (54%) providing permission for study staff to contact them about the clinical interview. Study staff called these adults a maximum of three times each in an attempt to contact them. Recruitment efforts resulted in a total of 75 participants completing the in-person evaluation. Information about the final sample is presented in Table 1. Of note, these 75 participants did not differ, on average, from the 237 clients who did not complete the in-person evaluation on SRS score (p = .23). Of the 75 participants, 29 declined to allow the research team to contact a family member for a phone interview about developmental history, and 15 did not have contact information for a family member who knew them prior to the age of 7 years. Family members of the remaining 31 participants were contacted via telephone, and 15 completed phone interviews.

Table 1 Sample characteristics
Full size table

Data Analyses

Primary analyses determined diagnostic agreement between the ADOS-2 classification and a study diagnosis of ASD based on DSM-5 criteria. We calculated the rates of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), using both the ADOS-2-WPS and ADOS-2-Revised algorithms. Secondary analyses examined the mean ADOS-2 domain scores and item scores by group (TP, TN, FP) to explore which types of behaviors are associated with FPs. We divided the TN group into those with and without psychosis for these analyses because all FPs had psychosis. Given the small number of TPs and unbalanced group sizes, we did not perform between-group statistical comparisons. However, we considered the clinical meaning of the ADOS-2 domain and item scores, based on the established cut-off values for domain scores and coding conventions for the item scores (i.e., scores of 0 suggest at least typical skill, and scores of 2 suggest behavior was either unusual, or unusually absent).

Results

Six of the 75 evaluated individuals met research criteria for ASD (one with co-morbid psychosis); 57 met criteria for psychosis (current or past). The remaining 12 participants had a mood disorder or another psychiatric condition. As shown in Table 1, of the six adults who met criteria for ASD, two had a documented ASD diagnosis in their charts and three had a mention of ASD in their charts. Among the remaining 69, two had a chart-documented diagnosis of ASD, and six had a mention of ASD in the chart.

Tables 2 and 3 display the sensitivity, specificity, positive predictive value, and negative predictive value estimates for the ADOS-2-WPS and ADOS-2-Revised algorithms, respectively. All six individuals with ASD obtained ADOS-2 scores in the autism or ASD range on both algorithms (TP rate of 100%). 18 of the 69 (26%; WPS algorithm) and 21 of the 69 (30%; revised algorithm) without ASD had ADOS-2 scores in the autism or ASD range, but in our clinical opinion did not have an ASD. The FPs were not consistent for the two algorithms: 7 individuals who were TN on the WPS algorithm were FP on the revised algorithm; 3 who were FP on the WPS algorithm were TN on the revised algorithm. All FP participants using either algorithm (n = 24) had a lifetime history of psychosis symptoms. We conducted all remaining analyses on both algorithms, and found very minor differences. Because the revised algorithm is updated and more likely to be adopted by researchers, we present the following results based on that algorithm. Results based on the WPS algorithm are available from the authors.

Table 2 Sensitivity, specificity, and predictive power of the ADOS-2 (WPS algorithm)
Full size table
Table 3 Sensitivity, specificity, and predictive power of the ADOS-2 (Revised algorithm)
Full size table

As shown in Table 4, the TP and FP groups had nearly identical mean scores on Social Communication, which were well above the total cut-off score of 8 suggestive of an ASD (Hus and Lord 2014). In contrast, the TP and FP groups had clinically different scores on the Restricted Interests/Repetitive Behaviors domain (4.7(0.8) and 2.0(1.4), respectively). The TN-Psychosis and TN-Other had Social Communication scores that are well below the ASD cut-off, and Restricted Interests/Repetitive Behavior scores that are close to zero. In addition, the FP group showed similar scores to the TP group on several items rated based on social communication behaviors (e.g., use of gestures, facial expression, and eye contact) that could reflect the negative symptoms of psychosis, rather than the atypical behaviors associated with ASD.

Table 4 ADOS-2 domain scores (Revised algorithm)
Full size table

Discussion

To our knowledge, this study is the first to evaluate the accuracy of the ADOS-2 with a diagnostically diverse and clinically complex sample of adults served in a CMHC. Although the ADOS-2 accurately identified all six adults with ASD in our sample, it also resulted in a high rate of false positives. Our specificity estimates (74% for the WPS algorithm and 70% for the revised algorithm) are in line with some previous lab- or university-based studies (e.g., 72%; Pugliese et al. 2015), and lower than others (e.g., 82%; Hus and Lord 2014; 93%; Lord et al. 2000). Similar to Bastiaansen et al. (2011) and de Bildt et al. (2016), we found that the ADOS-2’s accuracy in discriminating between ASD and psychosis was particularly limited. Indeed, all 21 of our false positives had a lifetime history of psychosis symptoms. Of the 57 participants with psychosis, 37% exceeded the clinical cut-off score on the ADOS-2. Elevated ADOS-2 scores in the false positive group were driven primarily by high Social Communication domain scores, but not by high Restricted Interests/Repetitive Behaviors domain scores. This finding, with identical mean Social Communication domain scores (10.5) in the true positive and the false positive groups, is likely due to the negative symptoms of psychosis, which are similar to some core ASD symptoms (e.g., flat affect, limited conversation, reduced eye contact; APA 2013). Overall, our results support the overlap of symptom domains between ASD and psychosis (e.g., Chisholm et al. 2015). It is important for clinicians and researchers to remember that prominent social communication difficulties are not specific to ASD; particularly in a clinically complex setting such as CMHCs, where many patients may present with impaired social communication skills. Conversely, clinicians who have more experience with schizophrenia may be well-served to consider the potential of ASD among their clients who exhibit social impairments.

Our findings also serve as a reminder that the ADOS-2 was not designed to be used as a stand-alone diagnostic measure. If we had relied on ADOS-2 score alone for inclusion in the ASD group, 36% of our sample would have been classified as meeting ASD criteria instead of 8%. This high proportion is particularly notable given the low base rate of ASD in the population. For our consensus clinical judgment, we combined observations from the ADOS-2 with developmental history information. Developmental history (e.g., report of reciprocal friendships and typical communication skills during childhood, report of a clear psychosis onset accompanied by decline in social and occupational functioning) was often key in our differential diagnosis of ASD or psychosis.

Another notable finding is that only two of the six adults who met study criteria for ASD had prior ASD diagnoses in their charts. Undiagnosed ASD may be common in outpatient CMHC programs, similar to inpatient psychiatric hospitals (Mandell et al. 2012). Of the four adults with ASD who did not have a prior formal ASD diagnosis, three had mentions of possible “autism,” “pervasive developmental disorder (PDD),” or a similar term in their charts. Thus, at least one clinician involved in their care had previously questioned whether an ASD diagnosis would be appropriate. The possibility of ASD appears to be considered in CMHCs, but not formally diagnosed in all cases, possibly due to few community resources for diagnosis and treatment of adults with autism spectrum disorders (Gerhardt and Lainer 2011). Among those who did not meet our study criteria for ASD, two participants had a prior ASD diagnosis and six had ASD-related terms mentioned in their charts, which highlights the challenges of accurately diagnosing ASD when the symptoms overlap significantly with symptoms of other psychiatric disorders.

Several study limitations should be noted. First, unlike prior studies, our in-person evaluations took place in a CMHC with adults currently receiving community services. Thus, our findings are based on a less selected sample than prior studies. Although this approach has several strengths, it also led to our recruiting a small number of adults with ASD, which limited our power for statistical analyses. Second, participants’ charts often lacked developmental history information, which meant that we relied on self-report during the in-person evaluation. Although we tried to take a developmental history (including the SCQ) from a family member for each participant, we were only able to complete family interviews with a small subset (n = 15) of our sample. Reasons for this low number include participant refusal to have us contact a family member, lack of eligible family member, no available family contact information, and inability to reach the family member. Lastly, the in-person evaluation did not include a structured assessment of cognitive functioning or of psychiatric disorders other than ASD and psychosis, which limits the characterization of our sample.

Despite these limitations, we found that Module 4 of the ADOS-2 provides important information about current social communication skills, restricted interests, and repetitive behaviors. However, the ADOS-2 should be used with caution as a diagnostic instrument with adults, particularly when the differential diagnosis includes the possibility of psychosis. ASD is the only disorder for which an interactive observational measure exists, but many of the social communication difficulties it measures are not unique to ASD. Thus, ADOS-2 results alone do not always lead to the same conclusions as experienced ASD clinicians would make when taking into account all the available clinical information, particularly when assessing adult outpatients served in CMH centers. The results of this study suggest that information from the ADOS-2 should be integrated with information from a thorough clinical assessment and developmental history whenever possible in order to maximize diagnostic accuracy. Future studies should determine if these results replicate in other samples, and also should investigate the accuracy of other measures in distinguishing between ASD and psychosis. While there may be clinical overlap in the broad domain of social communication, it may be possible to more clearly distinguish the specific features of social communication that are associated with either ASD or psychosis, but not both. Furthermore, given the difficulty of obtaining a developmental history from family members of adults served in CMHCs, future studies also should examine the minimum types and quantity of developmental history needed to make an accurate diagnosis in this setting. Based on our findings, these details about a person’s early years greatly inform diagnostic decision-making, and therefore influence appropriate treatment referrals.