Introduction

“Throughout the history of education, the adoption of instructional programs and practices has been driven more by ideology, faddism, politics, and marketing than by evidence” (Slavin 2008, p. 5). Perhaps nowhere is Slavin’s indictment of education’s failure to use objective scientific knowledge to inform and improve intervention more evident than in autism, where, despite increased research, funding, and public awareness, the field continues to struggle identifying, validating, and effectively implementing evidence-based practices (Callahan et al. 2008; Hess et al. 2008; Lord et al. 2005; Reichowet al. 2008). Although special educators and other public school-based autism service providers in the US have been mandated by law to adopt and deliver instructional programs based on research evidence (Horner et al. 2005; Yell et al. 2005, 2006), the extent to which this is broadly practiced is uncertain. Anecdotal reports by teachers, parents, and administrators suggest, however, that the inability of many school districts to deliver high quality autism programming is an area of widespread concern (Callahan et al. 2008).

The reasons why the research-to-practice disconnect is so deeply entrenched within autism intervention in public schools have not been fully investigated (Reichow et al. 2008). However, it is reasonable to conclude they are related to the lack of empirically validated Comprehensive Treatment Models (CTMs), and, consequently, sources of information and training for teachers and parents that vary greatly in their quality and veracity. Equally problematic is the vulnerability by consumers of autism services to unreliable and unsupported claims of effectiveness (INSAR 2008; Howlin 2005), a situation in which, ostensibly, “ideology, faddism, politics, and marketing” can play a powerful and potentially insidious role in the quality of classroom programming.

Indeed, for educators and parents intent on discovering current information about effective autism programs and interventions, only a relatively small number of evidence-based comprehensive program models (using Odom et al.’s 2003 definition: a combined “set of components to form a model” [p. 166]) have been broadly disseminated (e.g., National Research Council 2001; Odom et al. 2008). The National Research Council’s seminal review, for example, examined only 10 representative models, although more recent efforts have identified and systematically evaluated additional CTMs or their individual components (e.g., Eikeseth 2009; Odom et al. 2008; Wilczynski 2007).

Two program models in particular, Applied Behavior Analysis (ABA) and Treatment and Education of Autistic and related Communication handicapped Children (TEACCH), are among the most widely known comprehensive intervention models (Gresham et al. 1999; Humphrey and Parkinson 2006). Both of these treatment models feature several decades of development and use with students with autism (e.g., Mesibov et al. 2006; Lovaas 1987), robust national training networks (ABA is associated with national certification programs at the bachelor’s and master’s degree levels), and prominent attention both within the popular media and university teacher preparation and research communities. Thus, ABA and TEACCH are firmly established as among the most visible and frequently cited autism programs (Gresham et al. 1999) and they are among the most broadly requested and implemented public school treatments by educators, service providers, and parents (Choutka et al. 2004; Hess et al. 2008).

Despite their preeminence in the world of autism treatment, ABA and TEACCH are often viewed by proponents and consumers as competing (Choutka et al. 2004) and/or mutually exclusive (Eikeseth 2009) treatments, and it has been difficult to reach consensus about their equality as model programs (Lord et al. 2005). Indeed, it is not difficult to find descriptive materials associated with these models which contain examples of negative or critical language about the fundamental principles and practices of the rival model. For example, note the following passage from an early TEACCH training manual about the use of behavioral techniques in classrooms for students with autism and other disabilities:

Our experience has taught us that it is a mistake to confuse operant conditioning techniques for changing behavior with programming special education for autistic and similar children. When this technology is used instead of sound educational and developmental principles, thoughtful and individualized interaction between adult and child is too often sacrificed for pseudo-scientific techniques (Schopler et al. 1980, p. 115).

This verbiage is clearly dated and the philosophy of the TEACCH developers and trainers has undoubtedly shifted since the original publication, but more recent comments still emphasize perceived fundamental differences in philosophy and practice between ABA and TEACCH (Mesibov 2001).

Some advocates of Applied Behavior Analysis, variously referred to as “ABA only” (Prizant 2009) and “ABA is the only way” (Rimland 1999), are equally vehement about the perceived primacy of this approach for teaching students with autism (e.g., Howard et al. 2005; Maurice et al. 1996). The following conclusion in a popular ABA-based training manual typifies the strong partisan messages to which parents and professionals are often subjected: “Nonbehavioral special education classes…have not been established as effective treatments for children with autism” (Maurice et al. 1996, p. 56). The TEACCH model was included in the “alternative treatments” that were reviewed and to which these authors refer. Ironically, this text earlier discusses the “ferocious ideological warfare” (Maurice et al. 1996, p. 6) which dominates the autism world, a point of view with which we strongly agree. Nevertheless, such statements can make a significant impact on the selection of autism interventions by consumers, where, “in many cases, parents and advocates have identified a single program…discrete trial training…as the only approach that has evidence of effectiveness for young children with autism” (Odom et al. 2003, p. 166).

Despite the claims and counterclaims of some advocates, trainers, and other representatives of these leading autism program models, to our knowledge it has never been firmly established that the fundamental philosophies and practices of ABA and TEACCH are, indeed, oppositional and/or non-complementary. It may be the case, in fact, that these and other comprehensive intervention models share common components that are both socially valid (see, e.g., Callahan et al. 2008; Foster and Mash 1999; Kazdin 1977; Wolf 1978) and evidence-based (Odom et al. 2003).

The process of social validation is a critically important step in the much broader, but interrelated, enterprise of empirically validating effective educational or therapeutic outcomes (Foster and Mash 1999). Unfortunately, it is an area that has received only minimal attention in the autism research literature (Callahan et al. 2008). Social validity can be generally defined as consumer satisfaction with the goals, procedures, and outcomes of programs and interventions (Alberto and Troutman 2008; Wolf 1978). Whether or not a particular intervention—or a comprehensive treatment model—receives widespread social validation can determine the extent to which the intervention or model is adopted and implemented within schools, homes, and clinics (Gresham et al. 2004; Kazdin 1981; Kern and Manz 2004). Thus, ratings of social validity can provide an important indicator of the preferences of autism service providers for particular intervention components and for program models comprised of many such parts. Importantly, identifying a core of essential autism practices across the leading comprehensive treatment approaches could, by their effective implementation, result in significant improvements in the training, evaluation, and outcomes of public school autism programs.

The purpose of this paper is to investigate the social validation of the shared evidence-based practices inherent within the ABA and TEACCH treatment models for students with autism spectrum disorders, and to analyze the comprehensiveness and fidelity of these models. Specifically, we will analyze the socially validated, research-based components of these models and discuss the implications for effectively implementing these features within public school autism programs. Finally, we will discuss the relevance of our findings for the training, dissemination, and evaluation of school-based autism intervention models.

Method

In an earlier study (Callahan et al. 2008) a team of researchers including a graduate student and parent of a child with autism identified a list of evidence-based autism interventions that were socially validated by 187 parents, teachers, and administrators based on responses to a mail questionnaire. In our original survey, any terms identifying ABA or TEACCH were purposefully excluded from the items to avoid biasing the respondents. Thus, respondents expressed their opinions only about the perceived importance of each specific intervention component as part of a hypothetical high-quality autism program, rather than its role within a familiar and oftentimes emotionally invested treatment model. These consumers of autism services indicated strong, consistent support for specific treatment components falling within five key functional areas of a comprehensive public school intervention model, known by the acronym IDEAL: (a) Individualized programming (n = 16), (b) Data collection (n = 8), (c) the use of Empirically demonstrated strategies (n = 38), (d) Active collaboration (n = 15), and (e) a focus on Long-term outcomes (n = 7). In the present study we reanalyzed our survey responses in order to determine the relative support for ABA versus TEACCH, based on the socially validated IDEAL Model components inherent within each approach, and we examined the distribution of ABA and TEACCH components across the five IDEAL functional categories. Our purpose was to determine if either ABA or TEACCH enjoyed significantly greater support by teachers, parents, and administrators based on the evidence-based practices making up each model, and to analyze the comprehensiveness of these major treatment approaches in comparison to the IDEAL Model functional categories.

Participants and Procedures

In this study a team of researchers independently conducted a systematic review of the autism and special education literature to identify publications describing the ABA and TEACCH approaches and their application to autism spectrum disorders. The research team consisted of two university faculty members who teach doctoral and master’s level courses focusing on autism intervention, two doctoral students in an autism leadership and research program (these students had either participated in Basic Certification in Behavior Analysis (BCBA) or TEACCH-sponsored training) and a BCBA-qualified doctoral level student. The literature review illuminated seminal and other relevant publications describing the ABA and TEACCH models, which were analyzed individually and discussed as a group to ensure an adequate baseline level understanding of the key components of each model.

The IDEAL components (n = 84) were then independently rated by each member of the research team to determine if the item represented a practice clearly associated with the ABA Model, the TEACCH Model, both the ABA and TEACCH models, or neither model. These independent ratings were then analyzed and discussed in order to assign the IDEAL items into one of the four categories above. Items in which there was an inter-rater agreement of at least 75% were placed into the appropriate category. Items with less agreement were discussed and in some cases ratings were changed by one or more members of the research team before final categorization.

Expert Validation Process

In order to validate our categorization of which IDEAL intervention components represented the ABA and TEACCH approaches, we conducted a process of expert validation. Individuals with documented training and expertise in the ABA and/or TEACCH models were identified and asked to complete a survey to identify the items representing their model(s). Experts were defined as individuals who had completed national training in the respective model(s) and who were familiar with the use of the model within the field of autism intervention. ABA experts were required to hold BCBA certification. Experts were recruited from colleagues and acquaintances of the research team members throughout the United States. The experts remained blind to the purposes of the study. Eight ABA and eight TEACCH experts participated in the survey. A third group of eight experts consisted of individuals who had received certified training in both models. The validation survey directions required the experts to indicate which items they believe represented a “defining feature” of ABA or TEACCH, as that intervention component is commonly recognized and practiced today with students with autism spectrum disorders. Reliability coefficients were computed to determine the experts’ overall level of agreement with our assigned categories, with the following results: ABA = 77.8% agreement (R = 62.0–82.4); TEACCH = 70.6% agreement (R = 39.4–78.3); BOTH = 62.3% agreement (R = 53.5–71.4).

As a result of this expert validation process several items were discussed in detail, and some items were changed to a different category. The final categories included 19 ABA items, 15 TEACCH items, 37 BOTH items (interventions inherent within both the ABA and TEACCH approaches), and 13 items that are not applicable to either model. Table 1 contains a list of the items within each category, including overall mean social validation ratings (high to low) on a scale of one to seven (1 = “not at all important” and 7 = “absolutely important”), based on the results of our earlier analysis (Callahan et al. 2008). Item descriptions are exactly as the items appeared on the original social validation survey.

Table 1 Intervention components within ABA, TEACCH, “BOTH”ABA and TEACCH, and “Not Applicable” categories
Full size table

Results

Analysis of Social Validation Ratings, ABA Versus TEACCH

In order to determine whether evidence-based intervention components of the ABA model or TEACCH model were viewed more positively by the major consumers of autism programming (i.e., parents, teachers, and administrators), we analyzed social validation scores for the autism intervention components within each group. Table 2 contains a summary of the overall mean social validation scores, comparing the ratings of special education teachers, administrators, and parents. The two teacher groups represent special educators who teach only students with autism, typically more severe students in self-contained classroom settings (“Teachers Autism”), and those who teach a greater range of students with autism spectrum disorders within resource or life skills classrooms also containing a variety of other developmental disability categories (“Teachers Various”).

Table 2 Overall mean social validation ratings of evidence-based interventions within ABA, TEACCH, and “BOTH”ABA and TEACCH categories
Full size table

The mean social validation ratings were consistent across consumer groups, with the ABA intervention components being the lowest rated by all groups, and TEACCH components being rated similarly, but slightly higher, by all groups except the teachers of various disabilities. This group rated the TEACCH components much higher, a difference that was statistically significantly different than their rating of the ABA items, t (164) = −3.20, p < .01. All Means are relatively positive on our 1–7 scale of social validity, indicating a relatively high level of acceptability by all consumer groups. However, it is important to note that autism interventions reflecting both models were the highest rated items by all groups of consumers.

Analysis of Intervention Components Across IDEAL Functional Categories

The specific autism treatments within the ABA, TEACCH, and BOTH categories were analyzed in terms of their distribution across the IDEAL Model functional categories. Ostensibly, the IDEAL Model provides a comprehensive selection of socially validated activities which, if effectively implemented, would improve key outcomes for students in public school autism classrooms (Callahan et al. 2008). By analyzing the number and percentage of ABA and TEACCH intervention components within each IDEAL functional area, one picture of the comprehensiveness of these popular treatment approaches emerges (see Table 3). Note that significant gaps within important functional areas are present within both the ABA and TEACCH models alone. Although the ABA model is relatively rich in its use of empirically demonstrated strategies and its focus on data collection activities, it includes zero of the IDEAL components within both the collaboration and long-term focus areas, and only a single item in the individualized programming area. Interestingly, TEACCH is relatively well represented in the ABA deficit areas, but falls short in the area of data collection (zero components) and, like ABA, has only a minimal level of representation within the area of individualized programming. Importantly, the socially valid interventions that may play a vital role in both the ABA and TEACCH models appear to provide a much more balanced and comprehensive array of intervention components, accounting for 34–63% of the socially valid interventions within the five IDEAL functional areas.

Table 3 Number and percentage of ABA, TEACCH, and “BOTH” ABA and TEACCH intervention components within the IDEAL Model functional categories
Full size table

Statistical Analyses

Analyses were conducted to determine if respondents’ demographic variables resulted in significant differences in social validity ratings. Specifically, we examined the impact of self-reported knowledge of autism, level of autism training, primary source of training, and the severity and levels of autism functioning of the respondents’ students or children on their ratings of the intervention components. We also analyzed the impact of participation in formal ABA or TEACCH training, and formal training in both approaches, on social validity ratings. All analyses were run using Statistical Program for the Social Sciences (SPSS) software. Results of evaluations of assumptions of normality, and homogeneity of variance-covariance matrices were satisfactory for all tests, with one exception: Group variance in ABA items was not equivalent in the MANOVA for Level of Autism Knowledge. The analysis was run despite this violation.

The results of these analyses indicate few statistically significant effects based on the variables analyzed. Teachers who teach a mix of students including autism as well as other exceptionality categories tended to rate the TEACCH items higher. Respondents who reported lower levels of autism training generally rated the ABA items lower; similarly, persons with less self-reported knowledge of autism rated ABA interventions as less socially valid. Neither one’s source of training nor participation in formal training in ABA and/or TEACCH had a significant impact on social validity ratings. The severity and functioning levels of the respondents’ children with autism also had no significant effect on ratings. Detailed results of the statistical analyses are found in Table 4.

Table 4 Summary of statistical analyses of the impact of demographic variables on social validity ratings
Full size table

Expert Validation Results

Figure 1 depicts agreement outcomes of the expert validation process, by model. Each data point represents the number of expert validators, out of a total of eight experts for each model, who agreed with our categorization of individual survey items (as ABA, TEACCH or BOTH). “Yes” items refer to intervention component descriptions that we scored on the master list (i.e., the original social validation survey) as being representative of the respective model. For example, ABA data point 11 represents agreement with the master list on survey question 40, Instructional programming that targets all affected areas of development…, an item we categorized as representative of ABA. Only five of eight ABA experts agreed with this categorization. Interestingly, only three of eight ABA experts agreed with our assigned category on survey question 5 (data point 71), Use of punishment or aversive stimuli in order to decrease inappropriate behaviors, an item we categorized as representative of the ABA model.

Fig. 1
figure 1

Number of expert validators (ABA top panel, TEACCH center panel, or BOTH lower panel) that agreed with authors’ categorization of survey items, by item number

Full size image

“No” items on Fig. 1 refer to interventions we categorized as not representing the specific model. For example, on TEACCH data point 48 (survey question 6), Use of students’ preferences and/or obsessive interests as reinforcers…, the majority of TEACCH experts (7/8) disagreed with our categorization that this item was NOT a defining feature of the TEACCH model. Instead they marked this item as representing the TEACCH approach. On the BOTH chart, data points 13 and 60 represent survey questions 55, Ongoing data collection…, and 22, Systematic use of prompts…, items we categorized as not representative of both models. Agreement was zero for each of these items, indicating that all eight expert validators with certified training in ABA and TEACCH viewed these items as being a defining intervention component of both models.

Experts from all models (ABA, TEACCH, and BOTH) were more inclined to respond “Yes” to any given item than “No” (with the exception of item 71, question 5: Punishment). This trend is clearly seen by the concentration of (o) symbols near the top of each chart. This tendency was most pronounced with ABA experts and least pronounced with experts having training in both models. However, expert validators in the BOTH group were more likely to agree with “Yes” items that received high social validation scores, indicated by the decreasing trend in “Yes” responses.

Discussion

The purpose of this study was to investigate the social validity of evidence-based practices common within the ABA and TEACCH comprehensive treatment models (CTMs) for students with autism spectrum disorders. Our results indicate that the teachers, parents, and administrators in our sample showed no clear preference for the interventions associated with either the ABA model or the TEACCH model, when only the unaffiliated intervention components making up these models were rated. Significantly, however, the autism treatment components that were determined to be inherent within both the ABA and TEACCH approaches were rated as more socially valid than either approach alone.

Despite claims that certain CTMs offer the only effective treatments for students on the autism spectrum, our results suggest that the most important consumers of such treatments (special educators, parents, and administrators) favor using a combination of interventions. We believe that this finding has significant implications for developing and implementing valid, effective comprehensive treatment programs for serving students and children with autism spectrum disorders.

A preliminary analysis of the interventions representing both models (our “Both ABA and TEACCH” category) demonstrates a balanced variety of recommended activities including both general best practices (e.g., ensuring teachers are knowledgeable, experienced, and qualified in autism) as well as very specific curricula, materials, and techniques (e.g., the use of specialized strategies to teach social skills). The effective implementation of these items, in concert with specific ABA or TEACCH components (based on the skills, experiences, and desires of the parents and professionals implementing them), would arguably result in the most effective outcomes possible. More analyses of these items is warranted, including an assessment of other potentially relevant components of comprehensive classroom or home programs that may not have been identified in our original review. Quality of life indicators, for example, might include important intervention factors that are not well represented in the original list of IDEAL interventions.

In any case, these results suggest that neither of the most popular autism program models is dominant in terms of its social acceptability. These social validity data add to the research findings investigating the empirical validation of the major autism intervention programs, in which some researchers have concluded “there are no robust data favoring one approach over the others” (Francis 2005, p. 493), as well as researchers reporting that the effectiveness and quality of evidence for the currently available autism treatments have not yet been conclusively demonstrated (Gresham et al. 1999; Reichow et al. 2008; Rogers 1998). To this end, Odom et al. (2003) have proposed that autism researchers should focus on developing quality indicators for evaluating the effectiveness of evidence-based practices.

Important questions are raised in forwarding this conclusion. First, researchers in the field of autism must carefully define what is meant by comprehensive evidence-based programming. For example, the IDEAL Model providing the theoretical foundation for this study defines comprehensiveness as including five functional areas that cover a broad array of activities school-based multidisciplinary teams must implement effectively, ostensibly, to ensure successful school and life outcomes. These functional areas include critically important research-demonstrated actions that occur throughout the special education process (that is, from initial assessment and evaluation through post-public school transitions). It is important to note that the IDEAL Model conceptualization, although aligned with the NRC (2001) definition of comprehensive approaches, is still in development and its efficacy on student outcomes has not yet been empirically established. An alternative definition within the field of Positive Behavior Support (PBS) considers an intervention to be comprehensive “when it (a) addresses all problem behaviors performed by an individual; (b) is driven by the functional assessment; (c) is applied throughout the day; (d) blends multiple intervention procedures (change in structure, instruction, consequences); and (e) incorporates procedures that are consistent with the values, skills, and resources of the implementers” (Horner and Carr 1997, p. 94). Undoubtedly, many other definitions of comprehensive programming exist.

Our results indicate that based on some of these conceptualizations it may be questionable whether ABA or TEACCH can justify their reputations as true CTMs for children with autism, as both models appear to have significant gaps in providing key treatment aspects of hypothesized high quality public school programs. We believe an analysis of many, if not all, of the existing treatment models would likely illuminate similar treatment gaps. Thus, we recommend a renewed research focus on defining what is meant by comprehensiveness in public school autism programming and studying how the components of comprehensive treatment programs impact key outcome factors.

Fortunately, some researchers are currently investigating components of CTMs and their relationship to effective outcomes. These studies have focused on research-oriented aspects of autism treatment models that will hopefully allow for their systematic replication and evaluation (e.g., Odom et al. 2008; Smith et al. 2007; Wilczynski 2007). For example, Odom et al. (2008) have evaluated comprehensive treatment programs in terms of their operationalization, treatment fidelity, replicability, and peer reviewed evidence of outcomes. We believe that this and similar approaches will add much to the autism literature. For now, though, a key conclusion is that designating any autism treatment model as “comprehensive” should only occur after a process of objective empirical validation and only by the consensus of experts and informed consumers within the field at-large, rather than by the developers and advocates of the programs. At a minimum, any public school treatment model labeled “comprehensive” should be required to demonstrate that the components which make it up are evidence-based, address the full spectrum of students’ deficits and educational needs, and that they are effective on a wide array of outcomes for students and children with autism.

Defining comprehensive autism programming will also necessarily result in addressing the troubling and troublesome issue of what is meant by “eclectic” programming in autism. Within the current debates about the efficacy of ABA compared to other treatments, the word eclectic is being used in different ways by researchers and authors (e.g., Howard et al. 2005), perhaps adding unnecessarily to the confusion of teachers, parents and other consumers. Some, for example, use the word pejoratively or in a way that appears to diminish the value of individualized programming, a cornerstone of special education (e.g., Eldevic et al. 2006). Given the spectrum nature of autism and the emerging consensus that no single treatment works for all children with ASD such a philosophy seems questionable and unreasonably restrictive. The opposing view that “perhaps a complement of eclectic strategies is necessary to meet the complex challenges and spectrum of characteristics associated with autism” (Schoen 2003, p. 129) appears to us to be more in line with traditional special educational philosophies and practices. Lord et al. (2005) also address this key point and conclude that the main issues of eclecticism may be pragmatic. “Questions continue to emerge as to whether individual communities should endorse and fund one model, an array of models, or work primarily to embed effective practices into existing programs. While selecting from many resources implies a wealth of opportunities to individualize, one concern is that many providers engage in eclectic practices for which there has not been systematic planning or research.” (p. 704). The compelling question, in any case, might be, does not true comprehensiveness require some level of eclecticism? We believe this issue should be resolved systematically, empirically, and amicably.

Finally, our findings, especially the somewhat perplexing expert validation results, suggest that critically important issues related to training and treatment fidelity, as well as program evaluation processes in autism, must be immediately addressed. Smith et al. (2007) similarly point out the importance of “administering the [autism] intervention program, monitoring fidelity of implementation, and evaluating outcomes,” (p. 355), tasks they acknowledge are expensive and complex, but necessary, if the field is to continue to make significant gains. Likewise, Odom et al. (2008) conclude that “a great need exists for efficacy trials for most comprehensive treatment models [CTMs]” and that “some CTMs have a strong record of research on focused interventions that are features of their comprehensive model” (poster presentation). Others (e.g., Arick et al. 2005) have conducted preliminary outcomes studies which have identified positive results in a variety of key areas (e.g., social interaction, expressive speech, and use of language concepts). But more is needed. Simply stated, consumers of CTMs in autism desperately need to know which specific components of the models work in public school classrooms and/or homes, how and why they work, and how they can evaluate exactly how well the interventions are working with their students or children. It appears that neither ABA nor TEACCH by themselves have fully addressed these needs, or, at least, effectively communicated research-based answers to consumers. We believe a clear and universally accepted definition of what is a comprehensive evidence-based autism treatment model is needed in order to maximize the impact of this research on children with autism and their families.

Our expert validation results illuminate another potential focus area for future research. It must be noted that the expert raters in this study represent disparate training and certification experiences. Yet, the results of both groups indicated a surprising level of variability in recognizing what we identified as fundamental aspects of their approach as it related to autism treatment. This raises questions about the effectiveness and durability of autism training and certification, and more ominously, perhaps, a lack of complete understanding of the conceptual and theoretical underpinnings of the models that parents, teachers, and administrators adopt for their autism programming. More research in this area is warranted.

In conclusion, there is growing recognition in the field that serious problems exist within the autism research and literature base. The International Society for Autism Research (INSAR 2008), for instance, has made a public call for autism researchers to be “fair and balanced” (e.g., to avoid selective referencing and to include both supportive and contradictory citations within the text of research publications). In addition, researchers and practitioners must return to the fundamental basics of any good intervention research or training agenda. In order to objectively identify evidence-based Comprehensive Treatment Models in autism, we must first define our terms.