Introduction

The Individualized Education Program (IEP) is a multidisciplinary, team-developed plan required for every child receiving special education services under Part B of the Individuals with Disabilities Education Act (IDEA 2004). The IEP is the educational map for children with disabilities. As described in Public Law 108–144, legally required components of the IEP include (a) the child’s present level of performance; (b) measurable annual goals; (c) how the child’s progress toward meeting the annual goals will be measured and when periodic reports on the progress the child is making toward meeting the annual goals will be provided; (d) the special education (i.e., specially designed instruction) and related services and supplementary aids and services, based on peer-reviewed research to the extent practicable, to be provided to the child; (e) program modifications or supports for school personnel that will be provided for the child; (f) the extent to which the child will not participate with nondisabled children in the regular classroom; and (g) individual appropriate accommodations that are necessary to measure the academic achievement and functional performance of the child on state and districtwide assessments (PL 108-446 2004). As shown above, in the recently amended version of IDEA (2004), emphasis has been placed on accurate and objective measurement of student progress. Although IDEA delineates broad, minimum required components of the IEP, the law allows states to determine any additional requirements, specific procedures, and format for the IEP. Thus, IEPs may take on many different forms and styles.

The special education literature provides additional information that is helpful in implementing the requirements of IDEA for developing IEPs and complement the IDEA requirements. For example, IEPs should address students’ individual strengths and needs, and include a timeline (Jung et al. 2008) for meeting objectives that are specific (Jung 2007), measurable (Jung 2007), observable, and easily connected to classroom activities (Smith and Slattery 1993). Unfortunately, research indicates that many IEPs may be lacking in their consistency with recommended practice. In particular, IEPs have tended to include inadequate descriptions of present performance (Gartin and Murdick 2005), goals that were neither specific nor measurable (Michnowicz et al. 1995), and expectations that were unrealistic and misaligned with children’s abilities (Johns et al. 2002). Further, many contained placement recommendations that appeared to be based upon eligibility criteria rather than performance (Johns et al. 2002).

The issues that arise from these shortcomings in IEP development have been the source of numerous legal battles for the specific disability category of autism (Etscheidt 2003). For the population of children with autism, disputes arising from IEPs have increased in recent years and are characterized as the fastest growing and most expensive area of educational litigation (Etscheidt 2003). Contributing to the growth in IEP litigation for students with autism is the startling increase in identified students. Schools served almost 260,000 children with autism in 2006 (IDEA 2006), which although small relative to the total number of US students, represents a more than 500% increase within this eligibility category in the last decade (GAO 2005; Safran 2008). Common concerns from parents about the IEP process include difficulties in (a) being viewed as equals in making educational decisions regarding their children; (b) IEP objectives’ being properly followed in the classroom; (c) being fully informed about special education law and their rights, and (d) classroom practices, such as ineffective discipline programs and inappropriate placement decisions (Fisher and Meyer 2002).

Despite the drastic increase in numbers of students who have autism, and the growing base of IEP litigation for this population, reliable information on the content, effectiveness, and outcomes of IEPs for students with autism is missing from the literature (Wilczynski et al. 2007). An examination of the IEPs of children with autism in the context of IDEA requirements and nationally-recommended practice is needed to begin to describe the particular strengths and weaknesses of IEPs for this population. Furthermore, understanding possible factors that are associated with the quality and content of IEPs may help to identify teacher and student variables important to target in preservice training, professional development, and school district and state monitoring and accountability.

Accordingly, the aims of the current study were to (a) develop an IEP evaluation tool, (b) determine whether the tool could be administered with adequate reliability, (c) test the tool on a pilot sample of IEPs written for young children diagnosed with autism, and (d) examine potential associations between IEP quality and various teacher and child characteristics important for future research. Because there is little to no information on the relations between IEP quality and teacher and school factors, no directional hypotheses were proposed.

Methods

Participants

Thirty-five teachers of students with autism between the ages of 3 and 9 years were recruited from one midwestern and one southern state as part of a larger randomized controlled study on teacher consultation and coaching for young children with autism (Ruble et al. 2008). Initially, school systems were approached at the district level, and special education directors were asked to participate. A total of 79 teacher/child pairs were screened; 4 did not meet eligibility requirements, 15 declined to participate, and 21 refused for other reasons. Although we did not systematically collect data on refusers, there was no evidence of sampling bias at either the district or school level in terms of numbers or proportion participating. Once permission was obtained, a list of names of all teachers of students with autism was provided and teachers were contacted directly by the research team. After agreeing to participate, the researchers asked teachers to submit the initials of all the students with autism on her caseload or in her classroom. To qualify for the study, students had to be receiving special services according to the IDEA category of autism and meet the Diagnostic and Statistical Manual definition of Autistic Disorder (DSM-IV-TR; APA 2004) which was confirmed by an evaluation conducted by the research team. From the list of initials provided by the teacher, the research team randomly selected one student. The teacher requested permission from the parent/guardian’s of the student to be contacted directly by the researchers.

Two of the school systems were located in large cities and 14 were in small cities or in rural areas. A total of 15 teachers/students from urban schools (in cities with populations greater than 300,000 people) participated. The remaining 20 teachers/student participants were from rural or small city schools. Both teachers and parents or caregivers provided informed consent to participate. The mean age of the students was 6.1 years (SD = 1.7); 83% of the students were male, 74% of the students were Caucasian, 23% were African American, and 3% were bi-racial (see Table 1). Twenty-eight percent of household incomes of the students was less than $24,999; 36% fell between $25,000 and $49,999; and 36% was above $50,000 (income was not provided by eight caregivers).

Table 1 Description of teacher, student, and IEP variables
Full size table

Ninety-four percent of the teachers were female (N = 33). The teachers’ average class/caseload size was 12.8 students (SD = 7.9). Ninety-four percent (N = 33) of the teachers reported that they had formal autism training such as coursework, supervised field work, workshops, and in-services. Thirty-four percent of the teachers reported that in addition to teaching, they also possessed assessment skills for students with autism. Another 25% of teachers reported that they had also served as a consultant or trainer to other teachers.

Measures

For this study, a measure of IEP Quality for Students with Autism (see Appendix) was constructed using both (a) IDEA requirements for IEPs (e.g., measurability) and (b) National Research Council (2001) recommendations for educating children with autism.

IDEA-Related Indicators

There were eight indicators included on the tool which were based on IDEA requirements for IEPs of all students with disabilities, not only those with autism. The first item, scored dichotomously (0 = no; 2 = yes), measured the overall quality of the description of the student’s present levels of performance and asked if the child’s performance was described relative to the general or developmental curriculum clearly enough to establish well-written objectives. The remaining seven items were treated as a separate set of items and were based on the quality of the written descriptions of IEP objectives: (a) the student’s present level of performance for the specific objective; (b) a description of the association between the IEP objective and the general and/or developmental curriculum; (c) the measurable and behavioral description of the objective; (d) the conditions under which the behavior is to occur; (e) the inclusion of specific criteria and a timeline for goal attainment for each objective (i.e., not just the implied timeline from the IEP as a whole); (f) a method of goal measurement; and (g) the description of specially designed instruction (SDI) that is individualized for the goal/objective. This last indicator comes from IDEA’s (2004) requirement that a statement of the child’s special education be provided in the IEP; IDEA (2004) defines special education as “specially designed instruction” (IDEA 2004, p. 11). In all of the IEPs reviewed, which came from one midwestern and one southern state, a statement of SDI was provided and the corresponding response format allowed either open ended or forced choice answers. Because this information appears to be indicated by IDEA and was subsequently provided in the IEPs, we included it as part of the overall quality analysis of the IDEA indicators, recognizing that some states may have chosen a different format and interpretation of this requirement. Each of the items were rated using a 3-point Likert scale (0 = no/not at all; 1 = somewhat; 2 = yes/clearly evident) as applied to a subsample of specific IEP objectives.

Objectives chosen for scoring were selected to represent each of the three primary domains of need specific to autism: (a) a social objective; (b) a communication objective; and (c) a learning or work skill objective (NRC 2001). However, if an objective addressing one of the three domains could not be identified within the IEP, then a behavioral objective, academic objective or other objective was substituted for coding. A total of 105 IEP objectives (35 students × 3 objectives) were selected and coded. To verify the accuracy of the classification of IEP objectives into learning domains a two-step process was applied. First, all of the IEP objectives were written on 3 × 5 cards and categorized into one of the domains (social, communication, learning/work skills, academic, motor/sensory, behavioral) by two independent raters. Second, the raters compared their results and reconciled differences. Raters agreed on the categorization of over 90% of the objectives after step one. Overall, 22 social objectives, 33 communication objectives, 29 learning/work skill objectives, 12 academic objectives, five motor objectives, and five behavioral objectives were evaluated. The mean score across the three objectives was used as the score for the IDEA quality indicator. The scores for items 2 though 8 were then summed and a mean score was calculated. For items to be considered optimal in meeting the recommendations, the item had to be scored a 2.

The total score for the IDEA related items was based on the mean of two subscores: (1) the score for item one, which measures the present levels of performance and (2) the overall mean score of items 2–8, which measures the quality of the written objectives. Thus, item one contributed half of the weighting to the total scoring for the IDEA items, and items 2 through 8 contributed the remainder. Item one was weighed heavily because it set the stage for whether objectives could be written that were related to present levels of performance. Total scores ranged from zero to two.

NRC Quality Indicators

There were nine NRC quality indicators included on the tool plus two additional items—one that measured extended school year and another that measured the degree to which specific objectives were individualized from state academic content standards. The areas selected for analysis were based on the best practices outlined by the NRC (2001) and included assessment of the presence of descriptions of: (a) parental concerns, (b) social skills to improve involvement in daily activities, (c) expressive, receptive and non-verbal communication skills, (d) a symbolic functional communication system, if appropriate, (e) engagement in tasks or play that are developmentally appropriate, including an appropriate motivational system, (f) fine and gross motor skills for engaging in age appropriate activities, (g) basic cognitive and academic thinking skills, (h) replacement of problem behaviors with appropriate behaviors, (i) organizational skills and other behaviors needed for success in a general education classroom, and (j) full-year programming. Adaptive behavior skills are assumed to be embedded within the domains of learning described above.

A 3-point Likert scale (0 = no/not at all; 1 = somewhat; 2 = yes/clearly evident) was used to rate the degree to which the IEP was consistent with the nine NRC indicators. To be considered adequate in meeting the recommendations, the item had to be scored a 2. Although the NRC does not specifically recommend extended school year (ESY) for every child, it does recommend consideration of full year programming relative to the needs of the child. Therefore, ESY was included as an item: a score of zero was coded if ESY was not addressed, one if ESY was addressed, and two if ESY was not only addressed, but also recommended as a service. NRC guidelines recognize that in some cases a child may not need ESY. However, because this is considered to be unusual and to simplify scoring, the current scoring approach was used, even though it has the potential to penalize IEPs that discuss ESY but do not recommend it. A mean overall score was calculated across all 10 items. Scoring for the NRC indicators included all objectives on the IEP, and was not limited to the three objectives selected above for IDEA coding.

To establish inter-rater reliability of the IEP Evaluation Tool, sample IEPs not used in the study were coded separately by two evaluators. Differences in coding were discussed and revisions made to the IEP Evaluation Tool until an interrater exact agreement rate of .80 or higher was established for the entire IEP Evaluation Tool (not for each item). Three separate 2-h training sessions were required to develop the final revisions for the coding scheme for the tool. To verify the reliability of the coding scheme when applied to the study sample, twenty percent of the IEPs were randomly selected and coded separately by two raters. Inter-rater agreement was calculated using intraclass correlation.

Teacher Variables

Variables collected included number of years teaching students with autism and total number of students taught with autism. Teacher’s perceived knowledge of autism was assessed using a 15-item self-report measure developed for the study as part of the teacher training conducted by the first author (e.g., how knowledgeable are you concerning the characteristics of autism and their implications for intervention?). Internal consistency (alpha) was 0.91.

School Variable

Geographic location was classified as: (a) rural/small city (less than 300,000 people) or (b) large city (more than 300,000 people).

Student Variables

Variables examined included degree of autism severity, cognitive impairment, adaptive behavior level, externalizing behavior, and diagnosis. Autism severity was measured using the Childhood Autism Rating Scale (CARS; Schopler et al. 1980), a 15-item observational scale. Example items include social relatedness, resistance to change, communication, and body use. Items are scored using a 4-point scale (1 = no evidence of abnormality, 4 = severely abnormal). In the normative sample, test–retest reliability for the total score was .88 and the correlation between the CARS and clinical ratings of autism was .84.

Cognitive impairment was measured using the General Conceptual Ability (GCA) subscale score from the Differential Abilities Scale (DAS; Elliot 1990). The internal consistency and test–retest reliability of the GCA exceeds .89 for all age levels.

Adaptive behavior was measured with the Classroom Edition of the Vineland Adaptive Behavior Scales (VABS; Sparrow et al. 1995). The VABS is a standardized tool that allows for the measurement of adaptive behavior in the classroom. Four domains were evaluated: socialization, communication, daily living skills, and motor skills. An overall score was used, the Adaptive Behavior Composite. The coefficient alpha for the Adaptive Behavior Composite is .98.

The diagnosis/educational classification of autism was confirmed for each child by a clinical assessment of symptoms of autism using the Autism Diagnostic Observation Schedule—Generic (ADOS-G; Lord et al. 2003) and the Autism Diagnostic Interview—Revised (ADI-R; Rutter et al. 1994). The ADOS-G is a semi-structured student interaction assessment tool while the ADI-R is a semi-structured caregiver interview. Each instrument has its own scoring algorithm for diagnosis based on DSM-IV criteria and provides cut-off scores in the domains of social reciprocity, language and communication, and restricted and repetitive behaviors. Both tools have strong inter-observer agreement, internal consistency, and test–retest reliability (Lord et al. 2003; Rutter et al. 1994). All children met the cut-off scores for autism on both measures.

Analysis

Descriptive statistics were generated for each of the IDEA and NRC indicators and for the overall score on the IEP Evaluation Tool. Spearman rank order correlation was used to analyze the relations between teacher and child variables and IEP quality. To help control for Type I error rate inflation due to the number of correlations calculated, the p level was set at .01. Independent t-tests were conducted to determine whether there was a difference in IEP quality mean scores based on child characteristics (i.e., race; autism severity), teacher characteristics (i.e., number of year teaching; number of students with autism taught), and school characteristics (i.e., rural vs. uban).

Results

Reliability of the IEP Evaluation Tool

The interrater reliability of the IEP evaluation tool was calculated. An adequate interrater reliability of the total score (i.e., the sum of the IDEA and NRC items) was achieved as indicated by an intra class correlation of .70.

Analysis of IEP Quality

IDEA Related Indicators

As shown in Table 2 an area of relative strength was that about two-thirds of objectives were able to be linked to a description in the present levels of performance for the specific IEP objectives. The majority (83%) of the objectives, either did not reflect or only partially reflected adaptation of objectives from state academic expectations, e.g., “Shall develop abilities to become responsible members of a family, work group, or community.” Of the objectives analyzed, 12% were judged as adequate in describing how the objective will assist the child in involvement in and progress in the general curriculum and 22% as adequate in describing involvement in a developmental curriculum. Relatively few objectives were described adequately in behavioral terms (41%) or provided conditions under with the behavior should occur (39%). Other areas that were significantly weak were adequate descriptions of goal measurement (2%), descriptions of specially designed instruction (3%), or clearly stated criteria and timelines specifically written for the objective (0%). Most IEP forms did not provide for individualized stated modifications or adaptations or a specified timeline other than the annual IEP date.

Table 2 IDEA requirements for IEPs
Full size table

NRC Indicators

Table 3 lists the percentage of IEPs that included each of the NRC recommended best practice indicators. Descriptive analysis of best practice components of the IEPs indicated that of the 35 IEPs in the current sample, only 51% reported a review of extended school year services (ESY) services and even fewer, 9% (three students), explicitly recommended ESY. Recommended ancillary services included speech therapy for 34 students, occupational therapy for 31 students and physical therapy for four students. None of the IEPs met all 10 NRC indicators listed in Table 3. Across the 35 IEPs, the mean number of quality indicators met was 4.7 (SD = 1.6) out of 10 possible. A breakdown of the number of indicators met indicated that no IEPs met nine or more indicators, two IEPs met eight indicators; two more met seven, four met six, and the majority—16 –met five indicators. The remaining 31% met four or fewer of the 10 indicators. Twenty percent of the IEPs had no social goals and 15% had no communication goals (see Table 3). The content most frequently lacking (40% of IEPs) were objectives that included engagement in tasks or play that were developmentally appropriate; almost none described a motivation system. Finally, parent concerns were included in about half of the IEPs.

Table 3 Percent number of IEPs reflecting national research council (NRC) recommendations
Full size table

Associations Between IEP Quality and School, Teacher, and Child Factors

Correlations between teacher variables and IEP quality were not significant (see Table 4). Although it did not meet the adjusted p value < 0.01, a correlation of −0.39 (p = .02) was noted between number of students with autism taught and IEP quality. Similarly, child characteristics, such as level of functioning or family income, were not associated with IEP quality. Independent t-tests were conducted to determine whether there was a difference in total IEP quality mean scores based on child race and school location (urban—schools located in towns with more than 300,000 people vs. rural). The t-test was insignificant for IEP quality for both child race (t (33) = .−51, n.s.) and school location (t (33) = .17, n.s.).

Table 4 Spearman rank correlation of teacher and child variables and IEP quality
Full size table

Discussion

IEP Evaluation Tool

Preliminary evidence indicates that the IEP Evaluation Tool is a reliable tool to measure the quality of IEPs. Moreover, the content validity of the measure was implicitly established by generating items directly from recommendations derived from IDEA legislation (2004) and the NRC (2001). However, further study of the psychometric properties of the IEP evaluation tool is necessary, including sensitivity to detecting change, replication with a larger sample, and replication with IEPs from states that have varied IEP forms and procedures. Once fully validated, the tool may be particularly helpful in further research exploring the utility of IEPs for program development, monitoring, and accountability.

IEP Quality

Several noteworthy findings on the quality of IEPs of children with autism were identified. For example, the lack of association between most of the teacher (number of years of teaching students with autism), school (urban vs rural), and child factors (e.g., autism severity, level of cognitive functioning or adaptive behavior, race, and family income) and IEP quality has potentially positive connotations, given the importance of equal access and equal rights to services. However, any “positive” interpretations are clearly tempered by the fact that IEP quality was generally quite poor. That is, stated another way, the “good” news is that although IEP quality was poor, it was equally poor across all assessed school and child characteristics. At best this is a mixed blessing.

One further observation is worth noting. There was a nearly significant negative association between the number of students with autism taught (including years spent assessing, consulting, and training others on autism) and IEP quality. This finding is counterintuitive. That is, not only was there no apparent improvement in IEP quality with more experience teaching children with autism, there was evidence for poorer quality IEPs with more experience. Although the result was insignificant when using a more stringent p-level to control for experimentwise error, further investigation is clearly warranted.

Probably the most important finding of the study was that the IEPs of students with autism do not meet the requirements/recommendations of IDEA and NRC indicators and are sorely in need of improvement. Measurability of IEP objectives appeared to be one of the greatest areas of need. In particular, specified criteria for goal measurement and success were lacking. Further, most objectives either did not accurately reflect state standards or, when they did, tended to be adopted without individualization to the child. Although state academic content standards should be considered in developing plans that allow access to the general curriculum, such standards alone are likely insufficient to become an individual student’s objective without modification. Instead, for standards requiring modification or accommodation, teams should delineate measurable objectives that lead to the standards, so it is possible to monitor the student’s progress towards the broader state academic standards.

A second area of weakness was the inadequate description of specially-designed instruction for a specific objective. It is stated in Public Law 108–144 that a statement of the special education and related services and supplementary aids and services be provided in the IEP. Thus, the IEP provides the mechanism by which teaching methods and specially designed instruction are articulated across teachers, teaching assistants, other school personnel, and parents. The IEPs studied were from one midwestern and one southern state, and IEPs from both states asked for descriptions of specially designed instruction on the IEP form. However, many IEPs provided a list of teaching methods that were not adequately linked or individualized to IEP objectives. Adequate descriptions of specially designed instructional methods tailored to each individual’s specific teaching objectives, especially during times of transition from one school year to another, and for students who will be assessed with alternative standards are essential for program implementation fidelity and reliability across teachers.

Similarly, a related concern has been identified by others, the lack of clearly identified accommodations, which promotes the likelihood of inconsistency in program implementation (Ketterlin-Geller et al. 2007). Undoubtedly, objectives need to include an unambiguous method of measurement and overtly stated criteria for successful goal acquisition. Data collection on student progress is necessary for evaluating the effectiveness of programs and for making adjustments where necessary. In order to make these educational decisions, teachers need support on how to collect, interpret, and link data to educational practices, and how to alter educational strategies based on child progress (Yell et al. 2005).

Another notable finding was the lack of adherence to best practice recommendations from the NRC (2001). Only 9% of IEPs recommended that students receive ESY services and only half of the IEPs addressed ESY at all. In contrast, the NRC (2001) recommends that the great majority of all young children with autism receive year round schooling due to the nature of their disability- that is “active engagement in intensive instructional programming for a minimum of the equivalent of a full school day, 5 days (at least 25 h) a week, with full year programming varied according to the child’s chronological age and developmental level.” (NRC 2001, p. 219). This finding is likely due to several factors including a lack of funding, inadequate training, a failure to use appropriate decision making tools to qualify students for ESY, and a gap between best practices and policy. Another area of concern was that only about half of the IEPs included a description of parental concerns. This is a discouraging finding, but one that is consistent with research on parental perceptions of the IEP meeting (Fisher and Meyer 2002), ratings of inclusiveness in the IEP process (Fish 2006), and with qualitative observations on parent satisfaction with the educational decisions made on behalf of their children (Fish 2006). Parents’ active participation in the development of the IEP is critical if the educational program is to aid in the student’s development and generalization of skills across home, school, and community setting. Clearly, IEP teams are in need of strategies to solicit and include parental concerns in the IEP such as appointment of an IEP advocate or the use of an IEP integrity checklist to promote parent inclusiveness.

Finally, the objectives included in many of the IEPs did not sufficiently address the needs of those with autism. For example, although the majority of IEPs had social and communication goals, a surprising number did not—a concern expressed by others (Koegel et al. 2009). Considering that a diagnosis of autism is based, in part, on deficits in social and communication areas, this is an area that warrants attention.

Need for Further Research

The findings from the study suggest the need for teacher training, accountability, and monitoring in several areas. Improved teacher training programs and IEP development activities might include: (a) increased measurability of IEP objectives, including clear descriptions of conditions under which behavior is to occur and specified criteria of goal attainment described; (b) increased assessment of needs for ESY; (c) better specified individualization of IEP objectives from state academic content standards; (d) inclusion of parental concerns; and (e) inclusion of social as well as communication objectives. Additionally, research based on direct observation and evaluation of the implementation of IEP objectives into practice, including an examination of how well teachers are able to develop instructional procedures conducive to attaining objectives set forth in the IEP has been suggested for many years (Rodger 1995), yet remains an area lacking in the literature.

Conclusion

Students with autism require specialized interventions that address comprehensive skill development, including social and communication skills, task engagement and work skills, as well as adaptive, cognitive, and motor skills (NRC 2001). The IEP is the one universal tool available to schools for recording and tracking goals and objectives that are individualized and responsive to students’ specific needs. Litigation regarding the educational programs and IEPs of students with autism is the fastest growing area of dispute (Etscheidt 2003). According to a report by the National Council on Disability (2005), it is the burden of the state to prove the appropriateness of the IEP when challenged by the parent. Schools are obligated to show evidence that IEP decisions are based on assessment results and designed to provide some educational benefit regardless of the actual progress made by the child (Etscheidt 2003). Meeting these obligations safeguards against litigation over services provided to children with disabilities.

The detailed guidelines developed by professional organizations, such as NRC (2001), requirements provided by IDEA (2004), and measures, such as the IEP Evaluation Tool, may be useful resources in the development and implementation of high quality educational programs. Training teachers to use the IEP as a functional tool in designing and implementing educational practices within the classroom will not only facilitate the education of the student, but also ensure that the proper procedures have been followed and that the student is receiving optimal educational services and improved educational outcomes.