Attention-deficit/Hyperactivity Disorder (ADHD) is a common and oftentimes debilitating life-long psychiatric disorder which historically has been diagnosed and studied primarily in mid-childhood. In order to receive a diagnosis of the disorder, the DSM-IV requires the presence of a minimum number of symptoms and stipulates that they must cause significant impairment in the individual’s functioning (American Psychiatric Association 2000). While the symptoms are clearly defined in the manual, there are no guidelines as to what constitutes impairment, making it difficult to assess uniformly. Yet the need for a thorough and accurate assessment of impairment when making diagnostic determinations is becoming increasingly apparent. For example Gordon et al. (2006) examined several measures of symptom severity and impairment from four distinct ADHD samples and found that correlations between symptoms and impairment were modest, at best, and that symptom severity rarely accounted for more than 25% of the variance in impairment. In addition they found that when imposing impairment criteria on a group who met the symptom criteria for ADHD, only 33% met the full diagnostic criteria for ADHD (i.e. symptoms and impairment).

It is also well known that the current ADHD rating scales, most of which focus solely on symptom presence, result in many false positives (Faraone et al. 2003; Loughran 2003). This is likely due to the fact that not all inattentive/hyperactive/impulsive children experience significant impairment in their functioning (Graetz et al. 2001; Healey and Rucklidge 2006). Thus, it is important for the field to systematically assess impairment, in addition to symptom counts, across multiple settings, to assist in the accurate diagnosis of ADHD.

Notably, the reason most clinicians and investigators highlight the importance of a psychiatric interview, in addition to the use of rating scales, is that the interview serves as one way for the clinician to determine the extent to which behaviors cause impairment across settings and are thus “clinically significant.” However, the use of good screening measures may assist in reducing the amount of time clinicians need to spend in assessing impairment, thus making the assessment process more efficient. Additionally, time constraints may often not make it feasible to attempt to interview teachers in addition to parents, and thus it is important for the field to have solid impairment measures that are not too cumbersome for teachers to complete, so that information regarding impairment within the school setting is easily available.

To address the lack of focus on impairment in most rating scales, Wolraich et al. (1998) developed the Vanderbilt AD/HD Diagnostic Teacher Rating Scale which is comprised of two sections, one focusing primarily on symptoms as defined in DSM-IV, and a separate performance section which assesses academic functioning, classroom behavior, and social relations. Similarly, Rowland et al. (2001) developed an attention and behavior screener to be completed by teachers, which examines symptom presence as well as impairment in the areas of reading, mathematics, written expression, relationships with peers, following directions, assignment completion, and organizational skills. Finally, Fabiano et al. (2006) recently reported on an Impairment Rating Scale (IRS), which was designed for use with school-age children, and has both a parent and teacher version. This measure provides ratings across many domains (e.g., relationships, academic progress and self-esteem) as well as parents’ and teachers’ written opinion regarding the individual’s need for services and treatment. Thus measures of both behavioral symptomatology and impairment related to ADHD are available for school aged children, yet there are no analogous measures that can be completed by both parents and teachers to assess impairment in preschool children who might have ADHD.

Given that many preschoolers display inattentive and hyperactive behaviors similar to those associated with ADHD; which could be considered normative, or at least not uncommon, for this age group (Campbell 2002; McClellan and Speltz 2003; Sonuga-Barke et al. 2005), this population is particularly likely to yield many false positives when assessed using symptom ratings alone. Given that preschoolers are not yet engaged in a structured school setting where they receive grades, report cards, and are required to complete routine tasks, the available impairment measures are not suitable for use with this population. Therefore, the initial aim of this study was to develop and evaluate a measure of impairment specifically designed for use with preschool children. This scale was then used to assist in the primary focus of this study, which was to examine the clinical utility of adding an impairment rating scale, in addition to symptom ratings, in diagnosing ADHD in preschoolers.

Study 1

Scale Development

To assess impairment in preschoolers, we developed parent and teacher versions of the Children’s Problems Checklist (CPC). These scales were designed as brief screening tools, for use with very young children, that can be completed in just a few minutes. In contrast to impairment scales that are typically used with older children, academic functioning was not assessed. Rather, items focused on behavioral disruption, peer and adult relationships, and self-esteem; as well as frequent accidents and bedtime difficulties, which are more characteristic of younger children.

Method

Participants

Participants were a community-based sample of preschool children. Recruitment involved contacting a wide range of local preschools, where the majority of the children attended New York City sponsored universal pre-kindergarten programs. Those schools that agreed to participate were asked to distribute the Attention-Deficit/Hyperactivity Disorder—Rating Scale, Fourth Edition (ADHD-RS IV; DuPaul et al. 1998) and Children’s Problems Checklist (CPC; described below), along with consent forms, to the families of all students in the school who were between 36 and 60 months of age. After parents returned the consent and screening forms, teachers were asked to complete analogous screening forms. A total of 394 age-eligible children (211 males and 183 females) served as the community sample for this study. Among these, parent and teacher ratings were available for 393 and 385 children, respectively. Among the parent raters, the vast majority were mothers (92%), with additional ratings filled in by fathers (6%) or guardians (2%). We were unable to accurately track the number of rating scales distributed by schools and thus could not establish a definitive response rate. Therefore, to assess potential sampling bias, we compared our group means on the ADHD-RS to the large (N > 900) and representative (based on 2000 US census) preschool norms of McGoey et al. (2007). Our group means for ADHD-RS teacher ratings of Inattention, Hyperactivity/Impulsivity, and Total ADHD symptom scores were all less than one tenth of a standard deviation (SD) from their normative data. For parent ratings the scores were even closer, with all being less than three hundreds of a SD from their normative means. Thus, our sample appears to be representative of preschool children in the population. The mean age of the full sample was 4.17 years (SD = 0.53; range = 3.02–4.99). There was no significant difference in age between boys and girls (t = 0.51, p = 0.61).

From within this community sample, a subgroup was invited, based on the number of symptoms of ADHD that they presented with, to participate in a longitudinal study investigating hyperactivity/inattention in preschool children. Six-month follow-up ratings collected from this subgroup were used to assess stability of the CPC. Children who were judged to have at least six symptoms of inattention, or six symptoms of hyperactivity/impulsivity, across raters on the screening forms were invited to participate in the hyperactive/inattentive group (e.g., a parent endorses four symptoms of hyperactivity/impulsivity and the teacher three symptoms of hyperactivity/impulsivity, with two of them being different from those endorsed by the parent). A symptom was deemed to be present on the ADHD-RS-IV if it was rated as a 2 (often) or 3 (very often). The liberal criteria used to accept children into the inattentive/hyperactive group were designed to cast a wide net that would include many children who would not necessarily meet formal criteria for ADHD. One hundred and four children within the community sample met these criteria and 68 chose to participate. Those within the community sample who were rated as having fewer than three inattentive and hyperactive/impulsive symptoms by both the parent and teacher on the ADHD-RS-IV (DuPaul et. al. 1998) were invited to participate as non-hyperactive/inattentive controls in the longitudinal study. Among the 206 children who met these criteria, 99 were invited to participate based upon matching criteria to the hyperactive/inattentive group (i.e. gender), and 76 of these actually participated in the longitudinal study.

Data regarding race, ethnicity, and socio-economic status (SES) were not collected as part of the larger community screening. However, among those participating in the laboratory assessment, 40.2% were White, Non-Hispanic; 18.2% were White, Hispanic; 10.7% were Black, Non-Hispanic; 1.4% were Black, Hispanic; 10.7% were Asian; and 18.7% reported mixed or “other” ethnicity/race. Socio-economic status (SES) was measured using the Nakao–Treas Socioeconomic Prestige Index (Nakao and Treas 1994), where high scores are indicative of higher socioeconomic status (SES). The SES of this sample was variable (range: 20–89), but most of the children were living in homes with moderate socioeconomic status (mean SEI = 56.68, SD = 14.94). Those who participated in the follow-up assessment did not differ in age from those who did not (4.28 versus 4.18 years, p > 0.05).

Measures

Attention-Deficit/Hyperactivity Disorder—Rating Scale, Fourth Edition (ADHD-RS-IV)

The ADHD-RS-IV (DuPaul et al. 1998) is a rating scale based on the 18 DSM-specific ADHD symptoms for which a score on a four-point scale is assigned by the rater (i.e., “Never or rarely” = 0, “Sometimes” = 1, “Often” = 2, “Very often” = 3), that can be completed by parents and teachers. The psychometric properties of this scale have long been well-established for children above the age of 5 years (DuPaul et al. 1998). More recent data (McGoey et al. 2007) similarly indicate that the scales are highly reliable and valid when used with preschool children.

Children’s Problems Checklist (CPC)

The CPC was developed to assess impairment caused by ADHD symptoms in preschoolers. As such, we decided to avoid areas of academic function which are likely less stable in this age-group and may be affected by a variety of non-ADHD-related developmental factors. The following general behaviors were identified as indicators of impairment in preschoolers with ADHD: disruptiveness, difficulty with peer relationships, difficulties with relationships with adults, low self-esteem, trouble settling down to sleep, and having accidents. Separate but similar items for parents and teachers were developed (see Table 1 for exact items). For both forms, the rater was asked whether or not each domain was a problem for the child, and if so, to rate the severity of the problem on a three-point scale (i.e., mild, moderate, severe). Scoring for the scales was completed separately for parent and teacher reports. When the rater reported that a given behavior was not a problem, the item was scored as a zero. When a problem was indicated, it was assigned a numerical value (i.e., mild = 1, moderate = 2, severe = 3). The item values were then summed separately for parent and teacher scales to provide an overall indicator of impairment by setting.

Table 1 Children’s problems checklist items
Full size table

Data Analysis

Descriptive statistics for the CPC were computed separately for parent and teacher scores. Reliability for the CPC (both versions) was assessed using coefficient alpha (α) as a measure of internal consistency for the sample as a whole, as well as for boys and girls separately. Stability of the scales was assessed using Pearson correlations to examine the association between CPC ratings obtained at baseline and 6 months later. Finally, Pearson correlations were used to assess the inter-relations among parent and teacher CPC and ADHD-RS-IV ratings.

Results

Initially children were grouped by age in 6-month increments. No significant differences in CPC ratings by age were noted from parent (F [3, 377] = 0.70, p = 0.98) or teacher (F [3, 363] = 0.49, p = 0.69) report; thus scores for children between 36 and 60 months were collapsed. When children were grouped by sex, there were significant differences in CPC ratings by teacher (t [376] = 4.60, p < 0.001), but not parent (t [392] = 1.25, p > 0.10) report, with boys exhibiting higher levels of impairment at school than girls. Community sample means, standard deviations, and ranges for parent and teacher ratings on the CPC, along with raw score equivalent percentiles, are presented in Table 2. Percentiles rather than standard scores (e.g., t scores) were generated because they are more appropriate to use when scores are not normally distributed, as would be expected in a non-referred sample.

Table 2 Raw score to percentile conversions for children’s problems checklist
Full size table

Coefficient α for the parent and teacher versions of the CPC were 0.70 (boys = 0.72; girls = 0.65) and 0.84 (boys = 0.83; girls = 0.85), respectively.

Stability coefficients for the CPC from baseline to 6-month follow-up for parent (n = 97) and teacher (n = 96) were 0.687 and 0.701, respectively (both p < 0.001). Notably, in contrast to the parent ratings, which were completed by the same individual at both time points, many children had a different teacher by the time of the 6-month follow-up. When teacher follow-up data were divided into those with (n = 52) and without (n = 40) the same teacher (teacher name was omitted from four ratings) 6-month stability coefficients were 0.704 and 0.693, respectively (both p < 0.001).

As indicated in Table 3, parent and teacher ratings on the CPC were robustly correlated with their respective ratings of ADHD symptom severity. Although not shown in the table, CPC correlations with inattentive and hyperactive/impulsive symptom subdomains were not appreciably different. In contrast to the within-rater correlations, cross-rater correlations were more modest in magnitude, albeit statistically significant, highlighting the importance of the multi-informant approach.

Table 3 Correlations among impairment scales and ratings of ADHD symptoms*
Full size table

Discussion

Overall, the CPC appears to be a simple, yet psychometrically sound measure of impairment. In our community sample, we found it to have good temporal stability and concurrent validity. In addition, the teacher version had good internal consistency across gender; whereas for the parent version, internal consistency was acceptable for boys but somewhat weaker for girls.

Study 2

Study 2 was designed to examine the utility of employing the CPC measure of impairment, in addition to assessing symptom severity using the ADHD RS-IV, in order to more accurately identify preschoolers who may meet criteria for a diagnosis of ADHD. To meet this aim, in addition to examining the hyperactive/inattentive children within our community sample (described in study 1), we also studied a clinically referred sample of children reported to display elevated levels of ADHD symptoms. This allowed for a larger sample size, a greater range of symptom severity, and the inclusion of children that more closely resemble those seen by clinicians in the field.

Method

Participants

Participants for this study were 116 (77 males and 39 females) preschoolers who were rated as displaying at least six symptoms of either hyperactivity/impulsivity or inattention on the ADHD-RS-IV by one rater (i.e., parent or teacher) and at least two symptoms within the same domain, by the other. Among these, 65 were drawn from the community sample, based on parent and teacher ratings received through the screening procedure described in study 1, and 51 were clinically referred by schools, physicians and mental health professionals in New York City (e.g., pediatricians, neurologists, school psychologists). For this latter group, symptom screening was conducted following clinical referral. The children had a mean age of 4.31 years (SD = 0.51; range = 3.00–5.00), with no significant difference in age between boys and girls (t [113] = −0.92, p = 0.93; for one child, date of birth was missing and therefore precise age could not be calculated). Within this sample 40.6% of the children were White, Non-Hispanic; 19.6% were White, Hispanic; 13.4% were Black, Non-Hispanic; 2.2% were Black, Hispanic; 5.8% were Asian; and 18.8% reported mixed or “other” ethnicity/race. The SES of this sample ranged from 20 to 89, but most of the children were living in homes with moderate socioeconomic status (mean SEI = 53.12, SD = 15.52).

Measures

The ADHD-RS-IV and CPC (described in Study 1).

Data Analysis

Frequencies were calculated to determine the number of hyperactive/inattentive children who met various impairment threshold levels. Because “some impairment” is not operationally defined in the DSM-IV, we explored the impact of various threshold levels on diagnostic rates by examining 75th and 90th percentile cut-off scores in varying combinations. At the most liberal end, we examined a cutoff using impairment levels at the 75th percentile or above by both parent and teacher ratings (75 of 75), and at the most conservative end we required impairment levels at or above the 90th percentile by both informants (90 of 90). Because severity of impairment is oftentimes not consistent across settings, a middle level of impairment, requiring a 90th percentile cut in one setting and a 75th percentile cut in the other, was also examined (75 of 90). Due to the significant gender differences on CPC ratings reported in study 1, gender specific cutoffs were used. Notably, our more liberal cut score at the 75th percentile closely parallels, within our community sample, the percentile ranking associated with the presence of two symptoms of ADHD, which is what we required as a minimum for the secondary rater (must have at least six symptoms by the first rater).

Results

Coefficient α for the parent and teacher versions of the CPC in this group of symptomatic children were similar to those obtained in the larger community sample (0.70 and 0.80, respectively). Teacher ratings of impairment on the CPC were significantly higher for the clinically referred hyperactive/inattentive children (M = 8.59, SD = 4.69) than for those drawn from the community sample (M = 5.80, SD = 4.46; t [114] = −3.27, p = 0.001); however the clinically referred group (M = 5.88; SD) = 4.17 did not differ from the community sample (M = 4.80; SD = 3.82) on parent CPC ratings (t [114] = −1.45, p = 0.149). When our most liberal cutoff (75 of 75) was used, only 63 (30 from the community sample and 33 clinically referred) of the 116 hyperactive/inattentive children (54.3%) met the threshold for ADHD (i.e. displaying the symptoms and impairment), indicating a 45.7% decrease in “cases.” When using our intermediary level (75 of 90) only 55 (24 community; 31 clinically referred) of 116 children (47.4%) with symptoms indicative of ADHD met the threshold, indicating a 52.6% decrease. Finally, when the most stringent criteria were used (90 of 90) only 27 (9 community; 18 clinically referred) of the 116 hyperactive/inattentive children (23.3%) met the threshold; a 76.7% reduction in cases. Thus, when imposing even modest impairment criteria, almost half of the preschool children who are described as highly inattentive and/or hyperactive/impulsive do not appear to meet DSM-IV criteria for a diagnosis of ADHD.

Discussion

Although the presence of impairment is of key importance for the diagnosis of ADHD (APA 2000), there are very few measures that have been specifically developed to assess impairment in relation to ADHD symptoms; and even fewer that are appropriate for use in the preschool age group. The results of this study indicate that the parent and teacher versions of the Children’s Problems Checklist (CPC) are internally consistent and temporally stable measures of impairment, that demonstrate good validity relative to ADHD symptomatology in preschoolers. Thus, they can be useful additions to the currently available clinical tools used in the assessment of ADHD. Given that the diagnostic criteria for ADHD require cross-situational behavioral symptoms, along with impairment; and that research has clearly highlighted the need to assess both aspects separately (Bird et al. 1988; Breton et al. 1999; Costello et al. 1988; Gordon et. al. 2006; Graetz et al. 2001; Romano et al. 2001; Simonoff et al. 1997), it is essential that clinicians assess both factors when evaluating a child for ADHD. This is especially important during the preschool years as many ADHD-like behaviors can be seen as normative in this age group. This is evidenced by the fact that 65 of the 394 children within our community sample (16.5%) met the symptom threshold for ADHD; while prevalence estimates of the disorder within school-aged children range from 3 to 7% (American Psychiatric Association 2000).

An important consideration that warrants further research is the development of specific impairment criteria that can be adhered to when making a diagnosis of ADHD. The DSM-IV indicates that some impairment must be present in two or more settings, however what exactly constitutes some is not stipulated. In order to explore the effect of differing impairment criteria on rates of ADHD diagnoses, we applied various combinations of cutoff scores for impairment ranging between the 75th and 90th percentiles across settings. Even using modest criteria of cross-situational impairment (i.e., >75th percentile in each setting), the number of children identified as meeting criteria for ADHD was reduced by almost half (46%). When the most stringent criteria were applied (i.e. cross-situational ratings at the 90th percentile), the number identified was reduced by 77%. Thus, our findings have supported those of others in indicating that adhering to the requirement of impairment resulting from symptoms leads to a substantial reduction in the number of children that are diagnosable with the disorder. It is likely that at least the more conservative criteria are too strict, and may result in under diagnosis; however, the field needs to work toward reaching a consensus regarding the impairment threshold needed to receive a diagnosis of ADHD; along with a clear definition of what areas of functioning should be covered within the assessment of impairment.

It is possible that the large proportion of children in our study who met symptom criteria for ADHD, but not impairment criteria, reflects the unique characteristics of a preschool sample. In a longitudinal studies of preschool children, Campbell and collaborators (Campbell et al. 1986; Campbell 2002) found that nearly 50% of preschool children identified by parent ratings as being “hard to manage” no longer presented with clinically significant behavioral difficulties by the time they were 6 years-old. Although speculative, perhaps impairment levels rather than symptom severity, will allow for a greater ability to hone in on those preschool children at greatest risk for persisting difficulties. The ability to do this may be particularly crucial during the preschool period as considerable concern has been expressed regarding the large number of preschoolers who are currently being treated with medication (Zito et al. 2000).

As with all research, this study is not without limitations. First, although the CPC might be suitable for use with older children, no such data were collected. Therefore, the use of this scale and the impact of imposing cross-situational impairment criteria in older children who might have ADHD cannot be determined from these data. Second, given the brevity of this measure, it is not an exhaustive list of all possible areas in which impairment may be present. Nevertheless, it might serve as a useful screening instrument that can be completed along with the more commonly used symptom checklists. Third, the sample used in this study was recruited from a highly diverse, urban community, and thus the results may not fully generalize to other settings. Finally, because we do not have data regarding response rates to our scales that were distributed for the community recruitment, there is some uncertainty as to how representative that sample is. Nevertheless, at least in terms of severity of ADHD symptoms, it seems quite similar to other larger and more carefully recruited normative samples (McGoey et al. 2007).

Overall, these data clearly indicate that many, and perhaps most, hyperactive/inattentive preschool children do not experience cross-situational impairment. Therefore, caution should be used when diagnosing ADHD in this age group. Severity of inattentive and hyperactive behaviors during the preschool years does not appear to have substantial prognostic utility as to which preschool children continue to experience difficulties during the school-age years (Campbell et al. 1986; Campbell 2002). Perhaps measures of impairment will have greater prognostic utility. Finally, future studies should focus on developing clear guidelines for the assessment of impairment across various areas of functioning; determining whether detailed information about impairment can be used to effectively guide treatment; and whether targeting specific areas of impairment can improve functioning either in that specific area or even generalize to other areas.