Introduction

Structured clinical interviews are considered pivotal in the classification of psychiatric disorders. The use of semi-structured interviews enables clinicians not only to verify their hypothesis in terms of specific mental disorders they assume to be present in individuals, but also to falsify other psychiatric disorders that are not assumed to be present. Thus, good clinical interviews possess high sensitivity and specificity in the classification of psychiatric disorders. For adults, the Structured Clinical Interviews for DSM-IV axis I and axis II disorders are examples of clinical interviews that are widely used for classifying psychiatric disorders [1, 2]. In children and adolescents, a number of interviews are available including the Anxiety Disorders Schedule for Children and Parents (ADIS-C/P) [3], which assesses the most prevalent types of child psychopathology with specific coverage of anxiety disorders, the Diagnostic Interview Schedule for Children (DISC) [4] and the Diagnostic Interview for Children and Adolescents (DICA [5], which both assess a broad range of psychiatric disorders, and the Kiddie-Schedule for Affective Disorders and Schizophrenia for school-aged children (K-SADS) [6], which is designed to assess current as well as past episodes of psychopathology. The Kid-SCID [7] is based on DSM-IV [8] and has been specifically designed for children and adolescents. The Kid-SCID is, like the ADIS-C/P and the K-SADS, an investigator-based interview which involves a clinical judgment of the presence of psychiatric disorders based on information provided by the child/adolescent and their parent(s). The DISC and the DICA are respondent-based interviews which require minimal clinical judgment and have the risk to overdiagnose disorders [9]. The Kid-SCID overlaps largely with other interviews with respect to the psychiatric disorders that are included.

The Kid-SCID was modeled after the adult version of the SCID-I and adapted in a number of ways. First, modules of specific childhood disorders such as Attention Deficit Hyperactivity Disorder (ADHD), Oppositional Defiant Disorder (ODD), and Conduct Disorder (CD) were added. Second, separate interviews were executed for youngsters and their parents enabling clinicians to evaluate children’s psychiatric classifications from independent points-of-view. During the past decades, most researchers and clinicians based their diagnoses on DSM-IV criteria without evaluating whether the procedure they used is reliable. It is important to note that the Kid-SCID is based on the DSM-IV(-TR) criteria [8]. With the recent launch of DSM-5 [10], a number of changes have been proposed that range from the inclusion of new disorders (e.g., mood dysregulation disorder) to changes in the specific criteria that are used to define disorders. With the exception of PTSD, no major changes have been made to the criteria for the disorders that are included in the Kid-SCID and so evaluation of this interview as a mean to make reliable diagnoses is still relevant. Although new assessment tools will be available for DSM-5 diagnoses, the question remains whether the Kid-SCID can be used to reliably assess common psychiatric disorders in children and adolescents.

Only a handful of studies have investigated the reliability and validity of the Kid-SCID. Reliability is often assessed in terms of inter-rater reliability and expressed in terms of Cohen’s kappa [11]. In a small-scale study, Matzner et al. [12] assessed agreement between two successive clinical evaluations with the Kid-SCID in 15 clinically referred children. Results showed good agreement for ADHD, CD, and social phobia (kappa range: .84–1.0), and fair agreement for ODD, separation anxiety disorder, and posttraumatic stress disorder (PTSD) (kappa range: .44–.66). In another study [13], perfect agreement was found for ADHD (κ = 1.0) and good agreement for ODD (κ = .73). For CD, the inter-rater reliability could not be assessed due to the absence of variation in the interviewers’ diagnoses (i.e., 100 % agreement). Yet another study by van Vlierberghe et al. [14] found good to perfect agreement for the various Kid-SCID disorders (kappa range: .79–1.0) in a sample of overweight children and adolescents. Another aspect of reliability is internal consistency, which has been assessed for some of the Kid-SCID scales by using the criteria ratings for each symptom. Previous research has shown adequate to good internal consistency for ADHD (α = .72) and CD (α = .76) in 50 adolescents receiving residential treatment for substance abuse and severe behavioral problems [13]. Thus, there is some evidence to suggest that the Kid-SCID is reliable in terms of inter-rater reliability and internal consistency.

The agreement between child and parent ratings of psychopathology is in general quite modest. For example, the meta-analysis of Achenbach et al. [15] reported only a low correlation among child, parent, and teacher questionnaire ratings of psychopathological symptoms. Research has shown that agreement between parents and children was significantly lower for internalizing disorders than for externalizing disorders [15]. In a similar vein, the cross-informant agreement of psychiatric diagnoses has also been found to be weak [16, 17]. A number of factors have been hypothesized to be involved here. First, children as well as parents may sometimes provide socially desirable responses rather than valid reports [18]. Second, emotional mood states may interfere with cognitive processing compromising children’s memory retrieval and subsequent accuracy [19]. Third, some symptoms of psychopathology (and this is particularly true for internalizing problems) may manifest beyond parents’ awareness or may only occur in situations outside the home [19]. With regard to the Kid-SCID, the agreement between diagnoses as established with the child and parent versions has not been examined so far.

The validity of the Kid-SCID can also be established by determining the agreement between the diagnoses determined by the Kid-SCID and some kind of a “gold standard” of the child’s ultimate clinical diagnosis. Spitzer [20] has formulated the “LEAD” standard, which concerns a number of guidelines to reach an approximation of the ultimate clinical diagnosis. This standard involves conducting longitudinal assessment (L), done by expert diagnosticians (E), using all available data (AD) including family informants and clinician’s observations [21]. To the author’s best knowledge, no studies have yet addressed the validity of the Kid-SCID. That is, no study can be found in the literature that has made a comparison between the Kid-SCID (or another interview of childhood psychiatric diagnoses) and the ultimate clinical diagnosis based on a procedure that approaches the LEAD standard. In the current study, the ultimate clinical diagnosis was not made independently from the Kid-SCID diagnosis so a comparison of the Kid-SCID with the ultimate clinical diagnosis illustrates how the Kid-SCID might be used in clinical practice in combination with other available clinical data.

Taken together, the Kid-SCID seems to be a promising assessment interview for the classification of psychiatric disorders in children and adolescents, but clearly more research is warranted to further establish its reliability and validity. The present study addressed this issue in a population of clinically referred children and adolescents. Its specific aims were (1) to assess the inter-rater reliability and internal consistency of the Kid-SCID diagnoses as based on children’s and parents’ interviews separately; (2) to examine the agreement of various Kid-SCID classifications between children and parents; and (3) to compare the Kid-SCID classifications to an ultimate clinical diagnosis as established during an extensive intake procedure based on the LEAD guidelines. The kappa coefficient which is often used to address the abovementioned research questions has been criticized for being dependent on the base rate of disorders. That is, kappa values tend to drop when there is a low base rate and therefore it is preferable to include the proportion of specific agreement as an alternative index of agreement. More specifically, the influence of prevalence is then transformed into a probability of agreement for positive and negative ratings, which has clinical meaning [22]. Investigation of the reliability and validity of the Kid-SCID is clinically relevant. That is, it remains to be determined whether relying solely on the Kid-SCID interview may be sufficient for making an ultimate clinical diagnosis and whether there is agreement on the outcomes of child and parent interviews.

Method

Participants and Procedure

Participants in this study were recruited at the child and adolescent division of the Maastricht Community Mental Health Centre (RIAGG) in Maastricht, the Netherlands. This facility is specialized in the diagnostics and treatment of youngsters (age <18 years) with mental health problems. In the period between January 2010 and January 2012, all youths who were referred to the centre were subjected to the standard extensive intake procedure that follows the LEAD procedure. More specifically, the longitudinal aspect of the diagnostic evaluation includes revision of (prior) diagnoses as new information becomes available through further assessment or during treatment. The experts were all licensed psychologists who were trained in using the Kid-SCID. The outcome of the Kid-SCID interview was used in the multidisciplinary team in which ultimate clinical diagnoses were made. In addition to the outcome of the Kid-SCID and in line with the LEAD approach, information from teachers and clinical observations made during the intake phase were also used in establishing a diagnosis. For children aged 8 years or older, the Kid-SCID interview was administered to both the child and the parents in two separate sessions, whereas for children below the age of 8, only parents were interviewed. Children who were in an acute psychiatric crisis or seeking treatment after being diagnosed somewhere else were excluded.

Three subsamples were used in the current study. The rationale for using three subsamples was that each research question required the inclusion of a subset of respondents. Due to missing data on some of the assessments, the sample size varied for each of the three research questions. For the assessment of the inter-rater reliability of the Kid-SCID, a total of 55 audiotaped interviews (i.e., 26 interviews of parents and 29 interviews of children) were reassessed by a trained psychologist. The actual sample size ranged between 23 and 26 for parents and between 27 and 29 for children. Both parents and children signed a written consent form for having the sessions audiotaped and rated by the researchers. The mean age of these children was 10.0 years (SD = 3.4; range 6–17 years) and 28 % of them was female. To examine the internal consistency and parent–child agreement of various Kid-SCID diagnoses, data of 144 children and adolescents could be employed. The actual sample size ranged between 141 and 144 for the various Kid-SCID diagnoses. Their mean age was 10.2 years (SD = 3.3; range 8–18 years) and 32.6 % was female. Finally, to evaluate the correspondence between the Kid-SCID diagnoses and the ultimate clinical classifications, data of 169 parents and 113 children and adolescents were used. Mean age of the young people in this sample was 10.0 years (SD = 3.4; range 5–18 years) and 33.5 % was female. The study protocol was approved by a local institutional review board.

The Kid-SCID Interview

As already noted in the introduction, the Kid-SCID [7] is a semi-structured interview instrument to generate childhood diagnoses as specified in the DSM-IV(-TR), which is based on the SCID for adults [1]. The following clinical modules are included in the Kid-SCID: ADHD, ODD, CD, depressive disorder, dysthymic disorder, bipolar disorder, separation anxiety, social phobia, specific phobia, obsessive–compulsive disorder, PTSD, generalized anxiety disorder, panic disorder, agoraphobia, and adjustment disorder.

The Kid-SCID can assist clinicians in carrying out a clinical evaluation of children and adolescents. The probe questions are guidelines to determine whether the criteria of various disorders are satisfied. The rater has to include his/her own judgment in determining whether a certain criterion is present. As such, a rater can hear a ‘yes’ response from the respondent but nonetheless reject it if he/she considers the pertinent symptom as clinically insignificant. In a similar vein, clinical observation can also be included to score symptoms. For example, obvious fidgetiness would yield a positive score even if a patient denies being hyperactive. Each module ends with a decision as to whether a diagnosis is present or not. Administration of the full interview takes about 90 min. The Kid-SCID interviews were conducted by licensed psychologists who received training in the use of the Kid-SCID. The same interviewer administered the Kid-SCID to parents as well as to the child or adolescent. Six trained psychologists who were blind to the original scores coded the audio-taped sessions in order to assess the inter-rater reliability. Thus, each session was rated by the clinician who administered the interview and one trained psychologist.

Statistical Analysis

All analyses were carried out with SPSS for Windows version 21. Cohen’s kappa was calculated to assess inter-rater reliability, parent–child agreement, and to determine the correspondence of the Kid-SCID diagnoses and the ultimate clinical diagnosis. As a rule of thumb, kappa values below .40 indicate poor agreement, values between .40 and .69 point out moderate agreement, while values between .70 and 1.00 signal good to excellent agreement [23]. However, the kappa coefficient has been criticized for being dependent on the base rate of disorders (that is, kappa values tend to drop when there is a low base rate) and therefore it is preferable to include the proportion of specific agreement as an alternative index of agreement [22]. More specifically, in a 2 × 2 table, the a and d cells displays the number of cases on which there is positive agreement (i.e., disorder present) and negative agreement (disorder not present) respectively. Disagreement between raters is shown in the b and c cells. Positive agreement (PA) can subsequently be computed by the following formula: PA = 2a/(2a + b + c), whereas negative agreement (NA) is calculated as follows: NA = 2d/(2d + b + c). The influence of prevalence is now transformed into a probability of agreement for positive and negative ratings, which has clinical meaning [22]. In the current study, positive and negative agreement values were obtained in addition to the kappa coefficients.

Results

Before addressing the main results, one remark needs to be made. In order to be able to interpret Cohen’s kappa and the percentages of (positive and negative) agreement below, the actual number of cases for which no agreement (b + c), positive agreement (a), or negative agreement (d) was obtained, are presented as background information in Table 4.

Inter-Rater Reliability

Table 1 presents an overview of the inter-rater reliability (expressed in Cohen’s kappa and proportion of agreement) of the Kid-SCID diagnoses for parent and child report separately. Overall, the kappa values were reasonable to good for all modules of the Kid-SCID and the percentages of specific agreement were high across the various disorders. The kappa coefficient for specific phobia was low. For specific phobia assessed with the parent version, none of the 26 children were rated as having specific phobia by either of the two raters. For 24 children, this diagnosis was absent, whereas disagreement between raters was found for 2 cases. This illustrates that low kappa values can be obtained despite excellent (negative) agreement.

Table 1 Interrater agreement of the Kid-SCID diagnoses as expressed in Cohen’s Kappa and percentages of agreement for parents (n = 29) and children (n = 26)
Full size table

Internal Consistency

Internal consistency coefficients for various Kid-SCID diagnoses are presented in Table 2. As can be seen, satisfactory to good Cronbach’s alphas (ranging between .70 and 1.00) were obtained for most diagnoses in parents and children. Sufficient alpha coefficients (ranging between .60 and .70) were found for ADHD (parents), and generalized anxiety disorder (parents). Insufficient or even poor reliability was documented for separation anxiety (parents and children), social phobia (children), and obsessive–compulsive disorder (children). Further analyses revealed that in some cases the internal consistency of these diagnoses improved when removing an item (symptom) from the module. For example, in the case of separation anxiety disorder, discarding the symptom “persistent reluctance or refusal to go to sleep without being near a major attachment figure or to sleep away from home” increased the alpha of the parent version this diagnosis from .54 to .67, whereas removing the symptom “persistent and excessive worry about losing, or about possible harm befalling, major attachment figures” increased alpha from .49 to .55 for the child version diagnosis. In a similar vein, discarding the symptom “the person attempts to ignore or suppress thoughts, impulses, or images or to neutralize them with some other thought or action” in obsessive–compulsive disorder increased alpha from .58 to .66 (children). Further, for some clinical disorders internal consistency could not be calculated as there were no or only few cases who endorsed the pertinent symptoms and criteria (i.e., mania, hypomania, panic disorder, and agoraphobia).

Table 2 Internal consistency coefficients and parent–child agreement of Kid-SCID diagnoses (n = 144)
Full size table

Parent–Child Agreement

Table 2 shows the parent–child agreement also expressed in Cohen’s kappa and percentage of positive and negative agreement for the various Kid-SCID classifications. Overall, children and parents showed reasonable to good agreement. As can be expected, relatively higher kappa values and percentages of specific agreement were found for diagnoses with relatively high base rates in the sample (i.e., for which both children and parents indicated the diagnosis to be present). Moreover, the parent interviews revealed more ADHD and ODD classifications as compared to the child interviews.

Correspondence Between Kid-SCID Diagnosis and Ultimate Clinical Diagnosis

Table 3 presents degree of correspondence (Cohen’s kappa) and positive and negative percentages of agreement between Kid-SCID classifications and the final DSM-IV diagnoses based on the full intake procedure. As can be seen, kappa values ranged from poor to good agreement, with higher kappa values being obtained for diagnoses with relatively higher base rates in the sample. Note also that for some diagnoses positive agreement was higher for parents, whereas for other diagnoses positive agreement was higher for children. This indicates that both informants are necessary for obtaining diagnoses. Finally, ratings of negative agreement were generally higher than ratings of positive agreement. The percentage of agreement for ADHD was fairly low. Inspection of the data revealed that there was quite some overlap between the ADHD and ODD classification.

Table 3 Correspondence between Kid-SCID diagnoses with ultimate clinical diagnosis for parents (n = 169) and children (n = 113)
Full size table

Discussion

The purpose of the current study was to further examine the psychometric properties of the Kid-SCID, a semi-structured interview used for the classification of psychiatric disorders, in a sample of clinically referred children and adolescents. The results can be summarized as follows. First, indices of reliability including inter-rater reliability and internal consistency of the symptoms of each classification of the Kid-SCID were in general reasonable to good. Second, Kid-SCID diagnoses obtained from children and parents showed reasonable to good agreement. Parents more often indicated the presence of externalizing disorders (ADHD and ODD) as compared to children. Finally, a comparison of the Kid-SCID diagnosis with the ultimate diagnosis obtained after the full intake procedure (which included the information as obtained by means of the Kid-SCID) ranged from poor to good. Most symptom overlap was found between ADHD and ODD, indicating that for some children, it is difficult to differentiate between these two clinical diagnoses on the basis of the Kid-SCID.

The inter-rater reliability as expressed by Cohen’s kappa coefficients varied between reasonable and excellent for the majority of the Kid-SCID diagnoses. The kappa coefficient for specific phobia was, however, deceptively low. This was due to the low base rate of this disorder in the current sample. Although the findings with respect to inter-rater reliability generally concur with other findings [13, 14], it should be kept in mind that differences due to varying base rates make the kappa coefficient difficult to compare across studies. The percentage of specific agreement was very good to excellent for all diagnoses. However, no adequate comparison can be made with previous studies as these specific forms of agreement have not been reported.

Another aspect of reliability refers to the internal consistency of symptom ratings of various Kid-SCID classifications. In general, Cronbach’s alphas were acceptable to good, and well in line with previous findings [13]. For some diagnoses such as separation anxiety disorder and obsessive–compulsive disorder, it was found that some symptoms did not fit well with the rest of the symptoms. For instance, refusal to sleep as reported by parents and worrying about loss of attachment figures in children did not contribute to the homogeneity of separation anxiety disorder diagnosis, and the same was true for the tendency to suppress symptoms in obsessive–compulsive disorder as reported by children. Thus, these symptoms may occur in the absence of the remaining symptoms of the disorders, or vice versa may not occur when the other symptoms are present.

The parent–child agreement of Kid-SCID diagnoses revealed a mixed pattern of results. That is, for some disorders (e.g., adjustment disorder, dysthymic disorder) poor agreement was found, whereas very good agreement was found for some anxiety disorders (e.g., PTSD, panic disorder, social phobia) and CD. Our findings partly concur with results from other studies showing that there is often only marginal agreement in the ratings of childhood psychopathology between children and parents [15, 16]. It has also been noted that agreement between parents and children normally is significantly lower for internalizing disorders than for externalizing disorders [15] since the former problems tend to be inwardly focused and the latter are more readily observable. In line with this, we found that parents more often reported externalizing disorders as compared to internalizing disorders. However, it should be noted that the parent–child agreement for some of the anxiety disorders was (very) good, suggesting that parents are capable of noticing internalizing problems in their offspring when these problems are severe enough [24]. With regard to the results on the parent–child agreement, it should be noted that the same interviewer administered the child and parent interview, which conceals a potential bias [25]. The interviewers in our study were explicitly instructed that it was not allowed to use the information as obtained from the child during the interview with the parent(s) and vice versa, but it cannot be ruled out that such a “bleeding” of information did occur.

The agreement between the outcome of the Kid-SCID interview of children and parents on the one hand and the ultimate clinical diagnosis ranged from poor (e.g., generalized anxiety disorder, dysthymia) to good (e.g., panic disorder, obsessive–compulsive disorder), with the percentages of agreement varying between reasonable and very good. ADHD and ODD had somewhat lower percentages of agreement which seemed due to a partial overlap of ADHD and ODD symptoms. Previous research has indicated that hyperactivity and conduct problems may have similar etiological roots [26], which make it plausible that both disorders also share some symptoms. It is important to note that the ultimate clinical diagnosis also included the evaluation of the school teacher and all other diagnostic information gathered during the intake phase. This additional information may at least partly explain the disagreement between the outcome of the Kid-SCID and the ultimate clinical diagnosis. In line with LEAD guidelines, this additional information should always accompany the Kid-SCID when assessing psychiatric disorders.

There are a number of limitations of the current investigation that should be mentioned. First, we did not address gender differences (father vs. mother) as the Kid-SCID was administered to both parents. Second, we did not examine age effects (younger vs. older children) on the classification of mental disorders as this was not possible due to the limited sample size. Third, with respect to assessing the validity of the Kid-SCID, information obtained with the Kid-SCID was used for the establishing the ultimate clinical diagnosis. One could argue that this procedure limited the examination of the validity of the Kid-SCID given that the Kid-SCID outcome influenced the criterion. However, it is important to note that the purpose of this study was to compare the Kid-SCID with the ultimate clinical diagnosis for which all information was used (in accordance with the LEAD standard). Finally, not all DSM-IV disorders are included in the Kid-SCID. For example, autism spectrum disorder, which is a common psychiatric diagnosis of the children and adolescents in our treatment centre, is not assessed. An important area for future research is to extend the Kid-SCID with these disorders. Despite these limitations, the results indicate that the Kid-SCID is in general a reliable and useful instrument to assess psychiatric disorders in clinically referred children and adolescents. Moreover, the current study also highlights the proportion of specific agreement as an alternative index of agreement. As pointed out by de Vet et al. [22], this measure is helpful for clinicians and deserves a broad application. Future research should focus on the evaluation of other aspects of reliability and validity that were not addressed here (e.g., test–retest stability and comparison with other instruments such as semi-structured interviews) and fine-tuning with the specific criteria as listed in the DSM-5.

Summary

This study investigated some psychometric properties of the Kid-SCID in a Dutch sample of children and adolescents who had been referred to an outpatient treatment centre for mental health problems. Results revealed that the Kid-SCID can generally be seen as a reliable and useful tool that can assist clinicians in carrying out clinical evaluations of children and adolescents.