Introduction

Mental health and psychiatric disorders among children and youth (hereafter referred to as children) are a significant global health concern, as one out of every five children in North America will experience a mental health concern such as depression, anxiety, attention-deficit hyperactivity disorder, or conduct problems [1, 2]. However, only an alarming 25% of these children will receive the proper treatment and support needed for these disorders [3]. Further, disturbing statistics reveal suicide to be the second highest cause of death in adolescents and young adults 15–25 years of age, second only to accidents in the industrialized world [4]. It is clear therefore that childhood mental healthcare should be a top priority. Moreover, mental health issues are often present and persistent across the life course of an individual; indeed, 50–70% of adult mental health issues have their onset in childhood [5]. The research suggests that when the onset of mental health disorders is earlier than 25 years of age, there is a frequent pattern of remission and relapse over the life span [6]. This illustrates the importance of identifying mental health challenges as early as possible to implement individualized treatment plans that can be utilized, and as needed, adapted throughout the child’s life. The Mental Health Commission of Canada (MHCC) suggests identifying at-risk children and intervening as soon as possible is imperative, as it can improve their life trajectories and reduces the prevalence of mental health problems and illnesses in adulthood [7]. Unfortunately, the negative effects of childhood mental health issues place children at increased risk for a variety of negative outcomes, including academic difficulties and underachievement [8, 9], underemployment [10], involvement with the criminal system [11, 12], self-harm, and increased suicide [4, 13]. This speaks to the importance of early identification, treatment, and proper support to reduce distress and burden to the child and his or her family.

With mental health concerns growing steadily, the usage of both emergency rooms and hospitals in Ontario has risen by 60% over the last decade for children seeking help for mental health and substance use problems [4]. This increase in the use of public health facilities, in conjunction with the increase in prevalence rates of mental health concerns, strongly highlights the need for a mental health assessment instrument that reliably expedites, prioritizes, and triages children to needed services. While there have been numerous assessments created to examine children’s functioning, each assessment is generally confined to assessing symptoms specific to one disorder (e.g., anxiety). Consequently, it is often the case that numerous assessments must be employed in order to gather a comprehensive view of overall functioning and mental health needs of the child. This use of multiple assessments across several service sectors (e.g., school, mental health agencies, hospitals) has resulted in a truly fragmented system of care, one in which there is a lack of communication among service providers, additional clinical time and resources consumed resulting in the duplication of services, and longer wait times. This, in turn, results in increased financial and emotional burden to clients (and their families) as they are required to attend multiple appointments and often must tell their story numerous times [14, 15]. Further exacerbating this issue is that agencies often use their own “homemade” assessments, creating a system of unstandardized measures across services, often with questionable reliability and validity. This makes it extremely difficult to compare children across various service sectors, as well as making it challenging to track individuals, interventions, and progress over time. Consequently, this poses a significant challenge to efficiently deliver proper evaluation and treatment to those who may be struggling from a myriad of challenges. It is clear that an integrated health information system utilizing common screening and assessment instruments across service sectors is desperately needed.

In an international effort to support global innovation and foster continuity of care, interRAI is a collaborative network of researchers and expert clinicians in over 35 countries who aspire to improve care for vulnerable individuals from birth to elder care. The ultimate goal of this research network is to create a system of care that is seamless and continuous across the lifespan; one that is evidence-informed through the collection and analysis of high-quality data to support assessment, care-planning, outcome evaluation, and resource allocation. A common language across service sectors is utilized across the components of the assessment suite to foster continuity of care, reduce duplication, and enhance communication among clinicians in the varying institutions, facilities, schools and organizations working with the same individuals (www.interRAI.org). While many interRAI instruments have been implemented in the adult service sector, until the recent development of the interRAI Child and Youth suite, there has been no other instruments for children in use by the healthcare system that creates both a unified system across services while also being an effective tool for creating urgency algorithms for triaging and prioritizing mental health needs of children.

In response to the need for more coordinated and consistent assessment system for children, the interRAI Child and Youth Mental Health Screener (ChYMH-S) [16] was developed. The ChYMH-S is a brief screener that has been adapted from the full Child and Youth Mental Health (ChYMH) [17] assessment to serve as an initial assessment for early triaging, prioritization of services, and to determine if a more comprehensive assessment is required. The full ChYMH assessment instrument is a more far-reaching, comprehensive standardized measure for extensively assessing children’s functioning, challenges, strengths, and preferences in the key domains of life. The screener however allows for a quicker snapshot of the child’s functioning and takes only approximately 20 min to complete (compared to 60–90 min for the full ChYMH assessment). Specifically, the ChYMH-S was designed to be utilized as the first step in the assessment process across various service sectors to fill the void in the health care system for children aged 4–18 years. This screener provides an initial assessment of need by providing an overview of the child’s overall physical health and psychological well-being, which is designed to integrate into a more global assessment and health information system across multiple service sectors and various settings (i.e., mental health agencies, community agencies, hospitals, educational settings, and youth justice facilities). The primary use of the ChYMH-S is to support immediate decision-making related to placement and service urgency. Although it informs immediate care triaging, it is not intended as a replacement for, or an alternative to, the comprehensive use of the full ChYMH assessment and/or its associated evidence-informed care planning guidelines or Collaborative Action Plans (CAPs). This screening approach facilitates a more efficient preliminary assessment, allowing service providers to empirically identify children in most need for care. While not intended as a diagnostic tool, its use of urgency algorithms (i.e., categorization of risk level) supports efficient triaging and prioritization of children, and further, enhances referrals and transition to alternative care where needed [18].

Within the ChYMH-Screener, five full scales can be triggered, including the following: Anxiety Scale, Distraction/Hyperactive Scale, Social Disengagement Scale, Depression Severity Index, and Positive Symptoms Scale. In addition, two revised scale short-versions can also be activated, including the Internalizing-Short Scale and the Externalizing Short-Scale. Further, the interRAI Child and Youth Algorithm for Mental Health and Psychiatric Services [16] can be generated. The internal reliability and validity of several of the embedded scales within the interRAI Child and Youth Suite have previously been assessed [19,20,21,22,23,24]. An empirical investigation used data collected (N = 1297) from 15 mental health agencies throughout Ontario to assess inter-item reliability of several of the scales on the ChYMH (i.e., Aggressive/ Disruptive Behaviour Scale, Social Disengagement Scale (previously the Anhedonia Rating Scale), Anxiety Scale, Caregiver Distress Scale, Communication Scale, Cognitive Functioning Scale, Depressive Symptoms Scale, Distractibility/Hyperactivity Scale, Peer Conflict Scale, Sleep Difficulties Scale). Overall, the scales demonstrated strong inter-item reliability, with all scales showing Cronbach’s alpha of 0.70 or greater, with the exception of one scale (Sleep Scale was 0.66). Further, a smaller subset of participants (N = 48–53) also completed an existing established mental health measure with similar relevant domains. This was done to assess scale criterion validity. As predicted, ChYMH subscales correlated well with relevant criterion measures. These findings supported the clinical utility of the ChYMH for use among clinically referred children and were consistent with previous research using the interRAI adult assessment instruments [24,25,26] which demonstrated strong internal consistency and criterion validity. While ChYMH subscales demonstrate strong psychometric properties in previous review, an evaluation of these subscales within the screener version of the ChYMH has yet to be conducted.

Present Study

The purpose of the current study was to provide an initial empirical investigation of the psychometric properties of the ChYMH-S, including the criterion and convergent validity of the screener. In addition, authors explored the utility of the revised short-form versions of the Internalizing, Externalizing, and Disruptive/Aggression Behaviour subscales. This study provides important information as to effectiveness of the ChYMH-S in the successful identification of behavioural, emotional, and adaptive challenges in children in a much shorter timeframe, as well as preliminary information regarding the utility of the newly revised short-form scales for this brief screener that provides integration directly to the ChYMH.

Method

Participants

ChYMH-S Validity Sample

The present sample was comprised of 79 children (41 male; 38 female) between the ages of 4 to 13 years (M = 8.33, SD = 2.58). The sample included children who were predominately born in Canada (92.4%) and were Caucasian (89.9%). Demographic information of the child’s parents is provided in Table 1. Children and families were recruited from participating schools in the Southwestern Ontario area. Families were contacted by mass email outreach, school bi-monthly newsletters, and posters placed in the community and school partner organizations. To compensate the participants’ time and expense for transportation, a ten-dollar ($10) Tim Horton’s gift card was given to each family.

Table 1 Parental demographic information of the sample
Full size table

ChYMH-S Full Sample

In addition to the validity sample, the full ChYMH screener data set was also used to establish the psychometric quality of the revised short versions of the Internalizing, Externalizing, and Disruption/Aggression Behaviour scales. The ChYMH-S sample consists of data collected from 19,645 participants from various Ontario community-based (outpatient) and residential (inpatient) children’s mental health facilities. Of the sample collected, 10,531were male (53.6%) and 9,114 were female (46.4%). Participant age ranged from 4 to 18 years (M = 11.11, SD = 3.39). The sample also included children who were predominately born in Canada and were Caucasian.

This full ChYMH-S sample data was only used for the purposes of psychometric evaluation of the revised shortened scales. Behavior Assessment System for Children-Third Edition (BASC-3) validity data was only collected for a subset of the ChYMH-S sample; therefore, the full sample cannot be used for validation purposes. For all other analyses, the validity sample with contained data from both the ChYMH-S and the BASC-3 was used.

Procedure

ChYMH Screener-Validity Sample

The assessment data was collected by PhD and Master's-level students with three-day training to administer the interRAI ChYMH full and screener instruments. Interviews were arranged and conducted at various locations in a large city within Ontario including various schools. Consenting and assenting parent and child participants completed: (1) a short demographic questionnaire, (2) the interRAI ChYMH-S assessment; and (3) the BASC-3: Parent Rating Scales (PRS). Assessors collected data for the ChYMH-S using a combination of parent interview, child interview, and any other collateral information available (e.g. previous doctor and/or school reports). The demographic questionnaire and BASC-3 PRS were both completed independently by the parent. The entire assessment process took approximately 90 min. Collection took place between June 2016 and May 2017.

The study was approved by the University of Western Ontario Ethics Board (REB #106741). All procedures were in accordance with the ethical standards of the institution. Informed consent was obtained from all individual participants and/or guardians as part of their standard clinical care at each agency.

ChYMH Screener-Full Data Set

Individuals administering the screener assessment ranged in discipline and expertise, and included: psychiatrists, child and youth workers, speech and language therapists, developmental social service workers, social workers, nurses and psychologists. To learn how to appropriately administer and score the screener, all assessors were required to complete a full-day training session. Most assessors also had previously received training on other interRAI assessments as well. Upon completion of training the assessor completed the interRAI competency evaluation. This study was approved by the University of Western Ontario Ethics Board (REB #106415).

Measures

ChYMH-S

The interRAI ChYMH-S is a manualized, semi-structured assessment tool that integrates into a full comprehensive assessment designed for evaluating the needs of children aged 4–18 years of age with mental health concerns. It consists of 99 items and is divided into 10 subsections (identification information; mental state indicators; substance use or excessive behaviour; harm to self and others; behaviour; cognition, communication, and development; stress, trauma, and social relationships; education; summary; assessment information). The items on the ChYMH-S describe the child’s risk of danger to self or others, as well as an overall view of the child’s presentation and functioning. This enables an assessor to briefly assess key domains of function, mental health, family and social support, and risk of harm to self or others. This screener takes approximately 20 min to complete.

Measure Development

The items that appear on the screener were derived using the interRAI ChYMH assessment as a guide, and reflect a condensed, version of the full assessment, focused on key items related to prioritization and triaging. A detailed explanation of the development of the full assessment has been discussed in detail elsewhere [17, 20, 26, 27]. The assessment was developed to capitalize on the clinician’s judgement and observational skills, and to integrate brief, multidisciplinary evaluation of the child’s strengths, preferences, and needs. Similar to other interRAI instruments, the screener was based on a semi-structured interview format that captures the collection of a range of common issues encountered in children’s mental health. On the screener, the following domains are captured: (1) Mental State Indicators (e.g., Mood disturbance, Anxiety), (2) Substance Use or Excessive Behaviour, (3) Harm to Self and Others, (4) Behaviour, (5) Cognition, Communication & Development, (6) Stress, Trauma and Social Relationships, and (7) Education.

ChYMH-S Subscales

Anxiety Scale (ANX)

The anxiety scale has seven items which assess the frequency of symptoms of anxiety (i.e., Repetitive anxious concerns, Unrealistic fears, Obsessive thoughts, Intrusive thoughts or flashbacks, Episodes of panic, Hypervigilance and Nightmares). The items are assessed on a five-point ordinal scale (0 = not present to 4 = exhibited daily in last 3 days, three or more episodes or continuously). The total score consists of the total on the seven items, with a possible range of 0–28, with higher scores indicating more anxiety symptoms. The psychometric properties of the Anxiety Scale have been investigated and demonstrate good scale reliability. In an analysis of multiple samples, polychoric correlations among the items ranged from 0.284 to 0.546, and an unrestricted factor analysis show only one factor extracted from data with all items loading heavily on the first unrotated factor, explaining between 43.18% and 51.45% of the total variance. The standardized Cronbach’s alpha for the Anxiety Scale in these samples ranged from 0.73 to 0.81 [21].

Disruptive/Aggression Behaviour Scale (DABS)-Short

The DABS-Short has four items assessing the frequency and severity of aggressive and disruptive behavior (i.e., physical abuse, verbal abuse, socially inappropriate or disruptive behavior, and destructive behavior toward property). The items are assessed on a five-point ordinal scale (0 = not present to 4 = exhibited daily in last 3 days, three or more episodes or continuously), and the total DABS score consists of the total on the five items, with a possible range of 0–16 and higher scores indicating higher levels of aggressive and disruptive behaviour. A psychometric evaluation of the DABS scale has been conducted, and results from the factor analysis and Item Response Theory (IRT) analysis demonstrated good measurement properties. In a Receiver Operating Characteristic (ROC) curve analysis, the area under the curve (AUC) for the DABS is 0.75 for a diagnosis of disruptive behaviour disorder. Therefore, the converging results suggest that the interRAI DABS serves as an effective measure to detect externalizing mental health indicators [19].

Externalizing Scale (EXT)-Short

The EXT-Short consists of seven items used to assess the frequency and severity of externalizing symptoms (i.e., Stealing, Bullying peers, Impulsivity, Verbal abuse, Violence to others, Violent ideation, Intimidation of others or threatened violence). The frequency of each behaviour is assessed using a five-point ordinal scale (0 = not present to 4 = exhibited daily in last 3 days, three or more episodes or continuously) and are recoded to 0 (score of 0) or 1 (scores of 1–4). The scores range from 0–7 and higher scores indicate higher frequency of externalizing behaviour. The Externalizing scale represents a larger domain that reflects constructs related to both proactive, reactive aggression, violence and impulsivity. To support brevity within this sub-domain, a smaller number of items were utilized to effectively support triaging and prioritization. Impulsivity was included within this domain due to its strong relationship with other externalizing issues. The full version of the Externalizing Scale has undergone a psychometric evaluation, and results indicate high internal consistency and strong correlations with the appropriate criterion measures [17].

Hyperactivity/Distractibility Scale (HDS)

The HDS consists of four items that assess the frequency of hyperactivity and distractibility (i.e., impulsivity, ease of distraction, hyperactivity, and disorganization). The frequency of each behavior is assessed using a five-point ordinal scale (0 = not present to 4 = exhibited daily in last 3 days, three or more episodes or continuously). The total score has a possible range of 0–16, and higher scores indicate higher levels of hyperactivity and distractibility. A psychometric evaluation of the HDS scale has been conducted and results from the factor analysis and IRT analysis demonstrated strong measurement properties. The ROC analysis revealed the area under the curve (AUC) for the HDS is 0.79 for a diagnosis of attention deficit hyperactivity disorder. Therefore, the converging results suggest that the interRAI HDS is an effective tool to detect externalizing mental health indicators [18].

Internalizing Scale (INT)-Short

The INT-Short consists of eleven items used to assess the frequency and severity of internalizing symptoms (i.e., Repetitive anxious complaints/concerns, Hypervigilance, Unrealistic fears, Episodes of Panic, Lack of Motivation, Anhedonia, Withdrawal from Activities of Interest, Made Negative Comments, Self-Deprecation, Expressions of Guilt/Shame and Expressions of Hopelessness). The frequency of each behavior is assessed using a five-point ordinal scale (0 = not present to 4 = exhibited daily in last 3 days, three or more episodes or continuously) and scores range from 0 to 44. Higher scores indicate higher levels of internalizing symptoms (i.e., emotional distress/disturbance). The full version of the Internalizing Scale has undergone a psychometric evaluation, and results indicate high internal consistency and strong correlations with the appropriate criterion measures [28].

Demographic Questionnaire

The demographic questionnaire incorporated information not captured in the ChYMH-S or BASC-3 PRS forms. Specifically, it collected information in three categories. The first, Immigrant Generation Status obtained information about the parent and child’s birthplace and parental ethnicity. The second, Socioeconomic Status asked about parental employment and household income (e.g., for each parent, identify if he/she is employed or not within the labour force, current household income). The final section collected information on Parent Education, inquiring as to each parent’s highest level of completed education. The questionnaire took approximately 10 min to complete.

BASC-3 Parent Rating Scales

For the purposes of this validation project, the BASC-3 [29] was chosen as the comparative measure in which to assess the screener’s ability. The BASC-3 is a multidimensional assessment system that captures behavioural and emotional functioning across the developmental range of 2 through 25 years. The PRS forms are a comprehensive measure of problem behaviors that the child expresses in community and home settings. This measure is designed to be completed by the child’s parent or guardian. There are three different versions of the PRS that can be used depending on the child’s developmental age (Preschool; 2–5 years, Child; 6–11 years, Adolescent; 12–21 years). The form contains descriptors of behaviors that the parent/guardian then rates on a four-point scale of frequency (Never, Sometimes, Often, and Almost Always). The form takes approximately 20 min to complete.

The BASC psychometric properties are well established. The BASC-3 demonstrates good overall reliability with strong internal consistency (reliability coefficients ranging from good to excellent (0.80–0.90), most above 0.90), test–retest reliability (coefficients 0.80 or higher) and interrater reliability (majority falling in the 0.65–0.75 range). The PRS-specific validity has also been examined in depth, with internal structure analyses indicating an overall moderate to good model fit (Comparative Fit Index (CFI) ranged from 0.76 to 0.89), and further, strong correlations with BASC-2 (ranging from 0.80 to 0.99) and with expected scales on a variety of other assessment tools, including: Achenbach System of Empirically Based Assessment (ASEBA) Child Behaviour Checklist [30] Conners 3 [31], Autism Spectrum Rating Scales (ASRS) [32] and Delis Rating of Executive Functions (D-REF) [33]. Finally, the BASC has also demonstrated utility in clinical populations. While ample psychometric evaluation of the BASC-3 is not available (likely given its recent publication), many studies have evaluated the BASC [34] and BASC-2 [35], showing it to be effective for use in a variety of clinical populations [36, 37].

Analysis Plan and Scale Examination

Comparison of BASC-3 PRS and ChYMH-S

To allow for a comparison between scales on the two measures, this first step involved an in-depth evaluation of the assessments at the item, scale, and composite levels. This was an imperative preliminary step because having similar scale and/or composite names between measures does not necessarily imply that the scales have been defined in the same way. Thus, this content validation stage allowed for similar items, scales, and composites to be compared and assessed. This permitted the ability to determine which scales and composites would be appropriate for comparison and could be statistically evaluated to assess whether the two measures were similarly categorizing individuals.

Analyses

First, the inter-item reliability was examined by assessing the inter-item correlations of the scales. Next, the scale criterion validity was assessed by examining the correlations between the scales on the ChYMH-S and the BASC-3 scales. The scales that were compared were determined during the content validation stage, based on similarity between definition and approach in the development of the scales (as described above); Table 2 outlines the scales that were determined appropriate for comparison. Finally, the Internalizing-Short and Externalizing-Short scales were analyzed using the full ChYMH-S sample to establish psychometric properties.

Table 2 Scales to be compared between the ChYMH-S and BASC-3
Full size table

Interpretive Consideration: Short-Form Version of Scales

An important note when interpreting and considering these results is that the screener does not allow for the full Externalizing, Internalizing, and Disruptive/Aggression Behaviour scales (which have been previously psychometrically examined; [17]) to be generated due to the brevity of the measure. Therefore, these comparative analyses in the current study utilized a revised short-from version of these three scales. The Internalizing-Short scale is comprised of 11 or the original 12 items, while the Externalizing-Short Scale is based on seven of the original 12 items. The Disruptive/Aggression Behaviour (Short) scale is based on four of the five original items. These revised shortened versions however were psychometrically evaluated using the full ChYMH-Screener data sample (N = 19,645), and results are presented below.

Results

Scale Reliability: Preliminary Results Using Full Sample

Inter-item correlations are presented in Table 3. Overall, the scales demonstrated strong inter-item reliability (i.e., Cronbach’s alpha of 0.70 or greater for all scales).

Table 3 Internal consistency of ChYMH-S subscales
Full size table

Criterion Validity Results

The correlations between the ChYMH-S scales and the corresponding criterion BASC-3 scales are presented in Table 4. Given the non-normality of the distributions, Pearson’s r correlations were not appropriate; therefore, correlations were determined using Spearman’s rho. As predicted, the ChYMH-S scales were significantly and positively correlated with relevant criterion scales. The strongest correlation was between the interRAI ChYMH Externalizing and BASC-3 Externalizing Problems (rs = 0.766). Closely behind was the correlation between the interRAI ChYMH Externalizing and BASC-3 Aggressive Behaviour (rs = 0.748) and the interRAI ChYMH Distractibility/Hyperactivity and BASC-3 Hyperactivity (rs = 0.703). The smallest correlation observed was between the interRAI ChYMH Anxiety Scale and the BASC-3 Anxiety scales (rs = 0.469).

Table 4 Correlations between ChYMH-S scales and criterion measures
Full size table

Scale Reliability: Revised Internalizing- Short and Externalizing-Short Results

Internalizing-Short (INT-S)

The 11 items of the INT-S scale showed moderate to high inter-item correlations with Spearman’s correlations ranging from r = 0.156, p < 0.001 to r = 0.669, p < 0.001. The standardized Cronbach’s alpha was 0.82. The INT-S total scores had a mean of 9.29 (SD = 7.98), ranging from 0 to 44. The skewness and the kurtosis for the total score distribution was 1.113 (SE = 0.034) and 1.075 (SE = 0.068), respectively.

Externalizing Short (EXT-S)

The five items of the EXT-S scale showed moderate to high inter-item correlations with Spearman’s correlations ranging from r = 0.160, p < 0.001 to r = 0.589, p < 0.001. The standardized Cronbach’s alpha was 0.77. The EXT-S total scores had a mean of 2.49 (SD = 1.98), ranging from 0 to 7. Due to the differential response set, items were dichotomized to equate scores. The skewness and kurtosis for the total score distribution was 0.529 (SE = 0.031) and -0.697 (SE = 0.063), respectively.

Disruptive/Aggression Behaviour Short (DABS-S)

The four items of the DABS-S scale showed moderate to high inter-item correlations with Spearman’s correlations ranging from r = 0.132, p < 0.001 to r = 0.540, p < 0.001. The standardized Cronbach’s alpha was 0.72. The DABS-S total scores had a mean of 3.56 (SD = 3.67), ranging from 0 to 16. The skewness and the kurtosis for the total score distribution was 1.124 (SE = 0.031) and 0.794 (SE = 0.063), respectively.

Discussion

The ChYMH-S was designed to provide early and efficient detection of child mental health problems, enhance initial screening and prioritization across service sectors, improve service system efficiencies, foster improved continuity of care across service sectors, and facilitate transitions across the lifespan. The findings from this study indicate that the ChYMH-S shows stable inter-item reliability among the scales that were tested. These findings are consistent with findings from the full version of the interRAI ChYMH assessment. Based on the content-validation sample, the results reflect initial support for internal consistency and convergent validity via correlations with the BASC-3. Five scale comparisons (see Table 4) were hypothesized and tested; ultimately showing strong, positive correlations between the two measures’ scales.

When examining the content of these scales, an important note is that the BASC-3 uses a parent-rated assessment while the ChYMH-S utilizes a clinician-rated assessment that incorporates knowledge from multiple informants, including the child, parents, as well as other collateral information; therefore, the sources of information are different (e.g., child self-report, clinician-report, parent-report, allied professional reports, records on file). Previous research investigating the use of different informants has revealed that clinician-rated measures often outperform self-reported measures [38]. A meta-analysis by Achenbach, McConaughy, and Howell [39] used data from 119 studies to examine correlations across multiple informants. The findings suggest that overall, there will be lower inter-rater agreement when collecting information from different types of informants (e.g., a parent versus a clinician) (average r = 0.28) compared to when more similar types of informants (e.g., two parents) are used (average r = 0.60). Moreover, even when looking at the interrater reliability of the same type of informant (e.g., parents) one will not always see extremely high correlations. Consequently, different raters are not expected to give identical results (even when they are both parents) because there are likely to be differences in context and situations in which they observe the child, or because the child may act differently in the presence of different raters.

It is important to note that there is a body of literature suggesting that no single source of child assessment is uniformly superior, as different types of respondents can provide varied insights as they relate to clinical phenomena. Consequently, the approach that interRAI utilizes considers multiple perspectives to enhance best-practice methodologies to assessment. Furthermore, the fact that strong correlations were observed for most scales despite different reporting methods suggests that the ChYMH-S is effective at capturing similar constructs in relation to child psychopathology. The lower correlation observed with anxiety is consistent with previous literature that suggests internalizing problems such as anxiety and depression have been found to suffer from lower inter-rater agreements, especially compared to externalizing and behavioural problems [40]. While there are overt behavioural symptoms associated with internalizing problems (e.g., furrowed brow, tearfulness), internalizing problems often reflect the internal state of a child (e.g., negative thoughts, suicidal ideation). Gauging the subjective experience of a child is extremely important to accurately assess the underlying emotional state; thus, evidence is often more accurate when the viewpoint of the child is incorporated into assessment results. However, this can be challenging as the reliability of the information depends on the age, language/communication skills, competency, cognitive abilities, as well as the developmental level of children. Given these issues, this correlation of 0.469 observed across these two measures, which are different informant types and within an internalizing domain, is encouraging and provides evidence to suggest that the interRAI ChYMH-S is also adequately capturing anxiety-related symptomatology in children.

Further, while not all scales between the BASC-3 and ChYMH-S could be compared (see limitations below), the scales that did overlap conceptually demonstrate that the ChYMH-S is measuring the intended characteristic (e.g., hyperactivity, internalizing behaviour, externalizing behaviour) in a consistent manner to the BASC-3. This suggests criterion validity for the scales obtained from the ChYMH-S, and again these findings are consistent with the criterion validity that has been demonstrated with the full interRAI ChYMH assessment.

Finally, this study also sought to explore the psychometric quality of a revised, short-form version of the Internalizing, Externalizing, and Disruptive/Aggression Behaviour subscales. In general, shorter scales tend to be less reliable than longer scales resulting in the attenuation of validity but, as can be seen from the results, this reduction of items in the shortened versions did not substantially influence the psychometric properties. The current study findings suggest that the brevity of the revised, short-form versions of these subscales offer moderate to strong scale reliability. The slightly stronger reliability observed in the INT-S and DABS-S scales (compared to the EXT-S scale) is not unexpected given that the original item count was only reduced by one item in the shortened version for both of these scales, compared to seven items removed in the EXT-S. This first examination of the utility of these scales suggest that the revised versions will offer an appropriate snapshot of the child’s internalizing and externalizing behaviours and characteristics.

Limitations and Directions for Future Research

While this BASC-3 validation is an important step in establishing the efficacy of using the ChYMH-S, it is important to note that this screener and the BASC-3 assessment are not exactly alike, and therefore all scales cannot be compared across measures. In this study, only five comparisons between the two measures’ scales were determined to be appropriate based on overlapping content and sufficient number of item coverage using the screener. Therefore, further examination of the other screener scales compared to additional established adaptive and behavioural measures are needed to examine the criterion validity of these scales. Moreover, in addition to criterion validity, construct validity is also an important component to assess and was not included in the current study, though has been completed in previous research of the ChYMH measure; future research should include this examination to add to the psychometric evaluation of the ChYMH-S.

With regard to study samples, although the full ChYMH-S sample is comprised of a large group of clinically referred children across multiple mental health facilities in Ontario, this sample is likely not representative of the general child population. First, the sample had limited racial and ethnic diversity and is therefore not representative of the country at large. Moreover, given the absence of a normative sample, it is not clear whether the observed psychometric properties would replicate in children who do not have a wide range of physical and mental health problems. Further, the validation study sample was small, and was only available for the ChYMH-S screener sample, not the full ChYMH sample. Therefore, comparative data between the ChYMH and BASC-3 validity measure were only available for a subset of the larger sample. Future research should explore this with larger samples. In addition, the validation sample age range only included children aged 4–13, therefore, these psychometric findings apply to children but not necessarily to adolescents overall.

An important note regarding the use of this screener is that it was not designed to diagnose, but rather to identify signs and symptoms of potential mental health issues that need to be addressed. Rather, the interRAI Children’s Algorithm for Mental Health and Psychiatric Services (ChAMhPs) is the algorithm that provides the detection of cases that need more immediate or intensive follow up. The ChAMhPs is an empirically based decision-support tool designed to inform the need and urgency for comprehensive face-to-face mental health assessment or service for children between 4 and 18 years of age. It is subdivided by age group and provides a scoring range from 0 to 6 based on levels of urgency. A publication regarding this algorithm is underway [41, 42].

A final limitation of this study is that this is the first instance of the revised shortened versions of the Internalizing, Externalizing, and Disruptive/Aggression Behaviour scales; therefore, further examination and sampling is needed to bolster and support findings of psychometric integrity. This can be done with future ChYMH-S samples. However, the preliminary results show moderate to strong alpha levels among the reduced item count, which is encouraging.

Conclusion

The results of this initial validation study demonstrate that many of the scales captured by the ChYMH-S possess strong psychometric qualities. These findings suggest that this screener is effectively capturing emotional, behavioural, and adaptive characteristics in children in a manner that is consistent with a previously established behavioural assessment. These findings are also consistent with a larger body of research using the full version of the ChYMH, which show psychometric rigour, including internal consistency and strong criterion validity. This study explored the criterion validity of some of the ChYMH-S’s scales and is an imperative preliminary step in establishing the ChYMH-S usefulness and effectiveness at identifying signs and symptoms related to children’s mental health issues. This kind of data fosters trust in the measure, and points to the screener’s credibility and appropriateness for use in clinical settings.

The ChYMH-S is a relatively short, user-friendly screening instrument that offers clinicians and researchers important features that other assessments cannot. It was designed to enhance prioritization and triaging, while providing the first step in the childhood assessment process; therefore, it is part of the larger body of an integrated interRAI assessment method and practice. It provides a process of identifying risk and need based on a case finding methodology to support decision making and intervention [43] and utilizes an integrated approach rather than a conglomeration of instruments that cannot be compared and contrasted across the lifespan [25].

Overall, the ChYMH is part of a cohesive health information system, one that utilizes urgency algorithms upon data entry (i.e., categorizes risk level) and can be integrated into a more comprehensive assessment system with specific applications (e.g., care planning, quality assurance as well as resource allocation) for use across multiple stakeholders. Specifically, it provides algorithms to identify differential resource allocation and service intensity [25, 44,45,46,47]. This supports a “needs-based” care approach for planning protocols, outcome measurement, and quality indicators.

The screener can also be utilized to improve service usage to improve triaging. By identifying high-risk symptomology earlier, the screener supports more efficient identification of symptoms which is a key part of preventing childhood problems from developing into more complex, severe psychiatric disorders in later life. Further, by using consistent measurement strategies across service sectors, it can identify children who require a more in-depth assessment. Consequently, examining the efficacy and psychometric properties of the screener was an essential step in establishing its clinical and practical utility, and appropriateness for use in applied contexts. This kind of system is a uniquely helpful feature of the ChYMH, in that its components integrate into other interRAI assessments to support service system integration.

Finally, the use of the ChYMH over time allows for a consistent, standardized, and psychometrically sound assessment system [23, 24]. Therefore, as the use of the interRAI assessments grows, it will enable the creation of a large longitudinal database to examine the progression of child psychopathology into adulthood beginning in infancy. This kind of database could be incredibly useful in the development of personalized intervention strategies, as well as provide important information for service planning and resource allocation among agencies. Therefore, utilization of the brief screener across various service providers can serve to facilitate faster, more efficient triaging and prioritization, and allow for improved information sharing and service system integration both now and in the future.

Summary

Across North America, children suffer with various mental health concerns including depression, anxiety, and attention deficit and hyperactivity disorder. These issues are pervasive and can have both short- and long-term negative impacts on the child. However, only about 25% will receive intervention and the subsequent treatment and support that is so greatly needed. As mental health concerns continue to grow, emergency rooms and hospitals in Ontario continue to be inundated with patients; this increase in public health facilities use points to the need for a mental health assessment instrument that reliably evaluates and prioritizes children to appropriate services. interRAI is an organization of expert researchers and clinicians who develop integrated assessment systems that promote comprehensive and effective evaluation for various vulnerable populations. The interRAI ChYMH-S was designed as the preliminary step in the assessment process across various service sectors. This screener provides initial triaging and prioritization, which can then integrate into a more comprehensive assessment. The present study sought to empirically examine the psychometric properties of the ChYMH-S. In addition, authors explored the utility of revised short-form versions of both the Internalizing, Externalizing, and Disruptive/Aggressive Behaviour subscales. The results demonstrated strong scale reliability with inter-item correlations (i.e., Cronbach’s alpha) at 0.77 or higher across the measure. An examination of criterion validity demonstrated significant positive correlations in all hypothesized criterion scales, with correlations ranging from 0.459 to 0.766. Finally, the psychometric evaluation of the scale reliability for the revised short versions of the Internalizing, Externalizing, and Disruptive/Aggression Behaviour scales demonstrated moderate to high inter-item correlations, and Cronbach’s alpha at 0.82, 0.77, and 0.72 respectively. The results of this study provide preliminary evidence that the examined ChYMH-S scales demonstrate strong psychometric reliability and criterion validity. This provides evidence for the utility of the ChYMH screener for effectively identifying problematic and/or risky symptomatology in children. These findings are consistent with the larger body of research which examines the psychometric rigour of the full version of the ChYMH. Based on these findings, it is established that the screener can be utilized to facilitate improved triaging with the intent to improve access to more timely intervention. The utilization of the brief screener across service providers allows for more efficient triage and prioritization, while utilizing a common language to support continuity of care (e.g., transition into adult mental health care). The ultimate goal is to improve early identification and intervention with the goal of preventing childhood mental health issues from developing into more complex, severe psychiatric disorders.