Keywords

The Rating Scale of Impairment (RSI) (Goldstein & Naglieri, 2016) was developed to measure functional limitations across a range of life areas for youth ages 5 through 18 years. The RSI meets the need for a measure of impairment that can be used with symptom-based diagnostic tools as part of a comprehensive assessment. The RSI can be completed by a parent or a teacher. It yields scales measuring functioning in the following areas: school or work, social, mobility, domestic, family, and self-care. The RSI was developed to the highest psychometric qualities to provide clinically meaningful information that aids in treatment planning.

1 Uses of the RSI

1.1 Assessing an Individual

The RSI can be used during the assessment and diagnostic process of mental health and medical conditions providing information about an individual’s functional impairment in different life areas. Normative scores from the RSI allow the clinician to effectively compare an individual to a nationally representative group. Scores from the RSI can be integrated with other clinical, diagnostic, and medical information to provide a more complete understanding of a youth. When used in combination with other sources of information, results from the RSI help in guiding diagnostic decisions, developing treatment plans, and ongoing monitoring of treatment. The RSI can also be used to evaluate the effectiveness of treatment programs designed to improve a youth’s level of functioning in the identified areas of impairment, independent of a clinical diagnosis.

1.2 Screening a Group of Individuals

In some instances, clinicians may wish to obtain information about a group. For example, the RSI can be used to screen children or youth to identify those who might require additional assessments, or alternatively might benefit from additional support. High scores on the RSI suggest problems with the youth’s competence in meeting the demands of everyday functioning. Additional considerations are then required in this situation, such as a more thorough evaluation of the causes of the individual’s impairment, and/or intervention/treatment to improve the youth’s level of functioning.

1.3 Evaluating an Intervention Program

Results from the RSI can inform decisions about the effectiveness of a particular individual or group intervention. When used in a clinical setting, RSI results can be collected at the beginning of an intervention and at several points throughout the intervention in order to evaluate whether a particular program is associated with an improvement in the targeted area(s) of impairment. In research studies, group data from the RSI can be analyzed to determine whether change (pre- vs. posttreatment or experimental treatment vs. control group) is significant. Results from these types of evaluations can be helpful in supporting the continuation of a treatment program.

1.4 Use in a Research Context

The RSI can be used in a variety of settings for different research protocols. The RSI offers several advantages over other data collection methods. First, the scales were carefully developed to measure impairment across a comprehensive range of life areas, supported by the World Health Organization’s (WHO) International Classification of Functioning, Disability and Health (ICF) guidelines for measuring impairment (WHO, 2001). Second, the scales provide scores based on a nationally representative normative sample (ages 5–18 years) of a diverse group of youth. Third, the RSI possesses strong, well-documented psychometric qualities. Finally, the RSI is easily comparable to other instruments due to the use of standard scores.

2 RSI Form Options

The RSI can be used by parents and teachers of youth aged 5–18 years. For children (5–12 Years), the parent form (RSI [5–12 Years] Parent Form) includes 41 items, and a teacher form (RSI [5–12 Years] Teacher Form) includes 29 items. For adolescents (13–18 Years), the parent form (RSI [13–18 Years] Parent Form) includes 49 items, whereas the teacher form (RSI [5–18 Years] Teacher Form) includes 29 items. All scales are set to have a normative mean of 50 and a standard deviation of 10 (Fig. 12.1).

Fig. 12.1
figure 1

Overview of the RSI components

Full size image

3 Administration and Scoring Options

3.1 Paper-and-Pencil

All of the RSI forms are available in the MHS QuikScore™ format . The rater writes on the external layers of the form, and the results transfer through to a hidden scoring grid within the internal layers. The examiner then uses the internal layers to tabulate results. Each RSI QuikScore form includes conversion tables, which are used to convert raw scores to T-scores, percentile ranks, and classifications. For individuals who wish to use the MHS Online Assessment Center, users can print paper forms that do not include scoring pages.

3.2 Online

The RSI can be completed and automatically scored online wherever an internet connection is available. Paper-and-pencil forms can also be scored online by entering responses from a completed paper-and-pencil administration into the online program.

3.3 Report Options

RSI reports can be generated using the online scoring option. Three report types are available for all RSI forms. The Interpretive Report provides detailed results from one administration. The Progress Monitoring and Treatment Effectiveness Report provides an evaluation of RSI score changes over time for up to four administrations from the same rater. The Comparative Report provides an analysis of scores from two to five different raters.

4 Users and User Qualifications

The RSI is intended for use by professionals such as clinical psychologists and neuropsychologists, school psychologists, clinical social workers, physicians, school and community counselors, psychiatrists, and pediatric/psychiatric nurses. Professionals interpreting the RSI must possess appropriate qualifications (which require that, at a minimum, the professional has completed graduate-level courses in tests and measurements at a university or has received equivalent documented training), and must be familiar with the RSI manual and the Standards for Educational and Psychological testing developed by the American Educational Association, the American Psychological Association, and the National Council on Measurement in Education (AERA, APA, & NCMA, 1999). Users of the RSI should be members of professional associations that endorse a set of standards for the ethical use of psychological or educational tests, or be licensed professionals in the areas of psychology, education, medicine, social work, or an allied field. Although individuals who do not have advanced formal training in clinical psychology or psychometrics can administer and score the RSI by following the procedures outlined in this manual, interpretation should be conducted only by individuals with those qualifications described above.

4.1 Development

The development of the RSI encompassed 6 years of effort (April 2007 to August 2014), and include the three phases of conceptualization : initial planning and item writing, pilot study, and final scale construction and standardization, including the normative, reliability, and validity studies. The preliminary content was determined by a comprehensive review of current research literature, as well as the authors’ experience in the conceptualization and assessment of impairment. The content structure was then refined to correspond to key domains of functioning as identified by the World Health Organization’s International Classification of Functioning, Disability and Health (ICF; WHO, 2001). Items were developed to measure functioning in the following areas: Academic, Communication, Interpersonal, Mobility, Domestic, Organization, Mental and Physical health, and Self-Care. Separate items were created for the 6–13-year-old children and 13–18-year-old adolescents to account for developmental differences. Moreover, items related to behaviors not typically observed by teachers were not included on the teacher forms (i.e., domestic functioning, family interactions, socializing with friends/peers outside of school, ability to get around on one’s own). Where possible, items placed on both parent and teacher forms were identical.

Construction of the final scales began with the collection of the normative and clinical data. The normative samples include 2800 ratings—800 for each of the RSI (5–12 Years) Parent and Teacher Forms, and 600 for each of the RSI (13–18 Years) Parent and Teacher Forms. These samples included 50 males and 50 females at each age and are representative of the US population across several demographic variables. The clinical samples included 327 ratings of children/youths across the different normative samples, including 123 diagnosed with ADHD, 17 diagnosed with Autism Spectrum Disorder, 27 with a diagnosis of Depression or Anxiety Disorder, 24 diagnosed with Intellectual Disability Disorder, 96 with a diagnosis of Learning Disorder, and 40 with other disorders (e.g., Traumatic Brain Injury, Physical Disability [muscular-skeletal], and other disorders).

A series of factor analyses were performed on data from the normative and clinical samples for the demographic characteristics of the normative samples. Specifically, the normative and clinical samples were pooled together and split into two halves matched on age, sex, race/ethnic group, region, and clinical diagnosis, with the first half used for item-level exploratory factor analyses and the second half for parcel-level confirmatory factor analyses.

Exploratory and confirmatory factor analyses , based on this large epidemiologic sample, which include parent and teacher ratings, confirmed that the behaviors rated on the RSI represent a multidimensional construct resembling the structure of impairment proposed in the ICF (WHO, 2001). Specifically, a five-factor structure (School, Social, Mobility, Domestic, and Family) provided the best fit on the RSI (5–12 Years) Parent Form, six factors (School/Work, Social, Mobility, Domestic, Family, and Self-Care) for the RSI (13–18 Years) Parent Form, and three factors (School, Social, and Mobility) for the RSI (5–12 Years) Teacher and RSI (13–18 Years) Teacher Forms. The multidimensional factor solution of each RSI form persisted when tested across genders, age groups, race/ethnicities, and clinical status further supporting that the items on the RSI are best described as representing a multifaceted conceptualization of impairment.

Results of the factor analyses, coupled with practical considerations, guided the assignment of items to the RSI scales: School or School/Work (ten items), Social (ten items), Mobility (nine items), Domestic (seven items), Family (five items), and Self-Care (eight items) and resulted in creation of the final RSI forms. Forty-one items are included on the RSI (5–12 Years) Parent Form, 49 items on the RSI (13–18 Years) Parent Form, and 29 items on each of the RSI Teacher Forms. Inter-item correlations and Cronbach’s alpha values were calculated to evaluate the internal consistency of the RSI scales. The median inter-item correlation across scales on all forms was equal to .46, and Cronbach’s alpha values all fell above .75, indicating good internal consistency of the final set of items retained for each of the RSI Scales.

5 Reliability

Measurement error must be taken into account when observations are made during the assessment of human behavior. In classical terms, any observed score is a reflection of the true score of the attribute being measured, plus measurement error (Lord & Novick, 1968). Reliability is the counterpart to measurement error, and is defined as the consistency of measurements obtained by the instrument across populations or groups of individuals (AERA, APA, & NCME, 2014).

Internal consistency estimates demonstrate that the RSI Scale scores have excellent internal reliability. For the RSI (5–12 Years) versions, the median alphas were .85 and .89 respectively for the Parent and Teacher Forms in the normative samples, and .85 and .92 in the clinical samples. For the RSI (13–18 Years) versions, median alphas were .85 and .91 for Parent and Teacher Forms respectively in the normative samples, and .88 and .92 in the clinical samples. Internal consistency for the Total Scores was also excellent. The reliability estimates for the Total Score in the normative and clinical samples were all .94 or higher. In summary, the RSI Scale scores and Total Score all showed excellent reliability.

The stability of the RSI T-scores was evaluated by calculating the differences between Time 1 (pretest) and Time 2 (posttest) ratings. Inter-rater reliability refers to the degree of agreement between two raters. The average time interval was 2.3 days (SD = 5.0) for the RSI Parent Form and 4.4 days (SD = 5.8) for the RSI Teacher Form (range across forms = 0–31 days). As was done for the test–retest findings, data from the child and youth forms were analyzed together.

Substantial to almost perfect inter-rater agreement, according to the classification of Cicchetti et al. (2006), were found across all RSI scales for parent raters (corrected r ranged from .65 to .85), and moderate agreement was found across all RSI scales for teacher raters (corrected r ranged from .56 to .59). The inter-rater reliability of the Total Score was r = .87 between parents, and r = .77 between teachers. These findings are comparable (and in the case of parent ratings, superior) to the average inter-rater reliability of .60 across studies reported by Achenbach and McConaughy (1987). The median values for Cohen’s d for the RSI scales among the parent and teacher inter-rater samples were 0.10 and 0.08, respectively, and the values of Cohen’s d for the Total Scores ranged from 0.11 to 0.13 across the inter-rater samples, showing negligible rater effects across administrations.

The consistency between raters was evaluated by calculating the difference between T-scores for Rater 1 and Rater 2. Results suggest that scores on the RSI possess good consistency between parent raters; for the RSI scales, between 70 and 89 % of the differences across scales fell within one standard deviation (i.e., +/− 10 T-scores). Good levels of consistency were also found for teacher raters, with 71–73 % of the differences falling within +/− 10 T-scores. For both parent and teacher ratings, the mean differences were close to 0, providing further evidence for inter-rater consistency. Further these results suggest that RSI scores have excellent stability; for the RSI scales and Total Score, over 89 % and 81 % of the differences on the Parent and Teacher Forms respectively fell within +/− 10 T-scores (i.e., one standard deviation). The mean differences were very close to zero, supporting the stability of the RSI across administrations.

6 Validity

Validity is described as “what the test measures and how well it does so” (Anastasi & Urbina, 1997, p. 113). The preliminary content structure of the RSI was determined through a comprehensive review of current research literature, as well as the authors’ clinical experience on the conceptualization and assessment of functional impairment. The content of the RSI is also consistent with the structure of the World Health Organization’s International Classification of Functioning (ICF; WHO, 2001). According to the ICF, a societal view of functioning refers to the individual’s ability to participate in life activities, with major life areas broken down into several broad domains: education; learning and applying knowledge; communication; interpersonal interactions and relationships; community, social, and civic life; mobility; self-care; and domestic life. Multiple items were developed for the RSI to assess behaviors from the key domains of the ICF as they apply to impairment in youth. These items were organized into six scales. For example, the ICF education and knowledge domains are measured in the items from the RSI School/Work scale. Likewise, the ICF domains of communication, interpersonal interactions and relationships, and community and social life are reflected in the items of the RSI Social and Family scales. The remaining ICF domains correspond to the RSI scales of Mobility, Self-Care, and Domestic impairment. These six content areas measured by the RSI are intended to cover a wide range of observable indicators related to the general concept of impairment, as well as to more specific areas of impairment in everyday life functioning.

To evaluate the criterion-related validity of the RSI, mean differences in the RSI scores between the general population and samples of children/youths previously diagnosed with specific clinical disorders were examined. The mean differences between the general population and samples of children/youths with an increasing number of diagnoses were also examined. To further evaluate the criterion-related validity of the RSI scoring and interpretation methods, correlations between RSI scores and scores from other measures were examined. Overall, results from these analyses provide strong evidence for the criterion-related validity of the RSI.

The moderate correlations between raters, coupled with negligible effect sizes, provide support for the construct validity of the RSI. However, the correlations were only moderate suggests that ratings collected from different types of raters (i.e., parents and teachers) are not redundant. Instead, as Achenbach and McConaughy (1987) point out, the moderate correlations between raters of different types indicate that each type of rater accounts for some unique variance that is not captured by other types of raters. Therefore, obtaining information from multiple sources is important.

6.1 Summary and Implications

The RSI operationalizes the construct of impairment. The RSI is strongly correlated with other measures of impairment such as the Barkley Functional Impairment Scale—Child and Adolescent (Barkley, 2012) and the Children’s Global Assessment Scale (Shaffer et al., 1983). In addition, the RSI is correlated with scores from the Adaptive Behavior Assessment System—Third Edition (Harrison & Oakland, 2015). This suggests that there is some similarity, but also difference, in the behaviors assessed by these measures. Perhaps most importantly, the RSI correlates the greatest with the Comprehensive Executive Function Inventory (Naglieri & Goldstein, 2013) and the Devereux Student Strengths Assessment (LeBuffe, Shapiro, & Naglieri, 2014). Although these two rating scales may seem different, as noted by Goldberg (2009), the concept of executive function with its association with the frontal lobes is the foundation of social-emotional behaviors. Both of these rating scales, therefore, provide the means by which humans meet the demands of everyday life, especially the social demands —which the RSI also addresses. The low correlations between the RSI and intelligence, neurocognitive abilities, and achievement as well as the personality scales suggest that the RSI adds unique information that is not obtained from these measures. The lack of correlation between the RSI with the Wechsler Intelligence Scale for Children, Cognitive Assessment System-2 (Naglieri, Das, & Goldstein, 2014) and the Woodcock Johnson IV: Tests of Achievement (Schrank, McGrew, & Mather, 2014) illustrates that the constructs measured by these tests are independent. Perhaps most importantly, the modest correlation between the RSI and Conners Comprehensive Behavior Rating Scales (Conners, 2014) illustrates that symptoms and impairment are not strongly related, supporting the need to include measures of impairment in all assessment batteries. Similarly, the only modest correlations found between the RSI with informal evaluation of impairment by clinicians argue strongly for a psychometrically sound tool to evaluate impairment.