A key component of assessing, understanding, and delivering services to improve child and adolescent (hereafter referred to collectively as “youth” unless otherwise specified) mental health involves estimating the degree to which mental health concerns result in life interferences or psychosocial impairments (American Psychiatric Association [APA] 2013). Among youth who experience clinically elevated mental health concerns, psychosocial impairments may manifest in various ways, including dysfunctional family relationships or interactions (e.g., family conflict), maladaptive relationships with peers, poor school performance, and difficulties with fulfilling home obligations (e.g., completing chores and homework) (Fabiano and Pelham 2016). One can distinguish psychosocial impairments from related constructs such as adaptive functioning, which captures the degree to which one exhibits the requisite skills for coping with life interferences (e.g., Liss et al. 2001; Volkmar et al. 1987). The identification of psychosocial impairments often dictates such crucial elements of care as diagnostic decision-making and determining whether a course of treatment brought about sufficient change to ameliorate presenting concerns (Goldstein and Naglieri 2016).

In recent decades, researchers focused a great deal of their attention on developing and evaluating measures of mental health symptoms for use in youth mental health services. However, a long-standing concern involves the relative lack of attention paid to developing and testing the psychometric properties of measures of psychosocial impairments (Gadow et al. 2013; Rapee et al. 2012). The lack of firm evidentiary bases for measures of psychosocial impairments has important clinical implications. For instance, use of impairment measures that yield scores of questionable validity may result in diagnosing concerns for youth who do not warrant the diagnoses, or failing to identify diagnosable concerns for youth who require care. Use of psychosocial impairment measures that yield scores that lack reliability may impede a provider’s ability to track changes in impairment over time, or a researcher’s ability to test whether a treatment brought about change in research contexts. In fact, methodological issues surrounding the use and interpretation of currently available measures of psychosocial impairments point to the need for new instruments.

The most commonly used measures of impairment involve a single-item clinician rating. This includes the Children’s Global Assessment Scale (CGAS; Shaffer et al. 1983) and Global Assessment of Functioning (GAF; APA 2013), which measure youth impairments on a single-item scale that ranges from 1 to 100. Both measures have the advantage of brevity and the ability to assess psychosocial impairments among youth of varying clinical presentations or presenting concerns. However, these measures suffer from the issues inherent in single-item measures (Tabachnick and Fidell 2001), including wide variability in both reliability estimates and evidence supporting validity of scores taken from these measures (Blake et al. 2007; Francis et al. 2012; Schorre and Vandvik 2004).

Relative to single-item measures, multi-item and multi-dimensional measures provide richer information about various domains of psychosocial impairments relevant to understanding youth clinical presentations. However, currently available measures also display significant methodological limitations. For instance, many commonly used measures of psychosocial impairments assess impairment specific to a single condition or class of conditions. This format characterizes impairment indices collected as part of diagnostic interviews such as the Anxiety Disorders Interview Schedule for Children and Adolescents (ADIS-CA; Silverman and Albano 1996) and the Diagnostic Interview Schedule for Children (DISC; Costello et al. 1984), in which impairment items sequentially follow symptom items administered within modules for specific diagnoses. For example, on the ADIS Social Anxiety Disorder (SAD) module, the assessor asks parents the following after endorsing SAD symptoms: “Now I want to find out how much you feel this problem (i.e., SAD symptoms) interferes with your child’s life. That is, how much has it interfered with your child’s friendships, caused problems at school or at home, and stopped your child from doing the things he or she would like to do?”

Condition-specific assessment formats do not allow for distinguishing symptoms of a mental health condition from psychosocial impairments. This feature limits the ability of assessors to link any one mental health domain to the impairments thought to result from it. Importantly, mental health concerns and psychosocial impairment represent distinct constructs that are only moderately correlated (Allen et al. 2010; Storch et al. 2010). In fact, diagnostic thresholds do not consistently differentiate youth on levels of psychosocial impairments (Egger et al. 2006; Keenan et al. 2010). Further, youth often present for mental health services with multiple concerns (e.g., Merikangas et al. 2010). When taken together, these issues make it difficult to understand psychosocial impairments independent of any single condition. Indeed, key trends in the field involve developing services for addressing symptoms and impairments resulting from multiple co-occurring concerns (e.g., Ehrenreich et al. 2009; Weisz et al. 2012), and characterizing mental health using domains that cut across diagnostic boundaries (e.g., Caspi et al. 2014; Sanislow et al. 2010).

An additional concern with existing psychosocial impairment measures is that, in assessing impairment in the context of existing concerns, probes about impairment essentially result in use of leading questions about negatively worded symptoms. For example, the Impairment Rating Scale (IRS: Fabiano et al. 2006), which contains parent- and teacher-report versions, is a 6- to 7-item measure of multiple domains of youth functioning (e.g., relationship with peers, academic progress, self-esteem). On the IRS, informants first provide a qualitative description of the youth’s “primary problems” and then rate their severity, as well as the need for treatment or special services resulting from the problems. Items on the IRS instruct informants to consider the extent to which the youth’s problems result in impairment in specific domains (e.g., “How your child’s problems affect his or her academic progress at school”). This measurement format also characterizes individual impairment measures (e.g., Child Sheehan Disability Scale [CSDS]; Whiteside 2009) and impairment measures in commonly used diagnostic interviews (e.g., ADIS, DISC). To be sure, a strength of this format is that it allows for the tailoring of impairment ratings to an individual youth’s presenting concerns. However, use of leading questions leads to memory recall biases (Brainerd et al. 2008), and this format does not allow an assessor to probe the possibility that a youth can behave in such a way as to result in impairments without the behavior itself reflecting clinically elevated mental health concerns.

Given the measurement issues outlined above, there is a pressing need for a brief, multi-item measure of youth impairment that can be completed by untrained or lay informants (i.e., youth and their parents/teachers), thus paving the way for wide dissemination (see also Beidas et al. 2015). Further, there is a need for impairment measures that informants can complete about youth, regardless of the youth’s mental health status; including circumstances in which youth do not display clinically elevated mental health concerns. This second element might greatly facilitate identification of psychosocial impairments among youth for whom such impairments are a precursor to, as opposed to a consequence of, mental health concerns. The development of such a measure would dovetail with emerging trends in the field, namely the focus on evidence-based techniques that can be applied outside the confines of traditional diagnostic categories (e.g., Kotov et al. 2017; Sanislow et al. 2010; Weisz et al. 2012).

As a first step to addressing limitations in the assessment of impairment, we consulted the literature and identified a widely used measure of adult impairment that, with some modification, could address the fundamental measurement issues noted previously. The Work and Social Adjustment Scale (WSAS; Mundt et al. 2002) is a 5-item self-report measure of impairment in various domains (i.e., work, social life, home life, private life, close relationships). Prior work indicates that among adults, scores from the five items load onto a single factor of general disability (Mataix-Cols et al. 2005). The WSAS appears in hundreds of investigations across various clinic and non-clinic populations (Web of Science search conducted July 28, 2018). Further, the WSAS demonstrates sensitivity to treatment response and can assess naturalistic changes in psychosocial impairments over time (i.e., in the absence of treatment) (e.g., Hussain et al. 2011; Kenwright et al. 2005; Kristensen et al. 2015; Zahra et al. 2014).

Although the original WSAS addresses some of the limitations of psychosocial impairment measures noted previously, the wording of the items still links impairments to mental health concerns (e.g., “Because of my [disorder], my ability to work is impaired.”; see Mundt et al. 2002). Thus, a key aim of the current study was to test the psychometric properties of an adapted version of the WSAS for use with youth. Specifically, we modified the item content so that we could assess psychosocial impairment domains germane to youth (e.g., completing chores and homework). We also modified item content so that the items assessed psychosocial impairments resulting from behavior generally, not mental health concerns in particular (see Appendices A and B). This modified version of the WSAS allows for assessing impairments among youth, regardless of their mental health status.

The purpose of the present study is to advance the literature on evidence-based assessment of youth psychosocial impairments, using an adapted version of the WSAS for Youth (i.e., WSASY). We tested three hypotheses in a sample of adolescents whose parents contacted our laboratory to participate in either a clinical evaluation of their adolescent’s mental health (evaluation-seeking adolescents) or a non-clinic study about adolescent-parent relationships (community control adolescents). The inclusion of both adolescents evaluated for clinical concerns and non-clinic adolescents from the community resulted in a pooled sample that we expected to display not only wide variability in psychosocial impairments, but also psychosocial impairments that varied as to their links to mental health concerns. Further, in this sample the parents of evaluation-seeking adolescents sought assessments for their adolescents’ social anxiety concerns. Adolescent social anxiety often co-occurs with a number of mental health difficulties including depression and attention-deficit/hyperactivity disorder (ADHD) (e.g., Epkins and Heckler 2011; Jarrett and Ollendick 2008; Kessler et al. 2012). Further, social anxiety, depressive, and ADHD symptoms each often result in similar psychosocial impairments, including maladaptive family and peer relationships (e.g., APA 2013; Pelham et al. 2005; Silverman and Ollendick 2005). Thus, this sample provided us with an opportunity to test a number of transdiagnostic hypotheses regarding the psychometric properties of adolescent and parent reports on the WSASY.

First, we expected scores from the WSASY to display relatively high internal consistency estimates for a short, 5-item scale (i.e., α’s ≥ .80; see Nunnally and Bernstein 1994; Ryan et al. 2001). Second, we expected scores from the WSASY to display evidence of convergent validity in relation to scores from measures reflecting a broad array of mental health domains. That is, we expected the WSASY to index impairment in a broad sense and not specific to any particular mental health concern displayed by adolescents. Additionally, various aspects of family and parent functioning act as both domains of psychosocial impairments and outcomes of such impairments (e.g., conflict between adolescents and parents; parent mental health concerns; Drabick and Kendall 2010). That is, when youth experience mental health concerns, these concerns also impact the parent (Granic and Patterson 2006). In fact, when evidence-based mental health treatments targeting youth functioning successfully ameliorate their concerns, their parents’ psychosocial functioning may also improve (e.g., Kazdin and Wassell 2000). Thus, we expected scores from the WSASY to relate to scores from measures of (a) adolescent internalizing and externalizing concerns, (b) family functioning, and (c) parent functioning.

For our third hypothesis, we expected scores from the WSASY to display evidence of criterion-related validity, namely with regard to co-occurrence of mental health concerns and domains of psychosocial impairments. For these tests, we consulted the literature on associated features of elevated psychosocial impairments among youth and identified three questions of interest in the present study. Specifically, relative to a single concern or no concerns, the presence of co-occurring mental health concerns tends to signal greater levels of impairment (Rapee et al. 2012). Further, psychosocial impairments among youth often manifest as maladaptive social interactions with same-age peers, as well as social skills deficits within these interactions (APA 2013; Epkins and Heckler 2011; Pelham et al. 2005). Thus, we expected scores from the WSASY to (a) increase as a function of number of co-occurring mental health concerns and number of peer-related impairments, and (b) relate to adolescents’ observed social skills in interactions with unfamiliar peers.

Method

Participants

We recruited a sample of 105 14-15-year-old adolescents and their parents from the areas of Maryland, Washington, D.C., and Northern Virginia. Families recruited for this study had to (a) be fluent in English; (b) have a 14 to 15-year-old adolescent who could read at or above their grade-level, did not have any learning or developmental disabilities, and had not received any cognitive behavioral therapy for anxiety or any other related concerns in the three months prior to the phone screening; and (c) understand the consent/assent process. Following participation in the study, we provided parents of evaluation-seeking adolescents with feedback on whether their adolescent displayed clinically significant levels of social anxiety, mood levels, and/or ADHD symptoms, and referrals to services that could address these concerns. Parents of community control adolescents were not given feedback about their adolescents’ mental health.

Within the total sample, we recruited 37 evaluation-seeking adolescents and 68 community control adolescents. These 105 adolescents had a mean age of 14.47 years (SD = 0.50) and included 68 female and 37 male participants. The participating parent identified the adolescent’s racial/ethnic background as African American or Black (64%); White, Caucasian American, or European (33%); Asian American or Asian (6%); Hispanic or Latino/a (Spanish) (11%); American Indian (1%); or “Other” (10%) (rates total above 100% because parents could select multiple response options). Parents reported that 30 of the families earned $500 or less per week, 25 earned between $501 and $900 per week, and 50 earned more than $901 in income per week.

Parents were the adolescent’s biological mother/father (94%), adoptive mothers/fathers (3%), stepmothers/fathers (1%), primary caregiver’s significant other (1%), or the adolescent’s part-time guardian (1%). Parents reported their marital status as currently married (44%), never married (24%), divorced from a significant other (17%), separated from a significant other (9%), living with a significant other (5%), or widowed (1%). Our sample’s demographic figures are consistent with economic and racial/ethnic data for the geographic area of recruitment (U.S. Census Bureau 2016).

For the multiple aims tested below, we used an analytic approach that pooled the evaluation-seeking and community control groups as one sample. Prior work suggests that this recruitment approach results in a dimensionally varied sample of adolescents enriched for displays of and risk for various mental health concerns (e.g., De Los Reyes et al. 2012a; Beale et al. 2018; Glenn et al. 2018). Further, prior work demonstrates that these evaluation-seeking and community control groups are comparable (i.e., non-significantly different) on the demographic characteristics reported previously (see Deros et al. 2018; Karp et al. 2018; Rausch et al. 2017). Demographic data for the two groups are available upon request from the corresponding author.

Procedure

All procedures were approved by the Institutional Review Board of the large Mid-Atlantic university at which we conducted the study. We recruited participants through a variety of strategies, including advertisements posted online (e.g., Craigslist, laboratory website), on public transportation servicing the university and community, and on local advertisement boards. We also recruited through the offices of local clinicians serving our targeted demographic. Parents completed an initial screen for eligibility over the telephone and were subsequently scheduled to complete an assessment in our laboratory. After receiving parental consent and adolescent assent for participation in the study, dyads participated in an in-person assessment and independently completed a counterbalanced battery of survey measures on computers using Qualtrics Survey data collection software. Adolescents also participated in a series of social interaction tasks. Families received $100 compensation (i.e., parent: $50; adolescent: $50), and were debriefed on study activities, including study deception (e.g., that research personnel involved in the social interaction tasks were trained to act as same-age peers).

Survey Measures

We administered a multi-informant survey battery to address our research questions. As part of this battery, parents completed a demographics form to collect the adolescent, parent, and family demographic information described previously.

Adolescent and parent survey measures

Adolescents and parents completed a battery of survey measures. Surveys assessing adolescent and family functioning were parallel such that the item content fit the informant’s perspective (e.g., “I” for an adolescent self-report vs. “My child,” for the parent report version of the same measure). Surveys assessing parent functioning were strictly self-report. Tables 1 and 2 provide internal consistency estimates for all survey measures.

Table 1 Means (M), standard deviations (SD), and internal consistency (α) estimates of survey measures of adolescent and parent psychosocial functioning
Full size table
Table 2 Descriptive statistics for adolescent and parent reports on the work and social adjustment scale for youth
Full size table

Adolescent psychosocial impairment

We assessed adolescent impairment using a modified version of the WSAS (Mundt et al. 2002), as described previously. Our version of the WSASY consisted of five items, which we present in Appendices A (adolescent) and B (parent). Adolescents and parents provided ratings on their respective versions of the WSASY on a scale of “0” (Not at all impaired) to “8” (Very severely impaired). Total scores could range from 0 to 40, with higher scores indicating greater impairment. As noted in our Appendix, we created measure instructions that prompted informants to rate adolescent impairment resulting from the adolescent’s behavior, without mention of mental health concerns. In this way, adolescents and parents in our sample could provide reports about adolescent impairment on the WSASY, regardless of the adolescent’s mental health status.

Adolescent internalizing concerns

To assess adolescent mental health concerns, we administered several well-established survey measures of both internalizing and externalizing domains. First, we assessed adolescent social anxiety and depressive symptoms with the Social Phobia and Anxiety Inventory (SPAIC; Beidel et al. 1995) and Beck Depression Inventory-II (BDI-II; Beck et al. 1996). The SPAIC is a widely used 26-item measure in which each item describes a social situation and the respondent endorses how often the adolescent feels nervous or scared in that situation (e.g., “I feel scared when I meet new kids”). The three response choices range from “0” (Never) to “2” (Always). Total scores range from 0 to 52, with higher scores reflecting higher levels of social anxiety. Depending on the aim of the analysis, we examined either SPAIC continuous scores or discrete scores based on established cut scores on this measure to identify clinically elevated social anxiety (i.e., scores of 18 or above; Beidel et al. 1995). The BDI-II is a commonly used 21-item measure in assessments of depressive symptoms, and was originally designed to measure “the severity of depression in adults and adolescents aged 13 years and older” (Beck et al. 1996, p. 1). Respondents were asked to rate items describing symptoms of depression (e.g., sadness, guilty feelings, loss of interest) on a 4-point scale, with possible score ranges from 0 to 63 and higher scores indicating greater depressive symptoms. We excluded item 9, which assesses for suicidal thoughts and actions, and item 21, which inquires about loss of interest in sex. Consistent with prior work (e.g., Lipton et al. 2014; Rausch et al. 2017; Thomas et al. 2012), we did not administer these two items given the mature nature of the item content and because parents in our studies often decline to consent to having their adolescents respond to items that assess these behaviors. Internal consistency estimates of the 19 items administered nonetheless demonstrated high internal consistency (Table 1). As part of this approach and to ensure comparability with scoring for the full version of the measure (i.e., possible score ranges from 0 to 63), responses for items 9 and 21 were pro-rated or estimated for each participant, based on their mean score for the 19 remaining items. For tests based on discrete scores, we used the established cut score for identifying “mild depressive symptoms” (i.e., scores of 14 or above; Beck et al. 1996). Both of these measures have been used extensively to assess adolescent internalizing concerns (Byrne et al. 2004; Glenn et al. 2018; Lipton et al. 2014; Osman et al. 2008).

Adolescent externalizing concerns

Second, we assessed adolescent attention and hyperactivity concerns and callous-unemotionality with the ADHD Self-Report Scale (ASRS) (Kessler et al. 2007) and Inventory of Callous-Unemotional Traits (ICU; Frick 2004). The 18-item ASRS measures inattentiveness and hyperactivity using a 5-point scale ranging from “0” (Never) to “4” (Very often). The ASRS displays acceptable estimates of test-retest reliability (Kessler et al. 2007). We administered the first six items of the ASRS (i.e., ASRS-6). Prior work indicates that these six items are most predictive of clinically relevant ADHD concerns (Kessler et al. 2007), and recent work supports the psychometric properties of these items when administered to adolescents during social anxiety assessments (Beale et al. 2018; Keeley et al. 2018). In this study, we examined either ASRS-6 continuous scores or discrete scores based on established cut scores on this measure to identify clinically elevated ADHD symptoms (i.e., scores of four symptoms or above in the clinical range; Kessler et al. 2007).

On the ICU, informants rated items on a four-point scale ranging from “0” (Not at all true) to “3” (Definitely true), with higher scores reflecting greater callous-unemotionality. Example items on the ICU include “I do not care who I hurt to get what I want,” “I do not care about being on time,” and “I do not care if I get into trouble.” For the current study and consistent with current scoring recommendations we used the total score of 22 items. Prior versions of the ICU consisted of scores ranging from 0 to 72 (e.g., Kimonis et al. 2013). However, we modified scoring procedures for the current study to reflect updated scoring recommendations. Specifically, prior research has suggested two items from the ICU (i.e., item 2, item 10) fail to load onto a three-factor model, supporting the exclusion of these two items from further analysis (e.g., Essau et al. 2006; Kimonis et al. 2008). Consequently, only 22 of the 24 items were included in analysis, resulting in a maximum total score of 66. The ICU has extensive evidence attesting to its internal consistency and validity when assessing clinical, community, and incarcerated adolescents (e.g., Essau et al. 2006; Kimonis et al. 2008, 2013).

Identifying adolescents above clinical cut scores on mental health measures

For one of our hypotheses, we were interested in testing the relation between scores from the WSASY and the extent to which adolescents scored above the clinical cut scores of our adolescent mental health surveys (i.e., SPAIC, BDI-II, ASRS-6). We calculated this number separately by each informant. For example, for adolescent-reported surveys, the possible range of measures that could be above the clinical cut score was 0–3 (i.e., 3 = above the clinical cut score on SPAIC, BDI-II, and ASRS-6). We grouped adolescents by the number of measures for which they scored above the clinical cut score: “0” measures, “1” measure, or “2 or more” measures (Table 3). We created an adolescent-reported grouping and a parent-reported grouping. We also created dichotomous groupings based on whether the adolescent scored above clinical cut scores on 2 or more measures, again one based on adolescent report and another based on parent report.

Table 3 Frequencies (N) and percentages (%) of adolescents above clinical cut scores based on survey measures of adolescents and parents
Full size table

Adolescent–parent conflict

To assess adolescent-parent conflict, adolescents and parents completed the widely used Issues Checklist (Prinz et al. 1979). On the Issues Checklist, adolescents and parents report on topics of which they have disagreed in the past 4 weeks. We modified the measure for the purposes of time (i.e., to reduce participant burden) and to assess ranges of conflict related to topics about which parents and youth typically encounter at home (e.g., chores, homework, and friends), consistent with prior work (e.g., De Los Reyes et al. 2012b; Rausch et al. 2017; Treutler and Epkins 2003). Specifically, our modified checklist included 16 of the 44 topics listed on the original measure. A list of the 16 topics we assessed is available from the corresponding author. We also modified the response format so that participants, working independently, could rate conflict about each topic using a 5-point Likert-type scale ranging from “1” (Do not disagree) to “5” (Disagree much). Informants completed the checklist about the person with whom they participated in the study (i.e., parent about adolescent and adolescent about parent). We calculated total scores by summing the scores across the 16 items, with possible total scores ranging from 16 to 80. The psychometric properties of the Issues Checklist used in this study and evidence of its reliability and validity have previously been reported elsewhere (De Los Reyes et al. 2012b; Rausch et al. 2017).

Parent depressive symptoms

We assessed parent depressive symptoms using the BDI-II as described previously. As with the administration of the BDI-II with adolescents, we omitted administration of items 9 and 21. As seen in Table 1 and similar to the use of the measure with adolescents, parent self-reports displayed high internal consistency.

Parent emotion reactivity

We assessed parents’ emotion reactivity using the Emotion Reactivity Scale (ERS; Nock et al. 2008). This 21-item self-report measure assesses three subcomponents of emotion reactivity: sensitivity (eight items; example item: “I tend to get emotional very easily”), arousal/intensity (ten items; example item: “When I experience emotions, I feel them very strongly/intensely”), and persistence (three items; example item: “When I am angry/upset, it takes me much longer than most people to calm down”). For this study, we used the total score of all items, with item response options ranging from “0” (Not at all like me) to “4” (Completely like me), and higher scores indicating greater emotion reactivity. The ERS displays high internal consistency, convergent validity, and divergent validity (Evans et al. 2016; Glenn et al. 2011; Nock et al. 2008).

Parent quality of life

We assessed parent quality of life using the Quality of Life Enjoyment and Satisfaction Questionnaire-Short Form (Q-LES-Q-F; Endicott et al. 1993). The Q-LES-Q-SF is a 16-item self-report inventory on which respondents rate items on a five-point scale (from “Very poor” to “Very good”) based on quality of life in the past week in reference to 16 life domains (example domains: economic status, living/housing situation, work). Higher scores indicate a higher level of quality of life. As in prior work (e.g., Martz et al. 2018), we computed total scores on the Q-LES-Q-SF based on the first 14 items of the measure. We excluded item 15 (i.e., respondents can leave the item blank if they are not taking medications) and item 16 (i.e., assesses overall quality of life which is redundant with taking a total score of the first 14 items). The Q-LES-Q-SF demonstrates adequate internal consistency and test-retest reliability, as well as construct and discriminant validity (Endicott et al. 1993).

Adolescent Peer-Related Impairments

During the initial parent-completed phone screen, research personnel asked three questions from the Interpersonal Relationships Module of the ADIS-CA (Silverman and Albano 1996). These questions focused on number of friends relative to same-age peers, trouble with making friends, and trouble with keeping friends. Data on these items were available for 98 families in this sample. As in prior work (Beale et al. 2018), we grouped adolescents by number of parent-endorsed peer-related impairments (i.e., positive response for “number of friends” item = “fewer friends”; positive response for “making friends” item = “yes”; positive response for “keeping friends” item = “yes”), yielding three groups: “0” peer-related impairments (n= 47); “1” impairment (n= 20); or “2 or more” impairments (n= 31). This measure of peer-related impairments distinguishes youth on diagnostic and evaluation-seeking status, and the number of co-occurring mental health concerns (Beale et al. 2018; Scharfstein et al. 2011).

Behavioral Measures

Adolescents participated in a series of counterbalanced tasks with a total duration of approximately 20 min. These tasks included a Simulated Social Interaction Test (SSIT; adapted from Beidel et al. 2000), Unstructured Conversation Task (UCT; adapted from Beidel et al. 2010), and Impromptu Speech Task (IST; adapted from Beidel et al. 2010). Extensive descriptions of these tasks are available elsewhere (e.g., Deros et al. 2018; Glenn et al. 2018). Within these tasks, adolescents interact in a series of situations with undergraduate research assistants who we trained to pose as 14- to 15-year-olds. We masked these unfamiliar peer confederates to adolescents’ evaluation-seeking status and all other clinical information, and they had no contact with participants prior to the tasks.

The SSIT consists of a series of five role-plays between an adolescent and a gender-matched peer confederate (i.e., offering/accepting assistance, giving/receiving a compliment, and responding to inappropriate behavior). In the UCT, adolescents participate in a 3-minute role-play conversation with a peer confederate, with the instruction that the role-play calls for them to pretend that they are at a new school and do not know anyone. Peer confederates responded neutrally to the participant and allowed for the participant to lead the conversation. In the IST, adolescents deliver a speech to unfamiliar peers about topics not often discussed by adolescents (i.e., politics, public health). The audience consisted of the task administrator and two trained confederates with whom the adolescent had no prior contact before the beginning of the tasks. Adolescents delivered a 10-minute speech following a 3-min period of preparation. If after 3 min an adolescent wished to terminate their speech, we permitted them to do so.

Independent observers’ ratings about adolescent social skills

Independent observers received training on how to use the behavioral ratings of adolescent social skills described below. These observers consisted of undergraduate and post-baccalaureate research assistants. As with the peer confederates described previously, we masked observers to adolescents’ clinical status and they did not have access to adolescents’ clinical information. Further, none of the independent observers participated as a peer confederate in any of the social interaction tasks described previously. Independent observers made ratings using an extensively validated behavioral coding scheme to rate adolescents’ observed social skills (e.g., Beidel et al. 2000, 2010; Glenn et al. 2018; Scharfstein et al. 2011). For each adolescent and across the seven tasks, independent observers made macro-level ratings of social skills on a 5-point scale ranging from “1” (Not effective at all) to “5” (Very effective), with greater scores indicating greater social skills. The ICC’s (for average measures) testing inter-rater reliability for observers’ ratings displayed an average ICC(1,2) of .82. This average ICC is considered within the “excellent” range, based on thresholds recommended by Cicchetti (1994). Details regarding coder training and reliability procedures are available elsewhere (Glenn et al. 2018).

For any one adolescent, two independent observers rated their social skills, and thus we created composite scores for all seven social skills ratings by taking an average of the two independent observers’ ratings for each task. The internal consistency estimates for the seven social skills ratings was high, α = 0.91. Thus, to reduce Type I Error we aggregated the seven social skills ratings into a single mean social skills rating (M = 3.43; SD = 0.89). We used this composite rating for the criterion-related validity tests reported below. We computed this composite score for all 105 adolescents. Among these adolescents, one adolescent was missing data on one of the five SSIT role plays, and three adolescents were missing data on the IST because they declined to give a speech. For these adolescents their composite scores were based on six social skills ratings.

Data-Analytic Plan

We followed a three-step data-analytic plan. First, our preliminary analyses determined if our data met assumptions of parametric tests (i.e., skewness/kurtosis in range of ±2.0). We also computed Cronbach’s alpha (α) estimates to test the internal consistencies of all survey measures. In our preliminary analyses, we focused on computing α estimates and descriptive statistics for adolescent and parent WSASY reports, as well as Pearson r correlations to examine adolescent-parent correspondence between total scores on their WSASY reports.

Second, we tested convergent validity by computing Pearson r correlations among each informants’ WSASY total scores and the scores from their reports of various impairment and mental health concern domains. For adolescents, we computed correlations between their WSASY total scores and total scores from self-reports of adolescent mental health and adolescent-parent conflict. For parents, we computed correlations between their WSASY total scores and total scores from parent reports of adolescent mental health, adolescent-parent conflict, parent mental health, parent emotion reactivity, and parent quality of life.

Third, we took a multi-method approach to testing criterion validity. With regard to links between WSASY scores and co-occurring mental health concerns, we considered examining links between WSASY scores and continuous scales of mental health concerns. However, as seen in Table 3, the sample displayed wide variability in the nature of co-occurring concerns, with the grand majority of adolescents displaying co-occurring concerns in the form of social anxiety plus an additional clinically elevated domain (i.e., depressive or ADHD symptoms). We also sought to test links between the WSASY and co-occurring concerns using analytic models that most closely matched how one might use impairment measures in assessment contexts.

Thus, we conducted two analysis of variance (ANOVA) analyses, one per informant. In each ANOVA, we examined differences in levels of psychosocial impairment (i.e., WSASY total scores) among the groups of adolescents displaying scores above clinical cut scores on established mental health surveys described previously (i.e., 0, 1, 2 or more). For both ANOVAs, we conducted follow-up univariate comparisons. For these comparisons, we were interested in comparing the adolescent group elevated on 2 or more surveys relative to the other two groups. Thus, these tests were directional in nature such that in each comparison, we expected the reference group to display greater impairment levels relative to the two other groups. Thus, we conducted directional, univariate tests using the Dunnett t test statistic. We chose this test because it implements a consistent control group, thus minimizing Type I error and providing increased statistical power, relative to tests of comparisons of all possible group pairs. As an additional test of links among co-occurring concerns and WSASY scores, we computed Area Under the Curve (AUC) statistics to determine optimal cut scores on the WSASY for identifying adolescents displaying 2 or more elevations on cut scores versus 1 or 0 elevations. We followed procedures from Jarrett et al. (2018) and identified WSASY cut scores based on AUC values of 0.90 for maximal sensitivity and specificity.

With regard to links between WSASY scores and peer-related impairments, we followed prior work indicating that adolescents displaying 2 or more peer-related impairments could be distinguished from those adolescents with 0 peer-related impairments on various mental health surveys (Beale et al. 2018). In line with this work, we conducted two independent samples t tests, one per informant, in which we compared the WSASY scores of adolescents displaying 2 or more peer-related impairments to those displaying 0 peer-related impairments. We also reported independent samples t tests of all other possible comparisons (2 vs. 1; 0 vs. 1).

We computed Pearson r correlations to test a third form of criterion-related validity, namely the links between WSASY scores and independent observers’ ratings of adolescent social skills during the social interactions described previously.

For all tests, we interpreted statistical significance using a p-value threshold of <0.05. We inferred magnitudes of effect sizes based on Cohen’s (1988) effect size conventions for the effect size d (low: 0.30; moderate: 0.50; large: 0.80) and r (low: 0.10; moderate: 0.30; large: 0.50). We interpreted AUC statistics based on conventions reported by Youngstrom (2013) for behavioral checklists and inventories (i.e., optimal AUC estimates in the 0.70–0.80 range), and reported diagnostic likelihood ratios (DiLRs) to facilitate interpretation with individual cases.

Results

Preliminary Analyses

We examined normality distributions of all of our continuous measures. With the exception of the BDI-II, the distributional properties of all of our continuous measures fell within acceptable skewness and kurtosis thresholds (i.e., range of +/− 2.0). We addressed normality concerns for the BDI-II reports by applying a square root transformation to all BDI-II reports, which brought them all underneath the thresholds reported previously. All analyses reported below use these transformed scores. Tables 1 and 2 include the means, standard deviations, and internal consistency estimates for all continuous measures.

In terms of the WSASY, both adolescent and parent reports displayed high internal consistency estimates. Both WSASY reports displayed a wide range in scores, suggesting the absence of floor or ceiling effects in scoring. As seen among the subgroups of adolescents reported in Table 2 who displayed survey scores above clinical cut scores (i.e., 0, 1, 2 or more), we observed considerable variation in scores, including adolescents who displayed some degree of psychosocial impairments even though no clinically elevated concerns were evident among the internalizing and externalizing domains assessed. Consistent with prior work on multi-informant assessments of youth mental health (De Los Reyes et al. 2013, 2015), adolescent-parent correspondence on WSASY reports was in the low-to-moderate range, r = 0.22; p < 0.05.

Convergent Validity

In Table 4 we report Pearson r correlations used to test convergent validity for adolescent and parent WSASY reports. Consistent with our hypotheses, we observed significant relations such that increased WSASY reports related to increased adolescent internalizing and externalizing concerns, and increased adolescent-parent conflict. Further, increased parent WSASY reports related to increased parent depressive symptoms (r = 0.36) and emotion reactivity (r = 0.26), and decreased parent quality of life (r = −0.25), all p’s < 0.01. Observed effect sizes of tests of convergent generally ranged from moderate to large in magnitude.

Table 4 Correlations among adolescent self-reports and parent reports of adolescent impairment, adolescent mental health, and adolescent–parent conflict
Full size table

Criterion-Related Validity-I: Clinical Cut Scores

We observed significant effects of group for tests of both adolescent (F[2, 102] = 10.78; p < 0.001) and parent (F[2, 102] = 24.52; p < 0.001) WSASY reports. As reported in Table 5, for both adolescent and parent reports, post-hoc Dunnett t tests revealed significantly higher WSASY scores for adolescents above clinical cut scores on 2 or more measures versus adolescents above cut scores on only 1 measure or 0 measures. Cohen’s d’s were all in the large-magnitude range. We also calculated AUC statistics for the ability of WSASY scores to identify adolescents above clinical cut scores on 2 or more measures based on adolescent report (n = 32) and parent report (n = 30). Both adolescent (AUC = 0.80; standard error = 0.04; p < 0.001) and parent (AUC = 0.82; standard error = 0.04; p < 0.001) WSASY reports performed well in identifying adolescents above clinical cut scores on 2 or more measures. For adolescent WSASY reports, we observed approximately 0.90 sensitivity for scores of 6.5 (with specificity of 0.53, and a DiLR of 0.18 for those falling in the low score range) and 0.90 specificity for scores of 16.5 (with sensitivity of 0.38, and a DiLR of 3.91 for those falling in the high score range). For parent WSASY reports, we observed approximately 0.90 sensitivity for scores of 5.5 (with specificity = 0.56 and DiLR of 0.18 for those falling in the low score range) and 0.90 specificity for scores of 14.5 (with sensitivity = 0.50 and DiLR of 6.25 for those falling in the high score range).

Table 5 Descriptive statistics for adolescent and parent reports on the work and social adjustment scale for youth
Full size table

Criterion-Related Validity-II: Peer-Related Impairments

For adolescent WSASY reports, we observed significantly greater scores for adolescents displaying 2 or more peer-related impairments versus 0 impairments (12.71 [7.92] vs. 8.38 [8.13]; t = 2.32; p < 0.05; d = 0.54). For parent WSASY reports, we observed significantly greater scores for adolescents displaying 2 or more impairments versus 0 impairments (12.09 [8.72] vs. 6.51 [6.82]; t = 3.16; p < 0.01; d = 0.71). Both effects were in the medium-magnitude range. As in prior work using this measure of peer-related impairments (Beale et al. 2018), we observed non-significant differences between adolescents displaying 2 or more versus 1 impairments (both p’s > 0.10) and 1 versus 0 impairments (both p’s > 0.25).

Criterion-Related Validity-III: Observed Behavior

For both adolescent (r = −0.28; p < 0.01) and parent (r = −0.27; p < 0.01) WSASY reports, we observed significant relations with adolescents’ observed social skills, such that greater impairment related to poorer social skills. Both effects were in the low-to-moderate range.

Discussion

This study advanced the literature on evidence-based assessment of youth psychosocial impairments by testing a new measure of such impairments, the WSASY. In a mixed clinical/community sample of adolescents and their parents, we made three findings. First, scores from WSASY reports displayed internal consistency estimates that were above recommended thresholds for internal consistency (i.e., α’s ≥ 0.80; see Nunnally and Bernstein 1994; Ryan et al. 2001). Second, scores from the WSASY displayed evidence of convergent validity. That is, consistent with our hypotheses, scores from the WSASY appeared to index impairment in a broad sense in that increased scores related to (a) increased adolescent internalizing and externalizing concerns, (b) increased adolescent-parent conflict, (c) increased parent depressive symptoms and emotion reactivity, and (d) decreased parent quality of life. Third, scores from WSASY reports displayed evidence of criterion-related validity, in that these scores (a) distinguished adolescents who displayed co-occurring mental health concerns from adolescents who did not, (b) distinguished adolescents who displayed multiple peer-related impairments from adolescents who did not, and (c) related to adolescents’ observed social skills in interactions with unfamiliar peers (i.e., increased WSASY scores were related to decreased social skills). In particular, the findings regarding links between WSASY scores and observed social skills represents a crucial observation. Indeed, these relations rule out the possibility that shared method variance fully accounts for our findings. Perhaps more important is the observation that the magnitudes in correlations between WSASY scores and observed social skills were at or above the typical correlations observed among cross-informant reports on social skills measures with shared method variance (see Renk and Phares 2004). In sum, our findings support interpreting scores from the WSASY as indexing psychosocial impairments.

Theoretical and Research Implications

Our findings have several important theoretical and research implications. Prior work indicates that youth displaying clinically elevated symptoms also evidence substantial individual differences in terms of whether they also experience co-occurring psychosocial impairments (for a review, see Rapee et al. 2012). That is, some youth experience mental health concerns without evidence that these concerns result in significant life interferences. For the first time, the WSASY adds a complementary element to this line of work. Specifically, adolescents may display some degree of behavior-related psychosocial impairments, without strong evidence that these behaviors reflect significant mental health concerns. Given that in our study we implemented a cross-sectional design, we could not rule out that those adolescents in our sample who did not display clinically elevated mental health concerns evidenced no prior history of such concerns. As such, our findings beg for additional inquiry focused on the onset of psychosocial impairments. In particular, future prospective research should leverage the WSASY in large community samples that display few mental health concerns at baseline.

In terms of basic research, the WSASY paves the way for continuing to build an evidence base for the developmental psychopathology of impairment. That is, an emerging body of work seeks to understand the links between impairment and mental health symptoms across development (i.e., their increases and decreases over time), but this work focuses on these links once symptoms are already evident (e.g., Cleverley et al. 2013). The availability of the WSASY allows us to ask distinct yet complementary questions. Do psychosocial impairments give rise to mental health symptoms or vice versa? Alternatively, do psychosocial impairments and mental health symptoms arise concurrently? It may be that there are individual differences in whether psychosocial impairments precede the emergence of clinically elevated mental health concerns. If true, are there particular “ages of onset” for impairments that portend poor prognoses and place youth on a poor developmental trajectory if their impairments do not receive clinical attention? All of these questions merit further study, and the WSASY facilitates addressing these questions. Indeed, given its brevity and ease of scoring, the WSASY readily lends itself to use not only in clinical contexts, but in screening contexts before mental health concerns may be evident (e.g., primary care and school settings).

In terms of applied research and practice, the WSASY may help in addressing long-standing problems in dissemination of evidenced-based practices in routine care settings. We see the WSASY as holding particular promise for use in low-resource settings that experience large disparities in the availability of evidence-based assessments (Beidas et al. 2015). The WSASY is brief, freely available, and contains forms for youth and parents to complete (Appendices A and B). The changes in likelihood (DiLR values) associated with low and high scores were substantial enough to often be clinically useful when applied to individual cases (Youngstrom et al. 2017). Further, in our sample scores from the WSASY related to scores from measures of a host of domains of adolescent mental health (i.e., anxiety, mood, ADHD, callous unemotionality) and psychosocial factors (e.g., family, peer, and parent functioning). In this respect, the WSASY may have clinical utility when integrated with (a) techniques designed for clients displaying multiple co-occurring concerns (e.g., modular therapies; unified protocols; Ehrenreich et al. 2009; Weisz et al. 2012); and (b) nosologies that seek to characterize mental health using domains that cut across diagnostic boundaries (e.g., Caspi et al. 2014; Sanislow et al. 2010).

Limitations

Four limitations of this study warrant comment. First, we constructed the WSASY so that it would be developmentally appropriate for informants to complete about youth. However, our study is the first to examine the WSASY’s psychometric properties and did so using a sample of adolescents. Thus, future research should test whether the psychometric properties of scores taken from the WSASY in this sample generalize to samples of relatively younger children.

Second, for a subset of participants in our sample, we relied on clinic data from a group of adolescents whose parents sought an evaluation on their behalf for social anxiety. We were able to demonstrate that both adolescent and parent WSASY reports related to multiple constructs beyond social anxiety (i.e., adolescent mood, ADHD, and callous-unemotionality; adolescent-parent conflict; parent mood, emotion reactivity, and quality of life). At the same time, we recommend that future studies test the psychometric properties of the WSASY in both specialty clinic settings that focus on domains other than social anxiety as well as general clinic settings (e.g., psychiatric hospitals and community mental health centers).

Third, although we examined associated features of psychosocial impairment (i.e., co-occurring concerns) and domains of psychosocial impairment (i.e., peer functioning and observed social skills), we did not include a secondary measure of general psychosocial impairment. In this psychometric study of the WSASY, we wanted to test the measure in a sample of participants who varied as to the presence of mental health concerns. In this way, we were able to test a key question about the measure: Could we collect psychometrically sound data about psychosocial impairments, even among participants who do not evidence significant mental health concerns? By construction, our study could not include a secondary measure of impairment for reasons noted previously (i.e., all current impairment measures require the presence of clinically elevated mental health concerns in order to collect impairment data). Thus, we encourage researchers to test the psychometric properties of the WSASY in clinic samples in which participants complete the WSASY and alternative impairment measures.

Fourth, our study did not involve the collection of diagnostic data. We were able to demonstrate that WSASY scores could distinguish participants on the number of clinically elevated concerns indicated on established mental health surveys (Table 5). Further, each of the surveys we used have accumulated evidence for the ability of scores taken from them to distinguish participants on diagnostic status (e.g., Beck et al. 1996; Beidel et al. 2000; Kessler et al. 2007). Nevertheless, future research should involve examining WSASY scores in relation to the outcomes of diagnostic assessments.

Youth may display behavior-related psychosocial impairments even in the absence of diagnosable mental health concerns, and yet no measures exist that can capably assess for youth impairments regardless of their mental health status. In our study, we tested the psychometric properties of a brief, freely available and multi-informant measure that assesses psychosocial impairments, and demonstrated the psychometric soundness of this measure in a mixed clinical/community sample of adolescents. The WSASY opens doors to a variety of areas of inquiry regarding the developmental psychopathology of impairment. We encourage future basic research on developmental issues regarding the onset of impairments and their links to mental health concerns. Further, future applied research should examine whether the WSASY can be pragmatically integrated into assessment batteries in low-resource mental health service settings, which are historically under-represented in use and interpretation of evidence-based assessments.