Introduction

Children diagnosed with autism spectrum disorder (ASD) have difficulty with sociocommunicative functioning and restricted or repetitive behaviors or interests (American Psychiatric Association 2013) and often present with significant levels of emotional difficulties. For example, in a sample of 5–16-year-olds with ASD and no intellectual disability, 74 % had clinically significant emotional difficulties, such as anger, sadness or anxiety, compared to 18 % of typically developing peers (Totsika et al. 2011). Approximately 40–50 % of youth with ASD (as per DSM-IV-TR criteria) are estimated to meet criteria for two or more psychiatric disorders, often combining externalizing problems, such as ADHD, with internalizing problems, such as anxiety disorders (Leyfer et al. 2006; Simonoff et al. 2008), leading many to conceptualize emotional problems as involving underlying difficulties with emotion regulation (Mazefsky et al. 2013; Mazefsky and White 2014). Emotion regulation (ER) can be defined as ‘‘the extrinsic and intrinsic processes responsible for monitoring, evaluating, and modifying emotional reactions, especially their intensive and temporal features, to accomplish one’s goals’’ (Thompson et al. 2008, pp. 27–28). Considerable evidence has related poor ER to emotional problems in children with ASD, such as anxiety (e.g., Fujii et al. 2013; Wood and Gadow 2010), depression (Barnhill et al. 2000; Zablotsky et al. 2013), and anger (Rieffe et al. 2012). Further, ER impairments and detachment are more pronounced in children with ASD in comparison to a phenotypically similar sample of children with 22q13 Deletion Syndrome (Glaser and Shaw 2011).

Given these emotional difficulties, and evidence that ER is related to protective factors, such as prosocial peer engagement (Jahromi et al. 2013), there has been increased interest in assessing the effectiveness of cognitive behavior therapy (CBT) for youth with ASD, which until now has focused almost exclusively on anxiety (e.g., Ehrenreich-May et al. 2014; Reaven et al. 2012; Sofronoff et al. 2005; Storch et al. 2013; Wood et al. 2014). Although research exists documenting deficits in ER in individuals with ASD compared to peers (Mazefsky et al. 2013), empirical evaluations of ER interventions are needed (Sofronoff et al. 2014). In youth without ASD, CBT interventions that address ER have been shown to result in improvements in a wide array of emotional problems (Ehrenreich-May et al. 2013). One pilot study reported promising outcomes of a modified CBT program to address ER in a small sample of young children with ASD (5 in the treatment group and 6 in a delayed treatment control), 6–8 years of age, measured by child reported coping strategies in response to vignettes, parent reported negativity/lability and emotion regulation, and parent reported outbursts (Scarpa and Reyes 2011). Changes though were assessed using one-tailed comparisons, potentially inflating the likelihood of finding significance.

More research is needed to assess how cognitive behavioral interventions can improve ER in children with ASD. The purpose of the present study was to evaluate the preliminary clinical utility (feasibility and preliminary efficacy) of a CBT program, the Secret Agent Society: Operation Regulation (Beaumont 2013), designed to address ER skills in youth with ASD. The current assessment of clinical utility is similar to other preliminary assessments of CBT for addressing anxiety in ASD (McConachie et al. 2013; Reaven et al. 2012; White et al. 2009). We report on child characteristics (including IQ and ASD symptomology), parent, child, and blind clinician ER and psychopathology outcome measures, and various treatment feasibility measures. We operationalized feasibility by child, parent and therapist satisfaction, treatment adherence (i.e., attendance, attrition, engagement, homework completion) and treatment fidelity (American Psychological Association 2006; White et al. 2009).

Methods

Participants

Fourteen children (1 femaleFootnote 1) met the following inclusion criteria: (a) a confirmed ASD diagnosis from available clinician reports and/or the Autism Diagnostic Observation Schedule (ADOS; Lord et al. 2000); (b) average intellectual functioning (IQ > 80) on the two-subtest scale (Vocabulary and Matrix Reasoning) of the Wechsler Abbreviated Scale of Intelligence-2nd Edition (WASI-II; Wechsler 2011); (c) between the ages of 8 and 12 years; and (d) demonstrated willingness to attend research and therapy sessions. Participants did not need to meet clinical cut-offs for mood, anxiety, or behavioral disorders to participate, as the focus of the intervention was meant to address core areas of ER. See Table 1 for detailed participant characteristics.

Table 1 Child characterization measures and descriptives
Full size table

Parents completed an additional ASD screening tool, the Social Communication Questionnaire (SCQ; Rutter et al. 2003). All children exceeded the ASD clinical cut-off score of 15, recommended to discriminate ASD and non-ASD populations (Rutter et al. 2003). Parents also rated severity of ASD symptoms on the Social Responsiveness Scale-2nd Edition (SRS-2; Constantino and Gruber 2012), and all youth had total T-scores in at least the mild ASD range (60 or above).

Parent Report Outcome Measures

Emotion Regulation Checklist (ERC; Shields and Cicchetti 1997). The ERC consists of two subscales: Lability/Negativity (LN, i.e., mood swings, and dysregulated negative affect) and Emotion Regulation (ER, i.e., self-awareness, appropriate emotionality and empathy). Parents were asked to identify how frequently their child displayed behaviors described on a 24-item, 4-point Likert-type scale (1 = “rarely/never” to 4 = “almost always”). Low scores on LN indicate lower levels of negative affect and low scores on the ER scale imply lower levels of ER. The ERC has excellent internal consistency (α = .96 for LN scale; α = .83 for ER scale) and adequate validity, differentiating between regulated and dysregulated children and correlating with other measures of ER (Shields and Cicchetti 1997, 2001). In addition, the ERC has been previously applied in the ASD population (Jahromi et al. 2013). In the current sample, internal consistency was acceptable for LN scores (α = .78) and good for ER scores (α = .84).

Anxiety Disorders Interview Schedule: Parent Interview-4th Edition (ADIS-P-IV; Silverman and Albano 1996). The ADIS-P-IV is a semi-structured diagnostic interview designed to assess symptomatology, course, etiology, and severity of children with emotional disorders, based on parent report. A trained post-doctoral fellow conducted the interview and provided an overall clinician’s rating scale score (CRS), ranging from 0 to 8, with higher scores representing more severe psychopathology. Recent research with youth with high functioning ASD suggests excellent inter-rater agreement (Ung et al. 2014).

Behavior Assessment System for Children-2nd Edition (BASC-2; Reynolds and Kamphaus 2006). The BASC-2 is a standardized measure of clinical concerns and adaptive skills used to help identify typically occurring childhood and adolescent clinical diagnoses (Tan 2007). It has been utilized in a number of studies involving children with ASD (e.g., Mahan and Matson 2011; Volker et al. 2010). Parents completed the 150-item Parent Rating Scale, providing four general composites: Externalizing Behaviors, Internalizing Behaviors, Adaptive Behaviors, and Behavioral Symptoms Index.

Child Report Outcome Measures

Children’s Emotion Management Scale: Anger, Sadness, Worry (CEM; Zeman et al. 2010; Zeman et al. 2001). The CEM consists of an 11-item Anger scale, 12-item Sadness scale, and 10-item Worry scale, which assess children’s self-reported appraisal of inhibition of expression (e.g., “I hold my anger in”), coping (e.g., “I can stop myself from losing my temper”), and culturally unacceptable emotional displays termed “dysregulation” (e.g., “I say mean things to others when I am mad”). Children rated the frequency of each item using a 3-point Likert-type scale (1 = “hardly ever” to 3 = “often”). We analyzed overall dysregulation, coping, and inhibition across emotions. In the current sample, the overall inhibition and coping scores had good internal consistency (α = .91, α = .87) although the overall dysregulation scores were not as strong (α = .62).

Scenarios. Experimenters read children two scenarios: (1) James and the Math Test (Attwood 2004a), about a young boy, James, who is feeling anxious about completing a math test in class, and (2) Dylan is Being Teased (Attwood 2004b), about a young boy, Dylan, who is being bullied at school. The child was asked to offer suggestions for how the children in the scenarios could effectively cope with the difficulties, which were transcribed for later scoring (one point per appropriate response). The total number of appropriate responses was combined across the two scenarios. Both tasks have been used with children with ASD to examine learning of effective problem solving and coping skills (Beaumont and Sofronoff 2008).

Blind Rater Outcome Measure

Clinical Global Impressions Scale (CGI; Guy 1976). The CGI is a single clinician rating of illness severity (CGI-S) and treatment-related improvement (CGI-I). The CGI-S is used to assess the global severity of psychopathology on a 7-point scale (0 = “no illness” to 6 = “serious illness”) and the CGI-I is used to assess clinical improvement of psychopathology (0 = “very much improved” to 6 = “very much worse”). An independent clinical evaluator (ICE) who was not involved in data collection or in direct implementation of the intervention, reviewed de-identified copies of each participant’s ADIS-P-IV and BASC-2 summary sheets at pre- and post-treatment to complete the CGI-S and CGI-I ratings. This method is similar to the methods described in previous studies (e.g., Storch et al. 2013). Participants obtaining a CGI-I of 0, 1, or 2 were considered to be positive treatment responders.

Feasibility Measures

Child, Parent, and Therapist Satisfaction

After each weekly session, children and parents independently completed satisfaction questionnaires, rating each session activity on a 5-point Likert-type scale (1 = “not helpful” to 5 = “very helpful”; adapted from Reaven et al. 2012). Evaluations were completed with the therapist out of the room, and once completed, were put in sealed envelopes. Therapists completed similar ratings of the session activities.

Treatment Adherence

In addition to attrition and attendance, therapists recorded additional information after each session including: (a) whether the child completed homework (0 = “not at all” to 3 = “completely”); (b) the child’s level of engagement during the sessions (1 = “completely uninvolved” to 5 = “actively involved”) and (c) therapeutic alliance with both child and parent (1 = “very poor” to 7 = “very good”).

Summary Questions

After the post-intervention assessment, parents received an online survey, asking them to respond “yes” or “no” to the following summary questions (similar to those used in McConachie et al. 2013): Since completing the SAS-OR program… (1) has your child’s emotion regulation improved?; (2) do you feel that your child’s ability to deal with angry emotions has improved?; (3) do you feel that your child’s ability to deal with anxious emotions has improved?; (4) do you feel you child’s ability to deal with sad emotions has improved?; and (5) do you think the SAS-OR program has helped your child in/any other way? If yes, can you tell us how?

Treatment Fidelity

After each session, therapists completed self-report treatment integrity checklists that included session-specific treatment components (Beaumont 2013). Two independent observers also recorded therapist performance on a random selection of 26 video-recorded sessions. Overall treatment integrity was 89.6 % across 26 sessions (SD = 9.94, range = 65.4–100 %). Observers double-coded 23.1 % (n = 6) of sessions and inter-rater reliability was excellent (intraclass correlation = .81; Cicchetti 1994).

Procedure

This study was approved by the York University research ethics review board. Informed parent consent and youth assent were obtained prior to any data collection. Once parents expressed interest, children were screened for eligibility in three phases (phone, online questionnaires, and in-person). Participants were assessed at pre-intervention (1–2 weeks before the 10-week intervention) and at post-intervention (1–2 weeks following treatment). Participants were recruited from the community via postings on local advocacy websites and community organizations, and from clinician referrals.

Intervention

We implemented a modified version of the previously validated Jr. Detective Program (Beaumont and Sofronoff 2008; described as Secret Agent Society), a group based intervention that targets social skill development in children with ASD and at least average IQ. The current iteration, Secret Agent Society: Operation Regulation, is an individualized spy themed intervention that instead targets ER (see Table 2 for summary of session activities). Each 1-h session was made up of a progress check, multimedia activities such as computer games, modeling and role-playing to practice skills, education based in cognitive behavior therapy, relaxation and mindfulness activities (e.g., sensory grounding strategies to promote awareness), strategies to promote generalization to home and school, and a token reinforcement system to maintain attention and motivation. The child, his/her parent(s) and one therapist were present for each session, which progressed from targeting basic emotional awareness (i.e., identifying emotions in self and others), to implementing relaxation strategies and ER tools to cope with difficult emotions such as anger and anxiety. Parent involvement was encouraged in each session where appropriate (e.g., discussion around difficult situations for child, brainstorming ways to practice skills at home, etc.).

Table 2 Overview of secret agent society: operation regulation session activities and goals
Full size table

Therapists were four graduate students and one post-doctoral fellow, supervised by a clinical psychologist. Training included a 1-day seminar to familiarize therapists with the treatment manuals and procedures and to model and role-play various components of the intervention (e.g., practicing breathing exercise scripts). Therapists had to reach an acceptable level of treatment fidelity (80 %) in mock sessions before they were assigned a child participant.

Results

Feasibility Outcomes

Parent, Child, and Therapist Satisfaction Ratings

Mean satisfaction ratings of activities across the 10 weekly sessions (5-point scale) were high for parents (M = 4.4, SD = .26) and therapists (M = 4.2, SD = .50), and slightly lower for children (M = 3.8, SD = .75). See Fig. 1 for mean ratings for each session across children, parents, and therapists.

Fig. 1
figure 1

Mean child, parent, and therapist satisfaction ratings for each session

Full size image

Treatment Adherence

Of the 14 children originally enrolled in the trial, there was one family who dropped out after the first session for personal (non intervention) reasons. All other child–parent dyads completed the intervention. Session attendance was 100 % for treatment completers. Across participants and the 10 sessions, therapist ratings were high for homework completion (M = 2.75, SD = .32, 3-point scale); child engagement (M = 4.5, SD = .44, 5-point scale); therapeutic alliance with child (M = 6.3, SD = .47, 7-point scale) and therapeutic alliance with parent (M = 6.5, SD = .43, 7-point scale). See Fig. 2 for therapeutic alliance ratings across sessions.

Fig. 2
figure 2

Mean therapist reported therapeutic alliance ratings with child and parent for each session

Full size image

Summary Questions

Twelve out of 13 parents completed the post-intervention summary questions, and 92 % (n = 11) indicated an improvement in both ER and ability to deal with anger, 75 % (n = 9) indicated an improvement in ability to deal with anxiety, and 58 % (n = 7) indicated an improvement in the ability to deal with sadness. Most parents (92 %, n = 11) also provided positive feedback in an open-ended section, focused on the helpfulness of the program (e.g., “He has an overall improvement in understanding emotions in both himself and others”).

Preliminary Outcomes

Table 3 includes parent, child, and clinician reported outcome scores at pre- and post-intervention, with changes assessed used two-tailed paired-samples t-tests. Improvements were noted on parent reported child emotional lability [t(10) = 3.13, p = .001] internalizing symptoms [t(12) = 3.18, p = .008], behavioral dysregulation [t(12) = 2.38, p = .035], and adaptive behavior [t(12) = −3.24, p = .007]. Improvements were also found based on clinician rated overall severity [t(12) = 3.39, p = .005] and number of diagnoses [t(12) = 2.80, p = .016] on the ADIS-P-IV. Of the 12 children who completed the CEM, results indicated significantly more overall inhibition [t(11) = −2.32, p = .04], and less overall dysregulation [t(11) = 2.14, p = .061] across three emotions (anger, anxiety, sadness). Children also provided more appropriate ways of coping to the James/Dylan scenarios at post-intervention [t(12) = −2.07, p = .06]. Children came up with more appropriate hypothetical behaviors for the characters such as “James could ask for help with his math” and “Dylan could tell an adult he is being teased.” There was a significant decrease in CGI-S scores based on independent clinical judgment; nine children (69 %) were rated as “improved” to “very much improved” and no children were rated as having worsened [t(12) = 3.95, p = .002].

Table 3 Descriptive statistics and t-test results for outcome measures
Full size table

Discussion

Results demonstrate preliminary feasibility of a CBT intervention for addressing ER in children with ASD, an area of limited evaluation. Feasibility was demonstrated by high parent, child, and therapist satisfaction with session activities, high parent reported satisfaction with the effectiveness of the overall intervention, high attendance (100 %) and retention (one non-treatment completer), and strong treatment fidelity. Although children rated session helpfulness as lower than parents and therapists, we suspect that lower child ratings were related to session activities pertaining to difficult emotions (i.e., anger, anxiety) and the fact that activities are meant to challenge youth using systematic exposure to mildly distress eliciting activities, as well as to the differential reinforcing value of specific activities. Future research with larger samples is needed to identify the most and least helpful activities within the intervention. High levels of treatment fidelity also suggest that graduate students can adequately deliver the intervention with appropriate supervision, an important implication for the potential of the intervention being extended to use in community agencies.

An additional aim of this study was to assess the preliminary effectiveness of the intervention through various child, parent, and blind clinician reported outcome measures of ER. Overall, parent reports of child ER indicated general improvements (e.g., less lability and negativity reported on the ERC, less internalizing and behavior symptoms, and more adaptive behaviors on the BASC-2). Children reported an overall decrease in dysregulation and increase in number of appropriate strategies in response to the James/Dylan scenarios. Although no feedback was provided regarding the scenarios at pre-intervention, it is important to consider the potential for practice effects given that the same scenarios were administered at post-intervention. Parents reported fewer anxiety diagnoses and lower severity on the ADIS-P-IV post-intervention, and blind clinician ratings on the CGI-I and CGI-S indicated an overall decrease in psychopathology for the current sample. This is one of the few trials to focus on improving ER in children with ASD through CBT, rather than focusing on anxiety or anger only, and thus may better reflect the clinical reality of treating children who present with an array of emotional difficulties beyond a singular anxiety focus.

There are a number of important limitations to our findings. Due to the small sample size and multiple comparisons, results need to be interpreted with caution. The lack of a control condition poses potential risk of regression to the mean or expectancy biases, similar to what has been identified as a limitation in similar open trial evaluations of CBT for anxiety (e.g., Ehrenreich-May et al. 2014). In addition to child and parent report measures, teacher-report data are also missing from the current evaluation, which is pertinent to assessing generalization to the school environment. Results are also not generalizable to females given that the entire sample was male and not representative of the typical prevalence estimates of a 5:1 male to female ratio (Centers for Disease Control and Prevention 2014).

We also did not report on follow-up data for the current sample, so more long-term assessment is needed to comment on maintenance of any treatment gains. Further investigation is needed to confirm appropriate measures for assessing ER in ASD population (Mazefsky et al. 2011), especially given discussion around validity of self-report questionnaires in the ASD population (Mazefsky et al. 2011).

Emotion regulation is most often assessed using only one type of method (e.g., self report) with the most common types being self and informant report. Direct observation is less frequently used but also an important method, and ideally, a multi-method approach should be used to assess ER (Weiss et al. 2014). Although we provide multiple perspectives on change—child report, parent report, and semi-structured interviews of parents carried out by clinicians, future studies should also include direct measures of ER.

Despite these limitations, establishing a new intervention’s clinical utility is important, and the preliminary outcomes support further evaluation of the efficacy of the intervention in a larger randomized control trial. In a larger trial, primary outcomes will be measured by changes in child ER ability according to child report (e.g., CEM; Zeman et al. 2010) and parent report (e.g., ERC; Shields and Cicchetti 1997). This is especially important to inform future investigation and recommendations of interventions for youth with ASD who have multiple emotional difficulties and where the conceptualization of the presenting problems are based in difficulties with ER, an area lacking empirical support.