Social skills and peer relationships are crucial to the overall development of young people and important for wellbeing throughout life (Ladd 2005). Positive social relationships help to protect against the impact of challenging circumstances (Pettit and Collins 2011), academic underachievement and school disengagement (Rodkin and Ryan 2012). Social competence buffers against mental health difficulties, including internalizing disorders, such as anxiety and depression, and externalizing problems such as antisocial behaviors (Ingram and Price 2010). Longitudinal path modelling links social competence with peers in childhood to adaptive functioning in adulthood (Englund et al. 2011). Children lacking positive peer relations are also at increased risk of later substance abuse, severe psychiatric disorders, relationship difficulties and occupational problems (Ladd 1999; Segrin and Givertz 2003). Thus, addressing social skill deficits and peer relationship difficulties early in life through effective interventions is critical to optimizing children’s developmental trajectories.

A substantial number of children has been found to experience regular peer relationship difficulties, with prevalence data finding between 10–18% of children are rejected and 15–19% neglected by peers (Ettekal and Ladd 2015; Östberg 2003; Schuster 1999). Research conducted with over 10,000 school-age children and adolescents reported that nearly 20% experienced social skill difficulties and had very few friends (Bernard et al. 2007). Unfortunately, without effective intervention, social and emotional problems may get worse as children grow older (Rubin et al. 2009).

‘Social skills’ involve verbal and non-verbal behaviors that are appropriate, reactive to a situation and generally reinforce others for their social interaction, leading to positive peer relationships (Ladd 1999). These skills include play skills such as turn-taking, cooperation and compromise, and the macro- and micro-components of social communication skills such as physical distance, facial expression, voice tone, appropriately starting and maintaining conversations, speech duration, and smooth topic transitions (Bierman et al. 2010). Deficits in skilled social behavior can be the result of a failure to acquire social skills, a failure to produce social skills, or a combination of these factors (Gresham et al. 2010). Key reasons children may fail to produce socially skilled behaviors include (1) difficulties with social cognition (misinterpreting social situations or social problem solving), (2) difficulties with emotion regulation (not being in an emotional state to produce the appropriate social skills, or access social cognition skills) and (3) insufficient opportunities to interact with peers (e.g., peer rejection, neglect or victimization) (Spence 2003).

Other common contributors to social skill deficits in children include difficulties recognizing how others feel from their facial, postural, voice tone and contextual cues (Trentacosta and Fine 2010), and aspects of social cognition such as attention, information processing, decision-making and planning abilities (Bierman et al. 2010). These challenges are particularly prevalent amongst children who struggle with anxiety and anger management. For example, children who are anxious in social situations are prone to social perception and interpretation difficulties due to self-focused attention, gaze aversion, concentration problems and difficulties in recognizing emotions in others (Walker et al. 2011), and there is some evidence to suggest they are biased towards interpreting ambiguous information negatively (Miers et al. 2011). Children who display aggressive behaviors often have notable gaps in their emotional knowledge (Trentacosta and Fine 2010), problems in attending to and encoding social information, hostile attributional biases and challenges with social problem-solving (McMahon et al. 2006).

Children prone to anxiety and depression are more likely to have difficulty regulating their emotions, and be shy and socially withdrawn relative to their peers (Sanson et al. 2004). Children experiencing anxiety are more vulnerable to peer relationship problems than their non-anxious counterparts. This can occur through showing greater distress (e.g., crying) when upset or victimized, or through engaging in potentially annoying behaviors, such as excessive reassurance-seeking, compulsive behaviors, or clinginess (Beesdo et al. 2009; Spence et al. 2009). Using a sample of children with a broad range of anxiety conditions, and controlling for primary social anxiety and depression, Motoca et al. (2012) examined the relationships between anxiety, peer interactions and social skills. They found that higher anxiety was associated with both poorer social skills and greater peer interaction difficulties. Social anxiety in particular is associated with withdrawal and reduced participation in social activities, and self- versus other-focused attention during interactions (Rapee and Spence 2004). The reduction in anxiety provided by this avoidance serves to increase the future likelihood of avoidance and social withdrawal (Kingery et al. 2010). This in turn may stunt social-emotional development, lead to less effective peer interactions, reinforce beliefs in poor social performance, strengthen social anxiety, and reduce self-esteem (Kingery et al. 2010). The reciprocal relationships between social anxiety, avoidance, difficulties with social competence and peer relationship problems are well-recognized in both nonclinical and clinical samples (Alden and Taylor 2004; Rubin et al. 2009).

Numerous interventions have been used to teach children social skills, with meta-analytic evaluations showing that social skills training programs generally show an overall positive impact, with moderate effect sizes (e.g., Lösel and Beelmann 2003; Quinn et al. 1999). However, as the ecological validity of the outcome measure increases, treatment effect sizes tend to decrease. Specifically, direct measurement of children’s knowledge and ability to perform targeted skills in the treatment setting has shown the largest treatment effect sizes, followed by children’s application of specific skills in real-world contexts, with the smallest effect sizes usually seen for social outcome measures, such as social competence (e.g., number of real friendships; invitations by peers to social events) or adjustment (Nangle et al. 2010). This speaks to giving greater emphasis to measuring real-world outcomes that indicate whether training effects are meaningful, such as social competence or reduced distress (Nangle et al. 2010). Disparities in the magnitude of effects obtained by specific interventions also depend on factors such as the strength of the teaching content, training methods, ability to tailor to individual children’s needs while maintaining treatment fidelity, focus on generalization enhancement, and quality of implementation. Many interventions reporting positive outcomes have a duration of over a year (e.g., Hennessey 2007; Shucksmith et al. 2010), while strong and lasting outcomes have also been achieved with brief intervention programs (e.g., Beidel et al. 2005; Humphrey et al. 2010).

The majority of commercially available social-emotional skills training interventions have limitations. For example, they typically focus heavily on either emotion regulation (e.g., anxiety or anger management, such as the FRIENDS for Life® program – Barrett 2013) OR social interaction skills (e.g., Frankel and Myatt’s (2003) Children’s Friendship Training), but rarely both. This is a critical consideration, given that children are unlikely to achieve social success in their interactions with peers if they are unable to manage their emotions in social situations. Furthermore, while many of these programs include child and parent therapy sessions and resources, they fail to include resources to update school staff (e.g., classroom teachers, paraprofessionals and guidance officers) on what children are learning each week, and how they can support them to put these skills into action when needed in the classroom and playground to promote skill generalization. Finally, a common complaint made by community practitioners is that many of the published social skills curricula that are available are ‘boring’ and ‘outdated’, featuring text-dense, black-and-white books or handouts that encourage children to say catchphrases that are at odds with the language used by their peers (e.g., “Rocks on”; McGrath and Noble 2011, p. 192).

To address the issues described above (i.e., programs lacking engaging teaching materials, focusing on only emotion regulation or social skills and not including school staff resources to promote skill generalization to school), the Secret Agent Society (SAS) program was developed. The commercially available program includes a multi-level computer game, nine weekly child ‘club meetings’ where children play a range of spy themed games and activities, parent training sessions and teacher tip-sheets (go to www.sst-institute.net for further details). Children are introduced to core skills in the curriculum through the computer game, which they play with the support of an adult (e.g., parent, school staff member) between weekly club meetings. This teaching tool allows for self-paced learning and the provision of individualized multimedia feedback to the user. Learning core concepts through the computer game is intended to boost children’s confidence to engage in games and activities in the weekly child club meetings, which are designed to provide fun skill practice opportunities in a supportive group environment. Children are also assigned weekly ‘missions’ or skills practice tasks that involve applying the skills that they are learning in the program at home, at school and in other social environments and capturing evidence of their mission completion in their Cadet Handbooks, the Journal section of the computer game or with a phone or tablet (with photos, voice recordings or film clips).

The SAS program (formerly called the Junior Detective Training Program) was originally developed to teach emotion regulation and social skills to eight to 12-year-old children with Autism Spectrum Conditions without an intellectual disability (HFASD), and with expressive and receptive language skills at least equivalent to an eight-year-old. A randomized controlled clinic-based trial with this population (n = 49) found that children in the intervention condition made significant and meaningful gains in their social and emotional functioning compared to children in the wait-list control condition, with improvements shown at both home and at school (Beaumont and Sofronoff 2008). Treatment gains were maintained at five-month follow up, with 76% of children improving to within the range of typically developing children on a parent-report measure of social functioning (Beaumont and Sofronoff 2008). These findings have since been replicated in other trials of the program in mainstream schools (Beaumont et al. 2015) and specialist schools (Einfeld et al. 2017) with children who have HFASD, with treatment gains shown to be maintained up to one year after the program ended (Einfeld et al. 2017).

Given the prevalence of child friendship problems and the aforementioned limitations of many commercially available social skills training programs, this study aimed to examine the preliminary utility and effectiveness of the SAS program in improving the emotion regulation and social skills of neurotypical children experiencing peer relationship difficulties. Specifically, it was hypothesized that children who participated in the SAS program would make significant improvements from pre- to post-treatment in the following domains: (1) social skills (including emotion regulation skill use); (2) social competence; (3) knowledge of contextually-appropriate anxiety- and anger-management strategies; (4) overall anxiety and (5) social anxiety. Consistent with past research findings for the SAS program (e.g., Beaumont et al. 2015; Beaumont and Sofronoff 2008; Einfeld et al. 2017), it was also predicted that improvements in these domains would be maintained at six-week follow-up.

Method

Participants

Forty children and their parents were initially recruited for the study, with twenty-seven meeting the study criteria and completing pre- and post-treatment assessment measures. A power analysis using G*Power software and effect sizes of ɳ2 = 0.5 obtained from a previous randomized controlled trial of the SAS program with a different population (Beaumont and Sofronoff 2008) indicated that 23 participants would be sufficient to detect effects at p < 0.05 level of significance if they did indeed exist (Faul et al. 2009). Children ranged in age from seven to 12 years, with a mean age of 9.8 years (SD = 1.34) and more boys than girls participating (20:7). Participants resided in a major metropolitan centre or travelled from a surrounding region. Demographic details of participants are shown below in Table 1.

Table 1 Sample characteristics as reported by parents on a customized demographic questionnaire
Full size table

Inclusion criteria

Inclusion criteria required children to experience peer relationship difficulties and/or social anxiety symptoms, as indicated by a score of 14 or lower (i.e., a score between the normative means for average/ popular and rejected children) on the parent-report Social Competence with Peers Questionnaire (Spence 1995a), or a score of one standard deviation or more above the normative mean for their age and gender on the social phobia subscale of the Spence Children’s Anxiety Scale—Parent Version (Spence 1999). Children also needed to be enrolled in a mainstream school and available to attend the scheduled sessions. To meet the research goal of evaluating the program with a different, non-ASD population, children were screened for the presence of autism spectrum disorders using the Childhood Asperger Syndrome Test (CAST; Scott et al. 2002). Where children’s scores were above the cut-off of 15 indicating that further assessment for ASD was warranted, they were only included if parents reported that a pediatrician or child psychiatrist had already ruled out an ASD diagnosis, as occurred in five cases. Four children were excluded from the study based on elevated CAST scores.

Children who were deemed to be at risk of suicide based on findings from child- and parent-clinical interviews were referred on for individual treatment more suitable to their immediate needs. No further exclusion criteria were set; children were not required to meet any formal diagnostic criteria (e.g., for social phobia), and were not excluded on the basis of co-morbid diagnoses, conditions or behavioral problems (e.g., Attention Deficit Hyperactivity Disorder—ADHD). Participants were asked to refrain from involvement in other psychosocial therapies over the course of the trial.

Procedure

The study gained ethical approval from the University of Queensland’s Behavioral and Social Sciences Ethical Review Committee, in accordance with the standards of the National Health and Medical Research Council of Australia. Participants for the study were primarily recruited via local school newsletters and a newspaper article. Interested families contacted the second author via phone or email. They were asked a few initial questions via their contact method to pre-screen for study eligibility (e.g., child’s age, diagnoses given by a professional). Potentially eligible parents were emailed a brief brochure and parent information sheet and advised that a program facilitator would be in contact to organize for an initial set of questionnaires to be completed and to schedule an assessment session.

A pack containing the initial screening questionnaires (Demographic Questionnaire; Social Competence with Peers Questionnaire—Spence 1995a; Spence Children’s Anxiety Scale-parent—Spence 1999; Childhood Asperger Syndrome Test—Scott et al. 2002) and a reply-paid envelope were sent to potentially eligible families. Parents were requested to return the questionnaires at least a week prior to the scheduled assessment interview session. Where screening questionnaires indicated that families were ineligible for the trial, they were contacted by the second author who discussed their needs and provided referral pathways.

Child and parent assessment sessions were conducted concurrently by two clinical psychology postgraduate students trained in the SAS program and assessment administration. Sessions were scheduled for one and a half hours and held at the University of Queensland Psychology Clinic. After initial introductions, rapport building and an overview of the assessment process, the child went with one facilitator to a separate room, while the parent/s remained with the other facilitator. The consent/assent forms were then reviewed and signed prior to commencing the clinical interviews. Parent interviews focused on assessing children’s social-emotional strengths and challenges, parental responses, children’s diagnostic and treatment history, medication use, any evidence of suicidality and parents’ current program goals. Child interviews focused on rapport building, finding out about the child’s strengths, likes and dislikes, their insight into any social-emotional challenges and screening for any mood issues and/or suicidality.

Following the interviews, parents were asked to complete the remaining parent-report questionnaires (Social Skills Questionnaire-parent—Spence 1995a; Emotion Regulation and Social Skills Questionnaire—Beaumont and Sofronoff 2008) and children completed the self-report questionnaires (Social Skills Questionnaire-pupil—Spence 1995a; Spence Children’s Anxiety Scale—Spence 1998), with the exception of James and the Maths Test (Attwood 2004a) and Dylan is Being Teased (Attwood 2004b). These latter child assessments were administered during the first group therapy session, with children sitting in separate areas of the room where they were read the task and individually provided with allowable prompts to maximize their performance. Those children requiring assistance with writing, as established from the clinical interviews, were taken to separate rooms to enable them to give verbal responses that a trained administrator recorded for them.

Secret Agent Society (SAS) Intervention

The SAS intervention was delivered at the University of Queensland psychology clinic by post-graduate psychology interns who were trained in SAS program facilitation and received ongoing supervision. After an initial two-hour parent introductory session in which caregivers learned how to use the SAS computer game, nine weekly 90-minute child therapy group sessions (‘club meetings’) were held, each followed by a half hour large-group parent session in a separate room. During the parent sessions, children practiced applying the skills they were learning in the program during informal play sessions that were supervised by a facilitator (e.g., practicing turn taking while playing the UNO card game). A child- and parent-booster session was held six weeks after the weekly sessions ended. Each child therapy group consisted of three to five children aged within two years of each other.

All members within a group were of the same sex (two groups of girls and four groups of boys). While past published research (e.g., Beaumont and Sofronoff 2008; Beaumont et al. 2015) has shown the SAS program to be effective with groups of mixed sex, this study had a comparatively larger proportion of female participants, affording the opportunity for same-sex groups. Given eight to 12 year-olds typically socialize in same-sex groups (Rubin et al. 2009), adopting a same-sex group intervention framework was intended to encourage bonding amongst child group members and allow more in-depth exploration of how social skills and concepts most often present for children of a given sex (e.g., more subtle versus overt exclusion amongst girls versus boys). All children completed each therapy session and all but two children completed the six-week follow-up booster and assessment session. This high attendance rate was partially due to families who missed the occasional session attending one-on-one make-up sessions immediately before their next group meeting. All child groups were scheduled at the same time, so that parents could come together as one large group for the weekly 30-minute parent group meetings.

Initial sessions of the program taught children emotion recognition and emotion regulation skills, while later sessions applied this content to increasingly complex social interaction skills. A summary of the content of each child group session of the program is featured in the digital supplementary material. Full details of how the program is delivered are available in the Secret Agent Society Facilitator Manual (Beaumont 2010).

To facilitate skill learning and generalization, child club meeting content is presented in a full colour illustrated child ‘Cadet Handbook’ and children are given pocket sized collectible skill ‘Code Cards’ which visually illustrate the key steps of each skill that is taught in the program. Children can refer to these cards when needed ‘in secret’ to remind them of the skill code steps. Children are also assigned ‘missions’ (typically exposure and skills practice tasks) to complete each week and are asked to answer questions about these missions in the Secret Agent Journal section of their Cadet Handbooks or the computer game. An example mission (‘Relaxation Gadgetry’) involved children using their chosen relaxation ‘gadgets’ or strategies from the program to calm down when they felt low to medium levels of anxiety or anger during the week.

Parent group sessions were facilitated by two interns in a separate seminar room and involved briefly reviewing the activities that children completed in their child club meeting for a given week and discussing how parents could support their children to use these skills outside of session. For example, parents were educated on the same relaxation gadgets that children learned about in their child club meeting, and encouraged to model using these themselves at home, and to prompt, praise and reward their children for using these skills between sessions. Parents were provided with a Parent Workbook that summarized session content and weekly Teacher Tip Sheets to forward to their children’s classroom teacher and/or other school support staff.

Measures

Demographic questionnaire

A purpose-written questionnaire asked parents to initially provide demographic information, child diagnostic history, how they would categorize their child on items pertaining to peer relationships, and the duration of social and peer difficulties.

Social Competence with Peers Questionnaire (SCPQ)

This brief, 9-item parent-report questionnaire examines children’s positive interactions with others, specifically their level of social competence with same-age peers (Spence 1995a). While this measure is related to social skill, it more directly captures the outcome of children’s social functioning; their success in their relationships with others (Spence 1995a, 1995b). Items include “Has at least one close friend”, and “Has good relationships with classmates”. Parents indicate how reflective each statement has been of their child over the past four weeks using a three-point scale where 0 = not true, 1 = sometimes true, and 2 = mostly true. Normative data is based on a sample of Australian children (Spence 1995a). Cronbach’s alpha for the current sample was 0.90. The measure’s validity has been demonstrated through significant correlations between parent-reported SCPQ scores, and both teacher ratings of children’s social competence and peer-reported child sociometric status (Spence 1995a). The measure was used as both a screening and program outcome evaluation tool in this study (i.e., it was administered at pre-program, post-program and at the six-week follow-up session).

Spence Children’s Anxiety Scale–Parent (SCAS-P)

The parent-report version of the Spence Children’s Anxiety Scale (SCAS; Spence 1999) covers a broad range of symptoms that have been shown to cluster into six subscales - separation anxiety, generalized anxiety, social phobia, panic/agoraphobia, obsessive-compulsive and fear of physical injuries (Nauta et al. 2004; Spence 1999). All symptom subscale items also map onto a higher-order factor, anxiety in general, which is expressed as the total score (Nauta et al. 2004). The social phobia subscale of the SCAS-P was used as an initial screening tool in this study. Both the total score and the social phobia subscale score of the SCAS-P were used as treatment outcome measures for the purposes of this study (i.e., administered at pre-treatment, post-treatment and at the six-week follow-up session). The SCAS-P contains 38 symptom items, rated as occurring never (0), sometimes (1), often (2), or always (3). The validity of the individual subscales has been supported with exploratory and confirmatory factor analysis, and the measure is able to differentiate between diagnostic categories in samples of clinically anxious children (Nauta et al. 2004; Spence 1999). Spence and colleagues have also demonstrated that the total scale possesses strong concurrent and discriminant validity with numerous other well-recognized assessment tools using multiple informants. The internal reliability (Cronbach’s α) of the SCAS-P for the current sample was .87. Norms by age and gender, as relevant, are provided for the total scale and individual subscales.

Childhood Asperger Syndrome Test (CAST)

The Childhood Asperger Syndrome Test (Scott et al. 2002) was used to screen for ASD at study entry. This parent-report questionnaire consists of 37 items. Items are rated using a dichotomous yes/no scale and yield a total score, with a score of 15 or above indicating that further assessment for ASD is warranted (Scott et al. 2002; Williams et al. 2005). The CAST has been recommended as a screen for all ASD and Pervasive Developmental Disorders not otherwise specified (PDD-NOS) in primary school aged children with the capacity to attend mainstream schools (Allison et al. 2007; Williams et al. 2005). The measure has demonstrated good test-retest reliability and has shown 100% sensitivity; it is much more likely to err on the side of a high false positive rate, with a positive predictive validity of 50% (Williams et al. 2005).

Social Skills Questionnaire–Parent and –Pupil (SSQ-P, SSQ-PU)

The 30 item parent- and child-report versions of Spence’s Social Skills Questionnaire (Spence 1995a) cover a broad range of social skills (e.g., “Follows the rules in games or activities” and “Shares things with other kids his/her age”). Respondents are asked to rate each item as either not true (0) sometimes true (1) or mostly true (2) for the past four weeks. Normative data is available for both versions. Internal reliability of the SSQ-P and SSQ-PU is good, with Cronbach’s alphas of 0.86 and 0.87 for the current sample, respectively. The concurrent and construct validity of the parent version has been demonstrated in significant correlations with the teacher and child versions of the SSQ, parent reports on the Social Competence with Peers Questionnaire, and the correspondence between scores and sociometric status derived from classroom peer report (Spence 1995a). The child-report version shows some evidence of validity, with significant correlations with the parent-report form. However, children’s self-ratings were found to only minimally concur with teacher ratings of their social skills and did not correspond with peer sociometric status (Spence 1995a). Both the parent- and child-report versions of the SSQ were administered at pre-program, post-program and at the six-week follow-up session.

Emotion Regulation and Social Skills Questionnaire (ERSSQ)

The Emotion Regulation and Social Skills Questionnaire was specifically developed to assess children’s use of the social skills, including the requisite emotional regulation skills, that are taught in the SAS program. Parents rate the frequency with which their child uses each skill on a 5-point scale ranging from 0 = never to 4 = always, with some items reverse scored. Example items include “Starts conversations with others in a socially appropriate way”, and “Recognises when others are bored by his/her conversation and changes the topic”. This 27 item measure has shown good internal reliability with a coefficient alpha of 0.89, exploratory factor analytic support for the use of a single total score, and good concurrent validity (r = 0.86) with the Spence (1995a) Social Skills Questionnaire (Butterworth et al. 2014). Norms are not yet available for the ERSSQ. Cronbach’s alpha for the current sample was 0.87, with the measure administered before and after the program ended and at the six-week follow-up session.

James and the Maths Test (JMT)

Children were individually administered James and the Maths Test (Attwood 2004a) at pre-program, post-program and six-week follow-up, which assesses their knowledge of appropriate anxiety management strategies. They are read aloud a story about a boy (James) who feels anxious about a maths test and are asked to suggest ideas for how James can cope with his anxiety. One point is awarded for each appropriate anxiety-management strategy suggested. Depending on preference and writing ability, children can either write down their ideas or they are transcribed for them by a trained assessor. The second author initially scored all forms. Any ambiguities were discussed with a second, highly experienced rater, who was blind to time of assessment (pre-treatment, post-treatment or six-week follow-up), until 100% inter-rater agreement was reached. Additional forms were randomly selected and independently scored by the blind rater until 33% of the forms had been examined; in all cases, ratings were identical.

Dylan is Being Teased (DBT)

The Dylan is Being Teased measure (Attwood 2004b) assesses children’s knowledge and ability to generate appropriate responses to a hypothetical situation where a child, Dylan, is being bullied and becomes angry. The same administration and scoring process at pre-program, post-program and six-week follow-up was employed as for James and the Maths Test. Complete inter-rater agreement was also obtained for 33% of forms examined by both raters.

Spence Children’s Anxiety Scale–Child (SCAS-C)

In addition to the previously described parent version of the Spence Children’s Anxiety Scale, the child version of the measure (total score and social phobia subscale score) was used to examine program outcomes by comparing scores at pre-program, post-program and six-week follow-up. Similar to the parent-report form, the child version consists of 38 items and six subscales (separation anxiety, generalized anxiety, social phobia, panic/agoraphobia, obsessive-compulsive and fear of physical injuries—Nauta et al. 2004; Spence 1998), together with a total general anxiety score. The child-report version contains an additional six, positively framed, filler items, which are not scored. The validity of the scale has been well established (Nauta et al. 2004; Spence 1998) and in the current study, the internal reliability (Cronbach’s α) was 0.90.

Data Analyses

Repeated measures multivariate analysis of variance (MANOVA), with Time of Assessment as the within-subjects factor (pre-treatment, post-treatment and six-week follow-up), was used to examine changes in parent-reported social skills and child knowledge of emotion regulation strategies. As child- and parent-reported anxiety involved examination of the total anxiety score and social anxiety subscale from the same questionnaire, a series of repeated measures one-way analyses of variance (ANOVAs) was conducted with an adjusted alpha. Child self-reported social skill usage and parent-reported child social competence were also evaluated using ANOVA, with Time of Assessment as the within-subjects factor. Due to the large number of primary statistical analyses performed, a Bonferroni adjusted alpha of 0.006 was applied (i.e., 0.05/8 analyses).

Statistically significant main effects were followed up with a priori contrasts to assess changes between pre- and post-treatment, and between post-treatment and follow-up. To control for elevated Type I error when these pairs of comparisons were made, alpha was adjusted to a conservative p = 0.001. Effect sizes were examined and interpreted according to the guidelines proposed by Cohen (1988), where 0.01 is a small effect, 0.03 is a medium effect and 0.14 is a large effect size. Data missing for two families at follow-up were treated as intention-to-treat cases, and these cases were retained in the analyses.

The magnitude of child participants’ treatment gains was also analyzed by calculating reliability change indices (RCI) for each outcome measure. The percentage of child participants who demonstrated statistically reliable change (i.e., a reliability change index score of greater than 1.96) from pre- to post-treatment and pre-treatment to follow-up are described below.

Results

Social Skills

Parent-report measures

Means and standard deviations for the Spence Social Skills Questionnaire-Parent (SSQ-P) and Emotion Regulation and Social Skills Questionnaire (ERSSQ) across the three time-points are shown in Table 2. Results revealed a significant overall main effect of Time for the combined dependent variables, F(2, 25) = 10.63, p < 0.0001, partial ɳ2 = 0.65. Separate univariate analysis of the individual measures showed a main effect of Time for both the SSQ-P, F(2, 25) = 21.91, p < 0.0001, partial ɳ2 = 0.46, and ERSSQ, F(2, 25) = 35.60, p < 0.0001, partial ɳ2 = 0.58. Pairwise comparisons revealed that there was a significant improvement in children’s social skills use from pre- to post-treatment according to parent report on the SSQ, p < 0.0001 and the ERSSQ, p < 0.0001. Furthermore, non-significant differences between post-treatment and follow-up ratings indicate that these treatment gains were maintained for six weeks following treatment, SSQ, p = 0.84; ERSSQ, p = 0.89.

Table 2 Mean scores obtained (with standard deviations in brackets) for parent- and child-reported measures and Spence’s (1995a) published normative sample comparison (where data is available)
Full size table

Examination of statistically reliable change in social skills found 13 out of the 27 participants enrolled in this study (48%) showed a statistically reliable improvement in SSQ-P scores at post-treatment and six-week follow-up relative to pre-treatment, with none showing a statistically reliable deterioration in social skills over the course of the study. On the ERSSQ, 70% of parents reported a statistically reliable improvement in their child’s emotion regulation and social skills from pre- to post-treatment and from pre-treatment to six-week follow-up. None reported a statistically significant deterioration in their child’s social-emotional functioning at post-treatment or follow-up relative to pre-treatment.

Comparison between parent-reported social skills on the SSQ-P and social status obtained via classroom peer-reported sociometric analysis (Spence 1995a) is possible and these comparative statistics are reported in Table 2. The current sample’s mean fell below the mean for rejected children at pre-treatment. Following intervention, the sample scores had improved to between the average and popular social status groups.

Child-report measure

Comparative sociometric status statistics were not available for the self-report version of the SSQ as Spence (1995a) did not find any correspondence between child-reported social skills use and child sociometric status. Means and standard deviations for the child report version of the Spence Social Skills Questionnaire (SSQ-PU) are shown in Table 2 for both the study sample and the normative sample (Spence 1995a). The similarity of these means indicated that according to child self-ratings, the present sample did not have difficulties with social skills at intake. No significant overall main effect of Time was found, F(2, 25) = 1.19, p = 0.32, indicating that children did not report any differences in their social skill use over the duration of the intervention. From pre- to post-treatment, 7% of the total child participant sample (2/27) showed a statistically reliable improvement in SSQ-PU scores, while 7% showed a statistically significant reduction in their SSQ-PU scores. From pre-treatment to six-week follow-up, 11% (3/27) participants demonstrated a statistically reliable improvement in scores, with 7% (2/27) continuing to demonstrate a statistically reliable deterioration in self-reported social skills.

Social Competence

Results for the parent-report Social Competence with Peers Questionnaire (SCPQ-P) showed a significant main effect of Time, F(2, 25) = 17.12, p < 0.0001, partial ɳ2 = 0.58. Follow-up analyses indicated that significant improvements in children’s social competence occurred from pre-treatment to post-treatment, p < 0.0001. Furthermore, these gains were maintained at six-week follow-up, as indicated by the non-significant difference from post-treatment to follow-up, p = 0.54. As can be seen in Table 2, before the intervention, the sample mean, M = 6.33, fell almost three standard deviations below the normative mean (M = 14.82, SD = 3.12). By follow-up, the sample mean had moved to within one standard deviation of the normative mean. Fourteen out of 27 participants (52%) demonstrated a statistically reliable improvement from pre- to post-treatment on the SCQ-P, and 15/27 participants (55%) demonstrated a statistically reliable improvement from pre-treatment to follow-up. One participant (4%) showed a statistically reliable deterioration in social competence on this measure from pre-treatment to six-week follow-up.

Knowledge of Anxiety- and Anger-Management Strategies

Means and standard deviations for the James and the Maths Test (JMT) and Dylan is Being Teased (DBT) measures can be seen in Table 2. There was a significant main effect for Time of Assessment across the combined dependent variables, F(2, 25) = 24.13, p < 0.0001, partial ɳ2 = 0.81. When considered separately, the measure of appropriate anxiety management strategies (JMT), F(2, 25) = 50.15, p < 0.0001, partial ɳ2 = 0.66, and the measure of appropriate anger management strategies (DBT), F(2, 25) = 33.07, p < 0.0001, partial ɳ2 = 0.56, showed significant main effects over time. Follow-up analyses showed that children were able to generate significantly more appropriate emotion management strategies for hypothetical anxiety and anger provoking situations at post-treatment than they could at pre-treatment, p < 0.0001 for both JMT and DBT. These gains were also maintained at follow-up, with non-significant differences between post-treatment and follow-up for JMT and DBT, p = 0.93 and p = 0.41, respectively. Twenty-six percent of child participants demonstrated statistically reliable improvements on JMT from pre-treatment to post-treatment and pre-treatment to follow-up respectively. On the DBT measure, 26% of participants showed a statistically reliable improvement from pre-treatment to post-treatment, with 33% showing a statistically reliable improvement from pre-treatment to six-week follow-up.

Anxiety

To determine whether any significant reduction in child anxiety occurred over the course of the intervention, both the total anxiety scores and the social phobia subscale scores were examined on the child- and parent-report versions of the Spence Children’s Anxiety Scale (SCAS & SCAS-P). As can be seen from the mean scores displayed in Table 2, the pattern shows decreases in anxiety from pre-intervention to follow-up for all measures. Additionally, it can be seen that the parent-reported mean scores suggest elevated levels of anxiety at pre-intervention compared with the normative sample (Nauta et al. 2004), while child-reported mean scores were similar to the normative sample means at baseline (Spence 1998).

Overall Anxiety

Parent-report measure

Parent-reported total anxiety score results showed a significant main effect for time of assessment, F(2, 25) = 8.57, p = 0.001, partial ɳ2 = 0.41. Specifically, significant decreases in total anxiety were reported from pre-treatment to post-treatment, p < 0.0001 by parents. Pairwise comparisons also showed maintenance of treatment gains with no significant differences occurring between post-treatment and follow-up, p = 0.57. Forty-eight percent (13/27) of parents reported a statistically reliable improvement in their children’s anxiety levels at post-treatment and follow-up relative to pre-treatment, with one parent (4%) reporting a statistically significant increase in their child’s overall anxiety at post-treatment and follow-up relative to pre-treatment.

Child-report measure

Analysis of child-reported total anxiety scores showed that the main effect of Time was not significant, F(2, 25) = 4.05, p = 0.03, with an adjusted alpha of .006 due to the number of statistical analyses performed in this study. Thirty-three percent of child participants (9/27) reported a statistically reliable improvement from pre- to post-treatment, and 41% (11/27) reported a statistically reliable improvement from pre-treatment to six-week follow-up. While no child participants reported a statistically reliable increase in overall anxiety from pre- to post-treatment, two children (7%) reported a statistically reliable elevation in their overall anxiety levels from pre-treatment to six-week follow-up.

Social Anxiety

Parent-report measure

Examination of parent-reported reductions in social anxiety over the course of the intervention revealed no significant main effect of Time F(2, 25) = 4.62, p = 0.02. Twenty-two percent (6/27) of parents reported a statistically reliable reduction in their children’s social anxiety scores at post-treatment and six-week follow-up relative to pre-treatment. One participant also reported a statistically significant increase in their child’s social anxiety from pre- to post-treatment, which remitted at follow-up.

Child-report measure

Analysis of child-reported reductions in social anxiety showed a significant main effect of Time, F(2, 25) = 7.12, p = 0.004, partial ɳ2 = 0.36. Follow-up comparisons indicated that significant decreases in social phobia symptoms occurred from pre-treatment to post-treatment, p = 0.001, and that the scores were not significantly different from post-treatment to follow-up, p = 0.87, indicating maintenance of treatment gains. Thirty percent of children (8/27) self-reported statistically reliable improvements in their social anxiety levels at post-treatment and six-week follow-up on the SCAS-C relative to pre-treatment, with one child reporting a statistically reliable increase in their overall anxiety level from pre-treatment to follow-up.

Clinically Meaningful Change

As children were not preselected on the basis of psychopathology or clinically significant difficulties, it was considered beneficial to also examine the clinical meaningfulness of change for the subset of children who did present with scores in the clinical range at pre-treatment. For child- and parent-report measures with normative data available, the participants falling within the clinical and borderline clinical ranges prior to treatment were identified and the number and percentage of these children who improved to within the normal range following treatment are reported in Table 3.

Table 3 Movement of children from the clinical range to normal range after intervention on social ability and anxiety measures
Full size table

Social Skills and Competence

Parent-report measures

On the parent report version of the Spence Social Skills Questionnaire (SSQ-P) only eight of the 27 children were identified as having elevated difficulties with social skills, while 21 of the 27 children were identified as having clinical level difficulties with social competence with peers (SCPQ) prior to intervention. The social skills of the majority of these children (75%) were reported by parents to be within the normal range at post-intervention, with a small but non-significant deterioration at follow-up. Children’s social competence indicated greater improvements over time; with more children’s functioning having improved to within the normal range for social competence (57%) by follow-up than at post-intervention. Of the nine children who remained within the clinical range for social competence, five children (a further 24%) had still improved by over one standard deviation.

Child-report measures

Only four children self-reported difficulties with social skills prior to starting the program, with two reporting social skill use within the normal range after completing the intervention.

Anxiety

Parent-report measures

The clinical meaningfulness of decreases in elevated anxiety symptomatology was also examined. Parent-report for total anxiety (SCAS-P Total Scale) revealed that of those children initially presenting with clinically elevated symptoms of anxiety in multiple domains (n = 9), 67% were experiencing normal levels of anxiety at post treatment and follow-up (n = 6). On the SCAS-P social phobia subscale, parental report indicated that of the 11 children experiencing clinically elevated social phobia symptoms at pre-treatment, 55% (n = 6) were rated within the normal range for social anxiety at post-treatment.

Child-report measures

Of those children who initially self-reported clinically elevated overall symptoms of anxiety on the SCAS-C (n = 6), 33% (n = 2) and 50% (n = 3) respectively described experiencing anxiety within normal levels at post-treatment and six-week follow-up. With regards to social anxiety, seven children rated themselves as experiencing clinically elevated symptoms at pre-treatment on the SCAS-C, with all but one (86%) rating their social anxiety symptoms as having improved to within the normal range following participation in the program.

Discussion

The current findings provide some preliminary support for the effectiveness of the Secret Agent Society (SAS) social-emotional skills training program for typically developing children who are experiencing peer relationship difficulties and/or anxiety in social situations. Consistent with the first hypothesis of this study, children’s use of social and emotional skills increased significantly on parent-report measures from pre- to post-treatment, with 48 and 70% of the child sample respectively showing statistically significant improvements on the two parent-report measures of social-emotional functioning (the SSQ-P and ERSSQ). Involvement in the intervention was also associated with the sample mean on the normed measure of social-skill use increasing from below the mean for children sociometrically classed as rejected, to above the mean for children who are generally socially accepted by their peers (Spence 1995a). However, not consistent with the first hypothesis, children’s self-report of their social skills indicated that, as a group, they did not have baseline difficulties with their social skills use, and did not differ in their use of social skills following the intervention. While this finding was unexpected, it is consistent with findings that indicate that some typically developing children can have poor insight into their social abilities (Gresham et al. 2010), and/or have a different perspective on their social skills relative to adults (Spence 1995a), perhaps perceiving their social challenges to be a biproduct of others’ difficulties, rather than their own. Consistent with recent research on children with ADHD (e.g., McQuade et al. 2017), children in the current sample may have displayed a ‘social positive illusory bias’, over-estimating or over-reporting their social-emotional competence relative to their parents. Without independent observational and/or other informant data, however, one cannot conclude from this study whose perception of social functioning was more accurate: that of the children, or that of their parents.

As predicted by the second hypothesis, improvements in children’s social and emotional skill use appeared to translate into improvements in social adjustment, with a large effect size attained for parent-reported improvements in children’s social competence and over half the participant sample showing statistically reliable score increases on the SCPQ-P. This finding provides initial support for skill generalization to real-world outcomes. In examining the clinical meaningfulness of these gains, it was found that at baseline, three quarters of the children were identified as having clinical level difficulties with peer social competence. Of these children, over half had improved to within the normal range of functioning by follow-up, and of those who remained within the clinical range, the majority had still experienced improvements over one standard deviation in magnitude. While further evaluation with a longer follow-up period is required, the pattern of results also tentatively suggests that there was a lag in children experiencing improvements in their social competence with peers. This is consistent with literature showing that children often experience delays in improved social outcomes following improvements in their social behavior (Mikami et al. 2010).

As predicted in the third hypothesis, over the course of the intervention, there were significant improvements obtained on measures of children’s knowledge of appropriate anxiety- and anger-management strategies, although the percentage of children who made statistically reliably improvements on JMT and DBT was relatively low (26–33%). The hypothesis that children would experience a decrease in overall anxiety as a result of program participation (Hypothesis 4) was also only partially supported. Significant decreases in total anxiety were reported by parents for the sample as a whole, accompanied by findings that, of those children initially clinically elevated for total anxiety, two thirds moved to within the normal range for overall anxiety after the program. While the pattern of results was mirrored in children’s self-reports of their overall anxiety levels, with lowest levels occurring at follow-up, this did not reach significance. A possible explanation for this lack of significance is that of a sleeper effect (Segrin 2000). Delays in treatment effects have been reported for children with anxiety, with a number of studies reporting non-significant results from pre- to post-treatment, but achieving meaningful and significant improvements from pre-intervention to follow-up, indicating that extended intervals may be required for participants to practice, apply and consolidate skills (Silverman et al. 2008).

Mixed empirical support was provided for the fifth hypothesis of the study, with significant decreases in social anxiety being reported by children from pre- to post-treatment, but not by parents. When considering the subset of children (11/27) who were initially experiencing clinically elevated social anxiety symptoms, around half (according to parent report) and over three quarters (according to child self-report) improved to within the normal range for social anxiety. This suggests that improved social competence, emotional regulation, and/or the other components of best practice social anxiety treatment that feature in the program (e.g., exposure; McNally 2007) may translate to reductions in children’s social distress in cases where this is of clinical concern. However, the finding that only 33% of the total child sample showed statistically reliable improvements in self-reported social anxiety suggests that large improvements are less likely for participants whose social anxiety falls within the ‘normal’ range for their age and gender.

As predicted, treatment gains were maintained at six-week follow-up for all measures that reached significance. This provides initial evidence for the durability of the program’s beneficial effects (at least in the short-term) for typically developing children and suggests that children continued to use their SAS strategies even after weekly therapy sessions ended. As part of the program curriculum, children and parents planned how and when they would use target skills from SAS to address ongoing social-emotional challenges. However, a weakness of this study is that their follow-though in doing so was not systematically evaluated.

Strengths, Limitations, and Future Research

A strength of this study was the measurement of ecologically valid outcomes, such as children’s use of social and emotional skills (rather than just their ability to demonstrate skills in a clinic setting), their social competence with peers (such as whether they get invited to social events, whether they have any friends), and their experience of anxiety symptomatology. A primary limitation of this study was the absence of a control group. It is possible that improvements in children’s social and emotional outcomes were due to maturational effects or other intervening factors, rather than SAS program participation. However, evidence against this interpretation is provided by longitudinal examinations of anxiety in samples of primary school children, where it has been found that after Grade 1 and through to early adolescence, children’s anxiety levels generally remain quite stable (Duchesne et al. 2010; Ialongo et al. 1995).

The representativeness of the current sample is also questionable, given the high attendance rate and participant retention in the trial given that no financial compensation was offered for program completion. This finding suggests that the participant group was highly motivated, which may have contributed to the treatment effects demonstrated. Furthermore, generic treatment elements (as opposed to the specific SAS program content) and responder bias may have also contributed to the favorable findings reported in this study. As parents and children were both the intervention recipients and evaluators, their self-report data may have been skewed in a positive direction. Until more objective observational-, peer- and teacher-report data is collected through a waitlist and/or treatment comparison randomized controlled trial, only tentative conclusions about the program’s effectiveness can be made. A larger efficacy trial should also include more rigorous monitoring of treatment fidelity, as in the current study, this was only informally tracked and reported to be high by the clinical psychology interns who delivered the intervention. Including longer-term follow up evaluations will also provide opportunity for examination of durability of treatment effects over time, and the identification of mechanisms of change. Given that children’s scores on some measures that failed to reach significance at post-treatment had increased at follow-up, this may also permit detection of any delayed/sleeper effects.

The results achieved, despite the heterogeneous nature of the sample, suggest the program may be of benefit to a diverse range of children who experience social-emotional challenges. However, the flipside of this strength is that smaller numbers of children fell into the specific clinically elevated subgroups and diagnostic categories. This also applies to determining equivalence of program effects across genders. Boys represented two thirds of the typically developing sample, (a gender ratio that is reflective of the composition of most social skill interventions; Quinn et al. 1999), resulting in a smaller number of female participants. This may be due to findings that social withdrawal in boys is more likely to be associated with negative emotional and social outcomes than in girls (Rubin et al. 2009), and thus they are more likely to be referred for support. Replication with a larger sample, and/or with specific diagnostic subgroups of children is recommended.

Future research directions may also examine the value-add of integrating peer-mediated approaches into the program within a school context and evaluate the utility of the intervention as a whole-of-class curriculum. This latter delivery method is in line with the pedagogically driven inclusion stance of many schools, potentially facilitating access for children with HFASD and typically developing children with peer relationship difficulties, without the stigma of being singled out or withdrawn from class. Investigation into cross-cultural validity and factors that may mediate or moderate treatment effects (e.g., teacher implementation of class-wide strategies and parental variables) would also be of interest.

The findings from this study suggests that a slightly adapted version of the SAS small group program may be a valuable intervention to address the social-emotional challenges that many children face. The commercially-available program is brief, engaging, targets both emotion regulation and social skill deficits and incorporates parent- and teacher supports to promote children’s skill generalization, overcoming limitations of many other commonly-used social skills training curricula. Relative to the commercially-available SAS program originally developed for children on the Autism Spectrum (Beaumont 2010), the current program featured only three minor modifications: the Parent Workbook and Teacher Tip-Sheets were reprinted without reference to ASD, an additional relaxation strategy, the ‘relaxation radar’ (i.e., looking for calm or happy things around you) was added and standardized questions were included to challenge unhelpful thoughts (“enemy thought transformers”—e.g., What would I tell a friend in this situation?).

Future research should evaluate whether the costs associated with delivering resilience programs such as SAS to children (including professional training, program delivery materials and staff time) are more than compensated for by the potential benefits, which may include improved mental health outcomes, school achievement and greater employability in adulthood. These findings would have significant implications for government and insurance policies and funding initiatives.