Autism spectrum disorder (ASD) is a pervasive and prevalent childhood neurodevelopmental disability which affects approximately 1 in 68 children (Wingate et al. 2014). This disorder is characterised by core impairments in socio-emotional reciprocity, interpersonal communication, and repetitive behaviours (American Psychiatric Association 2013). Deficits in socio-emotional competencies might be due to the lack of theory of mind or perspective-taking, which is one’s ability to understand others’ mental states, including emotions, beliefs, and intentions (Baron-Cohen 2000). Diminished ability of children with ASD to understand the relationship between others’ cognitive states and their actions lead them to engage in fewer activities and have fewer peers in their social networks than peers without ASD (Kreider et al. 2016).

There have been various forms of social skills training for children with ASD. Social stories as a method of teaching can facilitate the understanding of social contexts that a child might find difficult to interpret (Delano and Snell 2006; Kokina and Kern 2010; Scattone 2007). These stories include descriptive, perspective, and directive sentences (Gray and Garand 1993; Reynhout and Carter 2006). Previous studies have shown that social stories can improve understanding social situations, inferring perspectives of others, and demonstrating appropriate behaviour (Balakrishnan and Alias 2017; Marshall et al. 2016; Sansosti et al. 2004). This is aligned with the theory of mind, which suggests that attributing to other people’s thought will improve interpersonal skills. Social stories can help children with ASD find the social cues and enhance their communication skills. Using written stories, or stories in a form of pictorial cuing or videos, can increase frequency and length of positive interactions and modify inappropriate behaviours (Thiemann and Goldstein 2001). Since children with ASD are usually visual thinkers and rely more on visual features of the situations, taking advantage of using visual cues and digital media may expedite the process of learning and increase their motivation (Kunda and Goel 2011; More 2008).

Virtual reality programs, as a type of three-dimensional computer programs, have been widely used to serve as a teaching modality for children with ASD (Parsons and Cobb 2011; Volioti et al. 2016). These tools can create a dynamic environment for learning, enhance children’s motivation and engagement, and provide visual feedback during repetitive practices (More 2008). Previous studies have shown that children with ASD enhance their social skills via a virtual environment that represents a virtual café or a bus (Mitchell et al. 2007), and demonstrate some improvements in perspective-taking, emotion recognition, and social perception (Kandalaft et al. 2013). However, the majority of previous studies did not integrate social stories within their program, nor did they contextualize the character’s feelings and emotions in relevant daily social situations for children with ASD (Golan and Baron-Cohen 2006; Ploog et al. 2013). For example, most studies only used facial expressions or facial images without contexts or social stories, thus making them difficult to be interpreted by children with ASD (Golan and Baron-Cohen 2006; Ploog et al. 2013). This raises a long-lasting concern regarding generalizing the learned skills to real life.

Previous studies suggest that to enhance generalization of learned skills, the “general case training” method can be used (Chezan et al. 2012; Day and Horner 1986; Horner et al. 2005). This method identifies natural variations of stimuli seen in real environment (such as various types of feelings or emotions) and then selects an adequate number of stimuli based on their natural variation to teach the skill (e.g., a portion of the stimuli from each variation) (Horner et al. 2005). This approach can help with identifying the differences and similarities of stimuli, and understanding where the appropriate responses should occur. Thus, it can facilitate demonstrating correct responses to any stimuli that have similar features to the trained stimuli (Horner et al. 2005). The opportunity of exposure to varieties of stimuli and rehearsal of responses in various contexts may help transfer the learned skills. By applying general case training that targets perspective-taking through social stories in the content of the virtual reality program, we can potentially improve the generalization of the learned skills. To develop the content of the program, we considered a group of stimuli that was representative of a variety of social contexts and emotional responses within various social situations. The stimulus variations were types of feeling or emotion (i.e., happy, sad, angry, scared) and intensity of emotion (i.e., slight, moderate, extreme) in three social contexts (i.e., home, school, community). The aim of this project was to involve stakeholders in modifying and validating these various social stories to ensure that they were representative of the situations the children with ASD might encounter; these social stories would then be used as the content of the virtual reality program to help children with ASD increase emotion recognition and perspective-taking.

Involving stakeholders, including parents of and clinicians working with individuals with ASD, will strengthen the quality, relevance, and effectiveness of the stories and assure they can meet end-users’ needs (Sanders and Kirby 2012). Incorporating consumers’ views help to create valid and meaningful products (Sanders and Kirby 2012). Due to frequent issues with non-use or noncompliance with technologies, it is imperative that these programs not only appeal to young users, but also are consistent with the expectations and goals of families and clinicians.

As far as we know, no previous study has developed a validated socio-emotional stories based on stakeholders’ feedback as the content of a virtual reality program. This study aimed to create a validated library of social stories with various levels of difficulty and emotion intensity based on stakeholders’ input to target perspective-taking of children with ASD.

Method

Study Design

To validate the socio-emotional stories, we used the modified Delphi method via online surveys as a structured and iterative process of refining a group’s judgement (Dalkey et al. 1969; Jorm 2015). This method has the advantage of reducing the possible dominant opinion often seen in focus groups, leading to more autonomous and reliable responses (Dalkey 1972). The online method reduces the substantial time commitment during the multiple rounds of the validating process and increases the likelihood of receiving higher response rates. Through this iterative and controlled feedback process, we developed and validated the scenarios until consensus was reached (Hsu and Sandford 2007). The Delphi method for this study consisted of two main rounds (round 1/round 2) with each containing two iterations (a/b).

Throughout this study, we consulted with a steering committee of stakeholders who provided insight into setting up the structure of our study and helped us develop the initial scenarios. This committee consisted of one high functioning youth with ASD aged 15 years old, two parents of children with ASD (each had a son with ASD aged 8 years and 10 years old), and two clinicians working with children with ASD for more than 10 years.

Participants

Due to the possibility of attrition during the iterative process and multiple rounds, we aimed to recruit a large sample size of stakeholders from the initial stage of the project. We targeted parents of and clinicians working with individuals with ASD. This study consisted of 63 participants in round 1a (39 parents and 24 clinicians), 48 participants in round 1b and 2a (26 parents and 22 clinicians), and 38 participants in round 2b (20 parents and 18 clinicians).

Inclusion criteria were as follows: (1) clinicians had at least 1 year of work experience with individuals with ASD and were certified in one of the relevant disciplines (e.g., Speech Language Pathology, Occupational Therapy, Behavioural Therapy); or (2) parents had at least one child diagnosed with ASD. Their children should have been diagnosed with ASD by a specialist such as a registered psychologist or psychiatrist. In addition, all participants including parents and clinicians were required to comprehend written English.

Participants were recruited via email, online blogs, social media, and posters through health organizations and community ASD networks across Canada. Participants were also recruited via snowballing recruitment (i.e., asking participants to send recruitment information to other potential participants).

Informed consent was obtained from all participants prior to the study. This study was approved by the University Research Ethics Board.

Procedure

Participants were asked to fill out a demographic form describing their age and background. Then, they were invited to participate in two rounds of an online survey. Round 1 aimed to validate whether the story represents a situation in which children with ASD would have difficulty understanding or responding to appropriately. Round 2 aimed to validate the emotion type and emotion intensity related to the validated scenarios from round 1. We asked participants to rate their level of agreement on the content, type of emotion, and intensity using a 4-point Likert scale (including disagree, somewhat disagree, somewhat agree, and agree), in which 1 denotes “disagree” and 4 denotes “agree” (Kirkwood et al. 2003). Participants had the opportunity to give reasons for their level of agreement/disagreement.

Round 1: Validating Scenarios

The purpose of the first round was to obtain consensus of participants on the content of socio-emotional scenarios that can be used for a gaming program. First, we shared with our steering committee the scenarios we developed according to the ideas gathered from focus groups with stakeholders held in previous phases of the study and based on the literature (Golan et al. 2010; Bernad-Ripoll 2007; Rao et al. 2008). Any comments related to rephrasing the stories or changing the terms were addressed prior to the validation process. Next, participants were provided with those scenarios in the online survey.

Round 1a

In round 1a, the scenarios were presented by emotions including: angry (18 scenarios), scared (21 scenarios), sad (17 scenarios), and happy (17 scenarios). Because children with ASD usually experience anxiety and have difficulty understanding the emotion of fear, this category had the largest number of stories (Uljarevic and Hamilton 2013).

Each scenario consisted of describing a short emotional story involving two or three characters (avatars), depending on the content. The instructional goal of all scenarios was taking perspectives of avatars and responding to their feelings appropriately. The scenarios varied in complexity and included avatars of different ages to encourage children with ASD to take various perspectives. Each scenario also had a directional sentence or corresponding action. The corresponding action was a fun, fine or gross motor activity that focused on helping the avatar resolve the issue raised in the scenario. For instance, below is a scenario from the angry category and the corresponding player action:

The avatar was putting together a puzzle. As soon as she finished it, her dog ran over the puzzle and kicked some of the pieces [descriptive sentences]. The avatar got angry because her puzzle on which she was working on so hard was destroyed. However, the avatar can try again and fix it all. [perspective sentences]. Action or Instruction: Take five deep breaths and stay calm. Help the avatar find the pieces and put the puzzle back together [directive sentences].

Participants were asked to rate their level of agreement with the following statement: “This scenario represents a situation that a child with ASD might have difficulty in understanding or responding to appropriately.” If the participant marked any level of disagreement (i.e., either disagree or somewhat disagree), there was an open-ended question to give participants the opportunity to provide feedback and explain why the scenario may not be relevant to children with ASD. At the end of the survey, there was also a section for the participant to provide ideas for additional scenarios.

Round 1b

In round 1b, participants were presented with the summary of the results from round 1a. We also presented the revised scenarios from round 1a that did not reach agreement. These revised scenarios incorporated the comments received from participants in round 1a. We also included new scenarios that were suggested by participants in round 1a. Participants were asked to re-validate the scenarios by rating their level of agreement with the same statement from round 1a for each scenario.

Round 2: Validating Emotion Type and Intensity

The purpose of round 2 was to validate the type of feeling or emotion and the intensity of that emotion for each scenario.

Round 2a: The participants were presented with scenarios from round 1 that reached agreement. Each scenario was presented with a proposed targeted emotion (i.e., angry, scared/anxious, sad, or happy) as well as proposed intensity of the emotion (i.e., slight, moderate, or extreme). The participants were asked to state their level of agreement with the proposed emotion and the intensity of the emotion for each scenario. If participants selected an answer with any level of disagreement (i.e., disagree, somewhat disagree), they were asked to comment on which emotion or level of intensity they think it should be.

Round 2b

The number of scenarios that reached agreement for both the identified emotion and emotion intensity in round 2a were summarized for participants. For scenarios that did not reach agreement in round 2a, we either revised the scenario and/or changed the emotion type or intensity based on participants’ comments and then presented them to participants again to validate. Participants were asked to rate their level of agreement for each variable (i.e., emotion and intensity level) on the same Likert scale.

Data Analysis

During data analysis, the options agree or somewhat agree were considered as in agreement and the options disagree or somewhat disagree were considered as in disagreement. Researchers used descriptive statistical analysis that included the percentage of agreement for each scenario. For all rounds, agreement (consensus) was set a priori as ≥ 75% agreement. For each item, percentage of agreement was calculated by summing the number of participants who selected either agreement levels (i.e., somewhat agree, agree) and dividing that number by the total number of participants. We did not separate clinicians from parents and we considered each vote as having equal value. If this total calculation was 75% or greater, agreement was reached. This level of agreement was consistent with other studies that used the Delphi method (Keeney et al. 2006).

Researchers used a summative content analysis to analyze written comments that participants provided (Hsieh and Shannon 2005). For the scenarios that did not reach agreement, these comments were used to revise the content and/or the emotion type or intensity level.

Results

Demographic Information

Clinicians

Based on the round 1a, the 24 clinicians who participated in the study were primarily female (two males) and the age range was between 25 and 65 years old with the average of 42 years. Among clinicians, 79% resided in the province of British Columbia and the rest were from other provinces of Canada, including Ontario (4%), Alberta (4%), and Saskatchewan (13%). Ten clinicians were speech language pathologists, eight were behavioral consultants, two were occupational therapists, and four were psychologists. They worked for 1–40 years with children with ASD for a mean (SD) of 16.3 (9.2) years. 86% of clinicians were somewhat familiar, 4% were very familiar, and 9% were not too familiar with technology being used with individuals with ASD.

Parents

Based on round 1a, the 39 parents who participated in the study were primarily female (two males), ranging in age from 30 to 59 years; mean (SD) age was 43.8 (7.8) years. Among parents, 74% resided in the province of British Columbia and the rest were from other provinces of Canada, including Ontario (3%), Alberta (8%), Newfoundland and Labrador (2%), Saskatchewan (5%), Prince Edward Island (5%), and Nova Scotia (3%). Among those, 84% had one child with a diagnosis of ASD, four families (13%) had two children on the spectrum, and one family (3%) had three children with ASD. The sex of the child with ASD was mostly male except two families (7%) who had a daughter with ASD. The age range of participants’ children with ASD was between 2.5 and 18 years old [mean (SD): 12.2 (5.7) years]. Half of the children had no co-occurring conditions, but 33% of participants’ children also had attention deficit hyperactivity disorder, 13% had anxiety, and 3% had auditory processing disorder; 17% had more than one co-occurring condition.

Results of Each Round

Results from Round 1a: Validating Scenarios

In round 1a, of the 73 total scenarios, three scenarios did not reach agreement (two from the angry category and one from the scared/anxious category) (Table 1). The scenario of the scared/anxious category was eliminated and was not re-validated in round 1b, based on participants’ recommendations, to avoid reinforcing fear of bees. The two scenarios that did not reach agreement for the angry category were revised based on participants’ comments. Researchers also reviewed new scenarios suggested by participants and presented five relevant ones to be validated in round 1b.

Table 1 Number of scenarios that reached agreement in each round
Full size table

Participants’ comments were related to the content of the scenarios and the structure of the game. These included incorporating elements of cognitive behavioural therapy and calming strategies, in particular for the scared/anxious scenarios. Participants suggested to use action items as teaching moments by explaining to the player what was wrong in a scenario, what action needed to happen and why, and teaching children with ASD safety concepts related to challenging social situations. Stakeholders also commented about the age appropriateness and level of complexity of the scenarios. Incorporating these suggestions allowed us not only to promote emotion recognition and perspective-taking, but also to facilitate the understanding of the ‘why’—why others feel this way, and the ‘how’—what needs to be done to rectify the situation. These ideas have been addressed in the multiple iterations of the revisions.

Results from Round 1b

In round 1b, all new scenarios suggested by participants (one in the angry category, two in the scared category, and two in the sad category) and revised scenarios (two in angry category) reached agreement (Table 1).

Results from Round 2a: Validating Emotion Type & Intensity

In round 2a, one scenario did not reach agreement for emotion type and 13 scenarios did not reach agreement for emotion intensity (Table 1). Researchers revised scenarios that did not reach agreement based on participants’ comments to match the emotion type or intensity level. The one scenario that did not reach agreement for emotion type (72%) was revised to emphasize the intended emotion type. Scenarios that did not reach agreement for the emotion intensity were also revised based on participants’ comments. For example, a scenario that was intended to be slightly sad and reached only 71% agreement was revised from stating “… When she took the cake out of the oven it was burnt” to “the cake was lightly burnt on the edges”. We then revalidated the revised scenarios in round 2b.

Results from Round 2b

In round 2b, we re-validated the one scenario for emotion type and 13 scenarios for emotion intensity that did not reach agreement from the previous round. Of these, two scenarios did not reach agreement for emotion intensity, one in the scared category and one in the sad category (Table 1). Subsequently, both these scenarios were eliminated, as we did not conduct subsequent rounds of surveys. The result of all rounds produced a library of 75 validated scenarios.

During the emotion validation, we told participants to share their ideas on the major inherent emotion so that we could reflect on the group’s ideas for each scenario. The general comments from participants highlighted that scenarios sometimes might have more than one associated emotion (i.e., frustration and anger) and the intensity level of emotion was difficult to determine for some scenarios as the experience of emotions is individual.

Discussion

Social story as a teaching method is widely used for children with ASD to help them understand social situations. Social stories can be read to children or be presented in virtual reality programs. Although evidence shows positive outcomes when using the social story, developing social stories and identifying appropriate responses might be subjective and thus challenging. To this end, this project developed and validated a library of 75 short social stories for the content of a virtual reality program by involving parents and clinicians of individuals with ASD to improve perspective-taking among children with ASD.

Individuals with ASD may have difficulty in generalisation of learned skills in various settings. This might be explained with the weak central coherence theory that describes difficulties of individuals with ASD in integrating information and extracting the gist of the situation (Happé and Frith 2006; Plaisted 2001). The atypical information processing and increased local bias in individuals with ASD may interfere with their ability to interpret social cues, such as emotional faces, a task that mostly requires holistic processing (Behrmann et al. 2006). Thus, the different cognitive style among individuals with ASD highlights the necessity of adequate amount of practice in transferring skills. Using the general case training method and integrating it into the social stories, we considered various types and intensities of emotion in three social contexts of home, school, and community. General case training focuses on a range of discriminative stimuli (in this study, we examined contexts and the type and intensity of emotions) with relevant features in which a response should occur (Horner et al. 2005; McDonnell and Ferguson 1988; Petursdottir et al. 2007). Previous studies showed that using sufficient and various stimuli during the training may help with acquisition and generalisation of the functional behaviours among individuals with severe disorders (Horner and Albin 1988). The current project is the first study to consider varieties of stimuli in the content of a virtual reality program that targets perspective-taking in emotional situations among children with ASD. However, it should be noted that consistent exposure to variety of stimuli might not be sufficient in generalisation and the role of cognition and information processing among individuals with ASD should also be taken into account.

Presenting social stories in a virtual reality setting may motivate children with ASD to participate and make the learning more enjoyable. Previous studies have shown that combining pictorial cuing or presenting social stories in a virtual learning environment could facilitate communication among children with ASD (Thiemann and Goldstein 2001; Volioti et al. 2016). In addition to capitalizing motivation, virtual programs can provide a versatile learning platform to reduce the cost and increase the accessibility for users (Goldsmith and LeBlanc 2004). Moreover, engagement of end-users in the design, selection, and development of the stories will help meet the clients’ needs and maximize the outcome (Sanders and Kirby 2012; Volioti et al. 2016; Walsh and Barry 2008). This potentially increases the sustainability and adherence to the program.

During the iterative process, stakeholders provided a benchmark for levels of difficulty of stories and intensity of emotions. The majority of stakeholders’ comments were on the action items of the stories, such as using calming strategies to better reflect the adaptive strategies and appropriate responses in emotional situations. This is consistent with the literature, indicating the positive effects of cognitive behavioural therapy in emotion regulation and anxiety control for children with ASD (Conaughton et al. 2017; Weston et al. 2016). It has been shown that effective social stories can enhance children’s abilities to stay calm under stressful situations and increase communication skills (Lau and Win 2017).

In addition to calming strategies, participants noted the importance of explaining why the behaviour is appropriate or why the character feels a specific type of emotion. Explaining the reasons that govern a feeling or behaviour gives information about what is expected to happen (Smith 2017; Ying et al. 2016). Illustrating the context, expected feelings, appropriate behaviours, and consequences in social stories will help children with ASD to make social inferences. This is consistent with the theory of mind because understanding affective states (e.g., others’ feeling or emotions) and cognitive components (i.e., why s/he feels that way) is essential in selecting the appropriate behaviour (e.g., how to help if in that situation) (Bensalah et al. 2016; Dziobek et al. 2008). Stakeholders mentioned that integrating the safety concept is a critical element of the scenarios. Previous research found that there is a discrepancy between parents’ expectation and teaching safety skills in the educational programs (Agran and Krupp 2010). The lack of sensitivity to danger and risky situations, and presence of impulsive behaviours among children with ASD, call for safety management training. Recent studies have shown that children with ASD benefited from social stories and virtual reality programs that focused on safety skills (Self et al. 2007; Ying et al. 2016; Josman et al. 2008). The majority of research, however, has been on street walking skills and road safety. Therefore, incorporating the safety concepts in other areas within social stories is required.

The current project provides a uniquely validated library of social stories focused on perspective-taking, with diverse difficulty and intensity levels. The strength of this library lies not only on the diverse stories and stimuli, but mostly on the fact that the library was created with close collaboration with end-users. This library can reduce a significant burden on clinicians and special educators, allowing them to more quickly and easily adjust the stories based on the unique needs of each child. Wide representation of how ASD is presented and its heterogeneity reflect a need for tailoring individualized programs based on children’s symptoms (Wilczynski et al. 2007). For those who have severe problems in perspective-taking, these stories in varying difficulty levels can offer adequate stimuli based on general case training. Of particular usefulness, this library can be used as validated content for any virtual reality program, presenting the stories in a gradual level of emotion intensity and difficulty to help children with ASD.

Limitations and Future Direction

There are some limitations in the project. First, those who participated in validating the scenarios may not represent the diversity of the stakeholders. Participants of the study consisted mostly of women between the ages of 40 and 44 years old and majority of the clinicians had backgrounds in either speech language pathology or behavioral intervention. Furthermore, due to the multiple rounds of the Delphi, there was an attrition rate in the sample size. Further research needs to include a wider population, larger sample size, and involve children and youth with ASD to include their ideas and reflect on their lived experiences. Second, this project only considered four basic emotions with three intensities and contexts as a starting point to develop the program. Future studies are warranted to include emotions other than the four included in this study, such as disappointment, frustration, surprise, and excitement, as well as include other intensities and other contexts. Third, this study only recruited participants from one country. As the stories might be influenced by stereotypical behaviours or norms of a culture, the findings and scenarios might be cautiously used internationally. Fourth, this study did not evaluate the effects of the program. It is suggested that these validated socio-emotional scenarios be tested with children with ASD in different formats, such as a virtual reality, to investigate the outcomes in the future projects. Fifth, although we involved stakeholders in validating the socio-emotional scenarios, we did not involve them in dissemination of the findings as it was beyond the scope of the project. Future studies should plan for the methods of dissemination and implementation of these stories.

Conclusion

This study is the first attempt to develop and validate the content of a virtual reality program using general case training and social stories by involving parents of and clinicians working with individuals with ASD. The feedback provided by stakeholders allowed the researchers to gain insight to what scenarios and strategies are useful for targeting perspective taking. The developed library of validated social stories in various contexts and with varied emotions may help to improve socio-emotional skills among children with ASD.