Keywords

1 Introduction

Serious games (SGs) are “games that do not have entertainment, enjoyment, or fun as their primary purpose” [9]. SGs take advantage of game elements to create engaging, fun, and challenging scenarios that motivate players to achieve the intended goals of the game. Although serious games are gaining momentum in a wide variety of health-based applications, there are various problems that must be overcome before their use can become more widespread. Developing SGs is not trivial, requiring the collaboration of an interdisciplinary team with expertise in game development/computer science, education, and content (health professionals when considering health-based SGs) [5]. Furthermore, most SGs include fixed scenarios that cannot easily be modified [7]. Such a fixed-scenario approach can produce predictable, boring, and repetitive game experiences, which after several sessions, can negatively impact their effectiveness. In medical applications, SGs have been employed to help patients increase physical health, or improve cognitive functioning. For example, SGs have been applied in treating dementia, and used to increase social and emotional health in depression. [9, 20]. Serious games have also been used to treat autism spectrum disorder (ASD) [12, 21].

As will be discussed in the following section, currently, therapists are using several methods (including the use of technology and serious games in particular), to help children with ASD. Children with ASD have difficulty imitating behaviour and around their first birthday are less interested to look at people [13]. According to the social motivation theory [8], since children with ASD do not find social interactions rewarding they attend less and less to faces and conversations and lose interest in social stimuli. Over time they become less adept in recognizing peoples’ intentions, goals and social cues. For this reason many existing interventions are aimed at motivating children to attend to faces and speech to reduce the cascading risk of ASD children’ s indifference to faces and emotional cues.

Given the increase in autism diagnosis and the wide range of symptom severities, therapeutic efforts may benefit from the inclusion of SGs to increase their reach. The goal of this work was to develop an autism-specific serious game framework (ASGF) that therapists could use to create new SGs (or modify existing ones) in a simple manner (without having an extensive background and experience in software development), to supplement their therapeutic efforts. Although formal testing is required to gauge the effectiveness of the games developed using the ASGF, the aim of the ASGF, therefore, is to assist therapists to independently develop interactive and engaging scenarios that may have the potential to improve functioning in children with autism. We provide a tool to help therapists create games, by using the scenarios, to target repetitive behaviour and facial emotion processing. Following a review of previous work in Sect. 2, details regarding the ASGF are provided in Sect. 3. In addition, two games developed with the ASGF are also outlined. Concluding remarks and plans for future work are provided in Sect. 4. It should be noted that formal testing has yet to be conducted and thus although the ASGF was developed working closely with experts in autism treatment, formal evaluation of the ASGF and the two developed games is required to determine their effectiveness.

2 Literature Review

As previously described, currently, therapists are using several methods to treat ASD, including Applied Behavioural Analysis (ABA), the Treatment and Education of Autistic and Related Communication Handicapped Children (TEACCH) approach [19], and the Developmental Individual-Differences and Relationship-Based Model (DIR) [16].

The ABA intervention uses careful behavioural observation and positive reinforcement or prompting to teach each step of a behaviour. When a child performs the desired steps correctly, a reward, something (s)he likes very much, is provided, increasing the probability of positive behaviour in the future. By analyzing and modifying antecedent behaviour, and reactions or consequences, tutors can implement strategies for extinction of behaviours. Discrete trial training (DTT) breaks tasks into basic steps of increasing difficulty. Each trial is repeated until the child, under guidance, can accomplish the task successfully and autonomously. Successful ABA training involves early intervention and intensive treatment for as much time as possible (25–30 h per week), well-trained practitioners and consistent ABA applications, at home and at school [1]. Although this approach is generally recognized to be very successful, gains can be slow, and such gains may not be generalized. Children are often not motivated, and may demonstrate disruptive behaviours in order to interrupt or quit the sessions [11]. A variation of ABA, the Pivotal Response Treatment (PRT), which offers a more naturalistic approach, focuses on engaging with preferred toys and activities rather than artificial stimuli such as picture cards. In PRT, teaching is conducted within the context of natural play [11]. Recent work by Mohammadzaheri et al. [11], showed that the PRT intervention was more effective at improving social communication skills for children with autism than structured ABA treatment. The use of preferred toys in natural play settings appears to have created more interest, improved communication skills, reduced off-task behaviours [11].

Park et al. [14] examined the effects of TEACCH structured teaching on independent work skills in individuals with severe disabilities in a job setting. They demonstrated that TEACCH was effective, and performance gains were generalized into a different job setting. Carpente [3], combined the DIR/Floortime with Improvisational Music Therapy (IMT) while relying on the therapist’s ability to be creative, flexible, spontaneous, and emotionally attuned with the child. Four children with ASD, aged between 4–8 years old, participated in twenty-four 30 min classes for 13 weeks. Results using the Functional Emotional Assessment Scale (FEAS) indicated improvements in the areas of self-regulation, engagement, behavioural organization, and two-way purposeful communication. The DIR/Floortime Model enables clinicians, parents, and educators to construct tailored experiences matching the child’s unique skills.

Malinverni et al. [6], proposed an inclusive method for developing therapeutic gaming interventions for children with ASD, and defined a model, which integrated the expertise of clinicians, the interests of children, and the experience of designers. Following this method allowed for the creation of an engaging therapeutic gaming experience using a Microsoft Kinect-based game called Pico’s adventures, where an alien requires assistance to get back to his own planet by repairing his space ship. To assess the effects on participants, 10 children with ASD between the ages of 4–6 years were studied. They each played the game independently in the first session, and for the second session, they required the help of their parent to retrieve items. The third session involved collaborative play with the parent to land the ship, and in the fourth session, the children earned rewards by collaborating with another child participating in the study. The authors concluded that social interaction could be promoted by designing cooperative games where the resources are distributed between the players to achieve a common goal. Conversely, it was suggested that game mechanics that use physical contact should be avoided since they may hinder social communication [6].

Van Zijl and Chamberlain [18], proposed a generic platform for the development of computerized ASD therapy and research tools with the specific aim of providing a culture-free independent software on a generic platform. The platform was built with OGRE, an open source game engine in conjunction with the Enginuity engine, which was modified to allow for multiple threads, thus improving better synchronization between video, sound, and reaction times in the therapy tools, among others. They used a number of third-party libraries to manage tasks involving input from peripheral devices, playing audio, executing scripts and game logic, and rendering graphics. These included the OIS (Open Input System) for managing input, and Nvidia PhysX for physics and collision detection. The platform was tested using two therapy tools, and one educational game. The game was designed in partnership with a local school, and targeted 1st graders. The situations were based on encounters children may experience in the classroom, and if the child successfully completed the exercise, the game advanced. All text was available in multiple languages with audio for those learners who were not capable of reading at a satisfactory level [18]. The game was tested at a local school for a period of 20 days. Teachers’ feedback was positive, and they found the data capturing aspect useful, which captures the learner’s attempts and answers to exercises [18]. The second therapy tool, which tested social skills in a restaurant scenario, required the player to order food, pay for it, and find a place to sit. The authors’ intended goal to develop a generic open-source 3D virtual environment platform, was reached.

Caro et al. [2] proposed that exergames, that is, the coupling of video games with exercise, can offer children with severe autism a natural interaction using multisensory stimuli to keep them focused during motor therapeutic interventions. Traditionally, physical therapy relies on t repetitive limb movement, which requires the child with severe autism to aim for a visual target. Often, the visual stimuli are confusing, and this leads to aimless movements. FroggyBobby, is a game to improve hand-eye coordination in children with autism [2]. The goal of the game is to encourage participants to use their upper limbs to perform different eye-body coordination exercises. Children must help the frog avatar eat as many flies as it can. Their limb movements, which involve moving their right or left arm to reach a visual target, control the frog’s tongue. FroggyBobby employs the Microsoft Kinect sensor to monitor and track which arm a child is using at a particular moment. Psychotherapists found FroggyBobby easy to use during eye-body coordination sessions, and experimental results indicate that children with severe autism maintained their attention for the total duration of the therapy, reduced their aimless limb movements, and developed aimed limb movements after several weeks of use [2]. The authors also contributes qualitative and quantitative empirical evidence to show how exergames help in the development of eye-body coordination. The authors concluded that further research is needed to fade out the guidance that children receive when doing the eye-body exercises [2].

3 The Autism Serious Game Framework (ASGF)

The ASGF was developed in conjunction with neuro-scientists who served as content experts and specified the requirements of the facial expression identification and the response inhibition games. The ASGF graphical user interface (GUI) follows a What You See is What You Get (WYSIWYG) [15] approach whereby users (therapists) are able to drag and drop (and assemble) components associated with each game. The ASGF allows users to import 2D, and 3D assets (with or without animations), and sound assets to configure the different types of games. For example, a card matching game can be re-purposed to create a facial expression identification game, where emotions can be displayed on pictures, illustrations, or rendered images and 3D models on the cards. Additionally, a 3D computer generated space can be tailored to focus on inhibitory control, whereby the therapist can replace the 3D models of the characters in the game and have the child interact with characters or the scene employing different audiovisual cues. The goal of inhibitory control is to teach the child to inhibit responses by not responding to a given character or event when required. Inhibition is the ability to suppress an action which is irrelevant, no longer needed, and/or inappropriate [4]. Set shifting “...is the ability to change from a learned rule to a new rule in response to changing behavioral contingencies [10]”.

In addition to changing the look and feel of the games, the therapist can also change the difficulty, scoring, and obtain metrics to assess the player’s performance, which can be modified in a simple manner using the ASGF, providing flexibility to tailor the experience to the children instead of a one-size-fits-all solution. The ASGF has two end users, the therapists who can modify, create, and assess the outcomes from the games, and the child playing the resulting games. Feedback is provided from ASGF to both end users, for the therapist a report of the child interactions, and for the children, audiovisual cues associated to their performance. The system architecture is presented in Fig. 1.

Fig. 1.
figure 1

The ASGF system architecture.

Full size image

The ASGF enables the therapist to create a variety of games, which could raise the child’s confidence, motivate the child to follow through and succeed, and to potentially target different learning issues. Model driven engineering (MDE) provides the environment for domain experts to produce a serious game via modelling (either using language or visual tools) without worrying about the intricacy of game development [17]. It involves modelling the flow of information, the user interface (UI), designing a room layout (e.g., a 3D model of a room) or importing assets, and then using this information to generate code in the target language [17]. “MDE offers an increase in productivity, promotion of interoperability and portability among different technology platforms, support for generation of documentation, and easier software maintenance [17]”. The therapist is provided the option to focus on the problem and use the tool to create the solution, for example, a game that would teach autistic children to recognize faces. By using a planned event model, the therapist only has to model small sections of the code, and the objects connecting the assets, codes, and functions will know how to behave or run when they are built. The GUI for developing the games is presented in Fig. 2. The GUI allows for tailoring of the materials by importing pictures, videos, audio files, 3D models, along with difficulty settings, scores and timers within the design screen.

Fig. 2.
figure 2

Graphical user interface (GUI) components for creating the games.

Full size image

3.1 The Games

The ASGF allows for two types of games to be created: i) card-style (for face expression recognition games), and ii) non-card style (for inhibitory games). The two games we developed include customizations such as add a score label, a level label, graphic and text labels and the inclusion of timers. These allow for the inclusion of the necessary game mechanics such as displaying the current score on the screen. Descriptive tool tips and a help display give feedback to the therapist in build mode. The ASGF includes a splash screen with terminology and tutorial sections.

3.2 Card Style Game

This is a face matching game (with one main sample card and two or more choice cards), that targets reading emotions from the face. The user must click on the card that matches the main card. Optionally, a timer can be added that counts up or down. When counting up, it records the game completion time. When counting down, if the counter reaches zero, the child loses that match and depending on the winning and difficulty conditions set by the therapist, the child may repeat or continue playing with at the same difficulty level. There are 20 matches (trials) per block and the therapist can adjust the number of trials per block. The difficulty level sets how many correct responses are required to advance to the next level or end the playback level.

Face Emotion Matching Game: The child is asked to select the emotional expression that matches the main sample card (top card) For example, if the main card shows a happy face, the child is asked to identify the card that also shows a happy face (see Fig. 3).

Fig. 3.
figure 3

Face matching game screenshot.

Full size image

3.3 Non-card Style Game

The non-card style games aim to improve the ability to inhibit responses. In this game, a main character is moving on the screen in three different environments with customized assets and difficulty level provided by the game framework. It is worth noting that in these games the character moves and its trajectory can be altered to increase or decrease the difficulty when interacting with the objects in the scene.

Rabbit Adventures Game: In this response inhibition game, the goal is for the player to catch as many rabbits as they can. While the user is exploring their game environment, rabbits and wolves may cross their path. When a rabbit crosses their path, they must press the space bar to catch the rabbit, and if a wolf crosses their path, they should not respond (i.e., they should not press any key). If they respond and press the space bar, they will catch the wolf who will steal the previously captured rabbits. The variable parts include: replacing the main character, setting the character that crosses the path, setting a different key other than the space bar (e.g., the “A” or “C” key), the number that must be correct to pass the level, and adding more characters that will cross the main character’s path. The difficulty decreases the number of consecutive correct responses to pass the level. Figure 4 shows a screenshot of a game created with the framework.

Fig. 4.
figure 4

Rabbit adventures game screenshot.

Full size image

4 Conclusion

Previous works in the field of serious games for children with autism have focused on fixed scenario games that allow little customization and modification. Here we present the development of an autism serious game framework (ASGF) that allows for the creation and customization of serious games that are aimed to help children with autism identify emotions and also inhibit responses. Using the ASGF, we developed two games. In the first game (Face Emotion Matching Game), the user selects a facial expression in the first game that matches the main card to develop facial recognition skills. In the second game (Rabbit Adventures), while exploring their game environment (world), the user rabbits and wolves may cross their path. They must press the space bar when they a rabbit crosses their path and avoid pressing the space bar (or any other key), when they a wolf crosses their path. The ASGF is GUI based and does not require programming skills, allowing it to be easily modified by a therapist or researcher. The ASGF has the potential to deploy the games on mobile devices so that the intervention can be extended while the child is at home, and the data recorded can help us to better understand the progression and customization of the therapy.

Future work will involve conducting a series of user-based experiments to examine the usability of both the ASGF and the two serious games that we have developed and described here. Future work will also include user-based experiments to examine the effectiveness of the two games developed with the ASGF.