Keywords

7.1 Introduction

The inclusion of technology to aid work with children with special educational needs has become increasingly popular in recent years. In general, technology allows for variety, attracts children's attention, increases their motivation, provides visual presentation of stimuli, helps develop additional skills that cannot be enhanced using conventional approaches. In addition, children are constantly immersed in technology and its use for skills development provides opportunities for indirect interventions.

Educational computer games—web based, for mobile devices, for desktop computers and even for the emerging fields of augmented and virtual reality can be successfully introduced in the assessment and therapy of children with autism spectrum disorders (ASD). These technologies are indispensable tools for aiding children with ASD acquire both academic and social skills.

7.2 Background

7.2.1 Autism Spectrum Disorder (ASD)

Autism spectrum disorder (ASD) is a set of neurodevelopmental conditions characterized by problems in the following two domains: (1) deficits in social communication and social reciprocity and (2) restricted and repetitive patterns of behavior, interests, or activities (American Psychiatric Association). The conditions included in the specter are characterized by the problems in these two domains, but the group is generally heterogeneous in terms of additional symptoms and their severity as well as in terms of etiology and the need for treatment. Usually, children with ASD receive treatment and support from a professional, sometimes from a team of professionals, more often individually, less often in a group. If the methods used are evidence-based practices for ASD therapy, this is often difficult to provide and, in many cases, too expensive, so the focus is on additional supportive strategies that can help achieve the results of the treatment without replacing it. These additional strategies may be aimed at the main symptoms or at additional, concomitant disorders such as anxiety, depression [2]; oppositional behavior [3], ADHD [4], language and speech disorders [5, 6].

7.2.2 Application of Information and Communication Technologies in Therapy

In recent decades, the use of Information and Communication Technologies (ICT) has grown exponentially and is now ubiquitous. Not only are personal computers a common household item but personal mobile devices have reached all corners of the world. Furthermore, smart gadgets are being developed thanks to cheap and easily accessible integrated electronic modules like ESP-8266 and ESP-32.

This opens new possibilities to help children and adults diagnosed with ASD to learn skills in a new way, tailored to their special needs.

There are several objectives for the use of ICT for ASD assessment and treatment:

  • to help with the sensory and cognitive difficulties of the patients.

  • to provide new ways for therapy and training.

  • to provide educational tools for the acquisition of academic skills.

  • to give the patient the opportunity to control her focus on a certain goal.

  • to entertain and provide fun.

More specifically, modern technologies can help improve the following skills:

  • learning

  • memory

  • attention

  • language and communication

  • reading and writing

  • problem solving

  • mathematical skills

  • cooperating with others/social behavior

  • first aid

  • working abilities

  • motor functions

  • skills to overcome frustration

  • self-control

  • social skills

  • understanding of social rules.

A benefit of using those technologies is that they can be designed in such a way as to adapt to the specific needs of a specific user. This includes not only customization, but also intelligent adaption during their use.

It is also important to note that the use of ICT is not limited to patients, but can also facilitate parents/guardians, caregivers, and professionals in the field of medicine, psychology, speech and language pathology, occupational therapy, and social work.

7.2.3 Types of Technologies

Various types or classes of technologies have been shown beneficial to children with ASD:

  • Websites—their main advantage is their accessibility. Browser-based text-based, 2D and 3D games as well as a variety of multimedia content can be developed and using responsive design played on an entire range of devices—from laptops to mobile phones or smart TVs.

  • Desktop applications—with the current dominance of the MS Windows operating system, desktop computers and laptops present a large, unified target for developing software. Several powerful 3D game engines like Unity and Unreal allow for the relatively easy creation of virtual worlds with impressive realism.

  • Mobile apps—the last decade has seen smartphones accessible to almost everybody. Touch screens, powerful processors and integrated GPUs allow for the development of any type of game or application. The market is dominated by just two operational systems: Apple’s iOS and Google’s Android.

  • Augmented/virtual reality devices—although still more expensive and less popular than the above-mentioned technologies, augmented or virtual reality devices are becoming more accessible with big companies, including Facebook, investing in such technologies.

  • Serious games—any type of game, including those listed here, whose primary goal is not ‘fun’ per se, but the gamification of an existing process.

  • Physical devices, robots, etc.—tangible objects that provide more than audio-visual experience and are a desired tool when dealing with children with ASD.

7.2.4 Serious Games

Probably one of the largest types of applications are the Serious Games (SG). Games are a form of entertainment that broadens our horizons beyond what is immediately conceived. All games share common characteristics such as established rules, rivalry, place and time, elements of fiction, elements of chance, pre-set goals and last but not least—gaining pleasure. Social games date back five millennia but it was the year 1974 that was marked as the most important year in the history of modern gaming as it was then that Dungeons and Dragons hit the market [7].

Presumably, the most common definition of serious games is “games that do not have entertainment, enjoyment, or fun as their primary purpose” [8]. Following this definition, serious games can be distinguished from video games by their design goals, because serious games have a main design goal other than entertainment. This means that the player is exposed to an environment that offers content based on know-how or experience. This experience relates to the specific context of serious games such as welfare, education, and health. Therefore, the authors [9] define serious gaming as an application with three components: experience, entertainment, and multimedia. The SG are also called learning or educational games and are a virtual simulation with a video-game structure whose purpose is to promote the development of important skills and strategies in order to increase the cognitive and intellectual abilities of the users [10]. These types of games are gaining popularity and have a share of the global game market [11]. Games can be used to promote thinking, visual attention, hand–eye coordination, visual spatial behavior, motor skills [12], social skill training [13]. Children respond positively to the introduction of gaming in learning and the majority of parents and experts have a similar attitude [14, 15]. Games can help children learn how to win and lose, to start from scratch, and to find various coping strategies. Multiplayer games teach children to take turns, wait for their turn, observe, and analyze other players’ behavior.

Studies have researched the application of SG for children with ASD [16]. Video games can be used to reduce the symptoms of ASD and although their effect is not significant, their help creates a positive attitude and adherence to treatment [17]. When working with autistic children, computer games can be beneficial for reducing anxiety [18], improving face recognition skills, [19] and teaching new skills, including social skills and emotion regulation [20].

In general, serious games are usually 2D, mobile based, iOS or Android games, fewer are web-based, and their primary function is a Picture Exchange Communication System (PECS)—where the users communicate their needs by exchanging pictures with their communicative partners [21,22,23], some specific social skill learning games like catching the bus or a general skill practicing application like drawing and playing music [24]. A key aspect of serious games is their ability to use personalization [25] and that could be greatly beneficial for children with ASD [26]. For example, mathematical tasks can be adapted to the needs of a specific user [27] and the adaptation of the content is beneficial to the learning process [28, 29].

Interactive robots in ASD research and treatment are still an area that is in its infancy. There have been a few attempts in this direction. The authors [30] initially developed a robot to mimic autism-like behaviors in order to better understand autism spectrum disorder. Their idea evolved and they subsequently researched its potential use in therapy sessions. They used an ADOS-2 diagnostic tool to program the robot to display ASD-type behavior when interacting with humans. The set of external stimuli was predefined.

Bartneck et al. [31] took a different approach. They studied the effects of integrating a powerful 3D game engine (Unity 3D) with sensors and actuators. The benefit of this approach is that the body of the robot can be made in many different ways. For their first prototype they used a Lego set, servomotors, and a USB webcam to make the robot “see.” They continued the experiments with a 3D-printable open-source robot called inMoov.

Augmented reality has still not gained much popularity but some early attempts at providing this technology to children with ASD showed it has potential [32]. They developed games targeting children aged 8–15. Even though they used a technology that was being phased out (Adobe Flash), the results were promising. For example, the user had to control an animated character by tilting a blue colored object left or right. With the current state of cameras, processing power, optimized for AI tasks, the potential of augmented reality is almost limitless.

Microsoft’s Kinect sensor combines multiple technologies for depth sensing and provides immersive gaming experiences [33]. A robot described by [34] used a Kinect sensor to imitate autistic children. Other uses of the Kinect sensor include games that allow children to play in teams [33].

Several attempts to use augmented reality (AR) for games have been made. A game called Augmented Reality Gamebook [35] contains several scenarios for stories that are audio-visually presented. In the newer versions of the iPhone mobile phone platform there are built-in sensors similar in principle to the Kinect sensor that allows for AR applications.

7.3 Aim of the Study

The aim of the study is to present the use of serious games for the development of social and communication skills in children with ASD by reviewing articles in this area that describe the development of such games and their use in working with children with autism. Given the diversity of technologies used and their different applications for children with ASD in terms of assessment, treatment, relaxation and entertainment, learning and mastering new skills, we want to emphasize technologies and games that have been tested and their authors present the positive impact on the development of social and communication skills, which are key to the overall functioning of children with ASD and to draw some conclusions on the use of serious games.

The review of the selected technologies does not claim to be a comprehensive and complete presentation of all the technologies in the field, but it highlights the contributions that could guide readers to embark upon including technology in their work, research, or practice.

7.4 Material and Methods

The results were identified through literature searches conducted in Central and Eastern European Online Library (CEEOL), EBSCO: Academic Search Complete, EBSCO: eBook Academic Collection and ScienceDirect, Academic Search Complete, Health Source: Consumer Edition, ScienceDirect, Scopus, Web of Science and Wiley Online Library for a time period between 2006 and 2021.

We searched those four major online databases, using the search terms (“autism” OR “ASD”) AND “social skills” AND (“serious games” OR “computer games” OR “computer-based” OR “mobile app” OR “portable” OR “software”) AND “education” AND “communication.” The search and selection of the results were conducted in January 2022. Forty studies of over 150 results were selected, part of these proving the need for and the benefits of adopting games for treatment of children with ASD. Eleven of the analyzed publications were selected as they present original technologies and developments by the authors. A brief summary of each of these publications is presented below.

7.4.1 Relevant Research

Play therapy is a method of treatment commonly used for children with ASD and other special educational needs (SEN). This technique allows children to optimize and improve their social and communication skills while playing in their own way [36]. Taking into account the facts that play is an essential part of all aspects of child development, and educational computer games are an effective support tool in teaching children without developmental disabilities, the authors [37] believe that games can also be used effectively in the education of children with autism. They analyze and describe the characteristics that dedicated educational computer games should have in teaching children with autism. The researchers [38] summarize articles with research results published over a period of 20 years and discuss appropriate theoretical models for designing serious games to improve the social behavior in people with ASD. The analysis of the pre-selected forty articles shows that a considerable number of studies in this area are aimed at children with high-functioning ASD or Asperger's syndrome. In addition to the general aspects of social behavior, more attention is paid to the recognition/naming and selection of emotions. The authors note a number of shortcomings in the design of existing serious games, offering improvements that can increase the effectiveness of games designed for people with ASD.

A survey among Bulgarian SLT specialists on the use of educational computer games in teaching practice involved 103 respondents who answered twenty-six questionnaires. The study shows the wide availability of games, which are not used excessively, as well as a clear need for structured policies, standards, rules, and methods for developing specialized educational computer and mobile game applications to support the work of SLT specialists in Bulgaria [14, 39].

Without claiming to be exhaustive, below we have listed some examples of games applicable to children with ASD resulting from our review of the literature on the usability of games and the results of their application.

Examples of games to help children with ASD.

The presented games were selected on the following criteria:

  1. 1.

    Games aimed at developing communication and social skills

  2. 2.

    Presentation of the technology

  3. 3.

    Year of development—after 2011

  4. 4.

    Tested on patients.

Example 1. ECHOES

The serious game ECHOES helps learning and communication in children with ASD. Children with ASD interact with a virtual character in specially designed social situations through a multitouch display and eye-gaze tracking. The interactions between the child and the agent include everyday situations at school and in the community that are part of the child's environment. The established social partner can act as a peer and as a tutor for children with ASD by supporting their communication skills and helping them to understand other people's language and interpret their non-verbal behavior. The training activities are organized according to the established principles for working with children with ASD [40].

Example 2. Emotiplay. Serious games have shown to be able to produce simplified versions of the social and emotional world around us. A study by Fridenson-Hayo and Lassalle [41], conducted an intercultural evaluation of Emotiplay, with the following participants: 15 children from the United Kingdom, 38 from Israel and 36 from Sweden, respectively. The game aims to teach emotion recognition (ER) in a fun and intrinsically motivating way to six- to nine-year-olds with high functioning ASD for a period of 8–12 weeks. Researchers report a significant improvement in participants’ performance in ER body language and integrative tasks. Thanks to Emotiplay, children acquire the ability to recognize faces with their emotions, voices, body language and their integration in the context of their social presentation in an effective, motivating and psycho-educational way. Parents also report that their children have improved their adaptive socialization, and that they show reduced symptoms of autism after using the SG.

Example 3. Lego Therapy. Levy et al. [42] study the impact of school-based Lego Therapy groups for adolescents diagnosed with ASD. The therapeutic program is based on the work of pre-trained school staff working with five adolescents with ASD, analyzing the intervention and its results on the duration of social engagement and the frequency of social initiations, reactions, and positive social behavior for five out of six participants. The results suggest that Lego Therapy groups may be an effective intervention at school to optimize social skills for adolescents with ASD.

Example 4. Learning emotions. The educational computer game developed by the Bulgarian scientific team: Stankova et al. [43], as an output of scientific project Pedagogical and Technological Issues of Educational Computer Games was introduced in the e-learning platform Moodle. It is based on the emotional intelligence development model and its main goal is to develop skills for recognizing, understanding, and naming emotions in children with ASD using storyboards and emoticons. The game consists of seven modules mainly based on plot pictures depicting emotions, distinguishing these emotions as well as working with emoticons and comparing them with emotion-related situations depicted in images. The game offers participants a specific combination of visual presentation and auditory perception of named emotions, which integrates the ability to upgrade emotion recognition skills by reporting errors, searching for additional information, right and wrong attempts in different modules of the game, as well as time for completing the tasks. To further evaluate the results, a survey for parents was used collecting information about the child's behavior, his/her interest in play and his/her motivation [44].

Example 5. SG Lego-like building blocks prototype. Barajas et al. [45] develop a serious game which, in the form of a game therapy tool, aims to improve social and cognitive skills for children with autism spectrum disorder. It consists of 2 components: a tangible user interface (TUI) and a graphical user interface (GUI). The TUI is made up of tangible Lego-like building blocks, complemented by electronic modules and representing a 3D virtual view of the board and the blocks. A total of nine children aged 6–15 participated in the preliminary experimental study. The proposed system showed an improvement in their social interaction through their joint play and exercise, and a reduction in solitary play was observed.

Example 6. ComFiM app—the developer team Ceccon et al. [46] of the ComFiM app aims to promote communication between people with autism spectrum disorders and to model it in an interactive and dynamic process. The game has several stages: first the player's knowledge is built while interacting with a virtual character, then at a later stage communication with another real participant in the game is stimulated using the skills gained so far. The results prove that the aesthetic experiences in ComFiM are in line with multigamers’ media, as the architecture of the game successfully realizes situations of communication between players.

Example 7. An SG prototype for emotional skills. The tool offered by Dantas et al. [47] is a free software developed using HTML ver. 5.2, JavaScript and Python 3, and aims to improve emotional skills by recognizing and interpreting facial expressions in children with ASD aged 6–13. The user's facial expressions are captured by a webcam on a computer or smartphone, and the application monitors and stores the level of attention, number of hits, number of errors and the time required to express an emotion. The app contains animations, 2900 videos and a set of images of 75 people expressing each of the six emotions based on the FACS theory.

Example 8. Multiplayer PAR game—the game, developed on the ideology of Silva-Calpa et al. [48] comprises four models of cooperation to contribute to the skills of social interaction of people with autism spectrum disorders. The collaboration models used in the application are specifically tailored for users with High-Functioning Autism and is itself assessed by criteria for the social interaction actions achieved by users during their joint play. There were 51 test sessions recorded with audio and video (17 sessions for each phase of the game) with five young people with autism who took turns as teams in the games where their roles changed. The results prove and confirm that each model of cooperation motivates the need for cooperation and encourages the creation of situations for social interaction between users.

Example 9. FILL ME APP—The mobile game application designed by the team of authors—Armas, et al. [49] and a therapist considered the needs of the specialist and based the solution on a technology platform so that the game results can be validated. The application contains four game modules for children aged 3–10 which stimulate emotions through different visualizations: (1) Emotion Bubbles: animated bubbles containing basic and mixed emotions. The player must release the bubbles that contain the desired emotion, which is visualized through text or audio. (2) Logical Sequence: Two consecutive images appear on the screen for a few seconds. Three new images are then presented to the player. The player must choose which image completes the sequence. When they choose the right answer, the image becomes a puzzle they have to solve. (3) Emotion Sudoku: A traditional sudoku game that uses cartoon images of emotions instead of numbers. (4) Humpty Dumpty: A game that allows the therapist to play with his/her patients. The results showing that 67% of patients improved their emotion recognition skills were achieved over a period of two weeks of using the mobile game application in every therapy session with each child.

Example 10. FaceSay™—The game FaceSay, developed by Tsai et al. [50], was initially tested on 25 children with autism who played it and achieved an average score of 14.8 on a facial recognition test, and a control group—an average of 12.8. The children with Asperger's syndrome—24 in total, achieved a much higher score with an average of 18.4 compared to 15.4 in the control group, all in the ages between 6 and 15. The sex distribution was forty-four boys to five girls. In an emotion recognition test, children with autism who played FaceSay™ scored an average of 6.53, while the average score in the control group was 5.2. The children with Asperger syndrome had an average test score of 8.7 compared to the control group score of 6.79. The computer tests were conducted twice a week for a minimum of six weeks for an average of 20 min each session.

Example 11. LIFEisGAME iPad prototype—the prototype game LIFEisGAME developed by a team of Portuguese scientists—Alves et al. [51], was tested during 15-min game sessions by 11 children with RAS, aged 5–15, 91% male and 9% female, 82% verbal with ASD and 18% nonverbal with ASD. All participants had experience in playing computer games, and among the most challenging emotions to recognize were fear, disgust, and surprise. A particularly popular and highly satisfying part of the game is the application Sketch Mee. Its high degree of interactivity gives players the opportunity to create their own facial expressions, with the ability to change eyes, eyebrows, mouth, and nose, and subsequently this becomes their avatar in the game. Parents offer enriching the prototype with musical stimuli, and therapists add a recommendation to include instructions for visual games.

7.5 Discussion

The analysis of the literature in the field of using serious games for children with ASD the following conclusions may be drawn:

  • The heterogeneous group of children with ASD is a challenge in terms of diagnosis, planning of therapeutic interventions, treatment, and inclusion in peer groups. In this sense, the application of technologies that are in themselves truly diverse would be an even greater challenge in terms of applicability, compliance with cognitive development levels, age, cultural and linguistic environment. Respectively, a personalized approach would be useful in the preparation and design of games to overcome at least some of these difficulties. This approach is already being applied and tested by some researchers who suggest the use of personalized elements like avatars and specific audio effects [26]. Each game developed for therapeutic purposes must be adapted to the potential, capabilities, needs and interests of the individual player, i.e., the content of the game should allow precise filtering on a number of parameters, such as age, gender, psychological state, disorder, etc.

  • Games are often applicable to children with good cognitive functioning. On the one hand, the group of children with ASD is heterogeneous in terms of the level of functioning, and on the other hand, given the good applicability of games for learning specific knowledge and skills, they would be a good approach to support children with lower functioning. More effort into developing games suitable for children with intellectual disabilities is needed [52].

  • Very often, in testing the effect of a game, single cases or small samples of children with ASD were used, which further complicates the statistical processing and conclusions. Particular attention in these studies should be paid to the heterogeneous nature of ASD and the selection of children who test the game should take into account many factors—age, level of cognitive development, level of social functioning, language, and cultural environment, if the game has such elements, the level of computer literacy of the children.

  • A serious challenge specialists who develop or use such technologies face is to devise reliable methods for assessing their effect. Also, it would be necessary to assess the content validity and reliability of these methods so that they can gradually enter the regular programs to support children with ASD, but only after checking their compliance with the goals of treatment. As mobile technology evolves, a new concept called “mobile learning” gets acknowledged. To effectively achieve their goals of helping people with disabilities, mobile applications need to be assessed in a structured way [53].

  • It is necessary to develop special methods for measuring the usability of games, and to develop and validate additional testing of the final effect of their application. In these methods, special attention should be paid to the satisfaction of children with ASD and their parents, as well as the possibility of using technologies for learning through fun and enjoyable experiences.

  • Standardization and verification of the assessment: at present there are no validated standardized tools to prove that a game is really “serious” or to measure how “serious” it is. Game authors and creators, together with software engineers, need to build a scientifically based game model from the point of view of user experience. What are the elements that make up a “fun” design and how can its effectiveness be measured? Currently, many developers/researchers offer several types of serious games. However, there is great difficulty in evaluating a game for clinical or classroom use.

  • Apart from being an additional therapeutic tool, serious games can also be used to teach basic skills in children with ASD—e.g., language, speech, reading, writing and math skills, as well as general knowledge. For this teaching to be successful, the games created should correspond to children's learning schedules and be made available to teachers and special educators, who will be able to supplement learning with games or replace basic learning approaches with serious games. This would be especially useful if the child experiences difficulties with concentrating, memorizing, or sustaining interest to what is taught in school.

  • Serious games for learning specific skills should be accompanied by instructions for use and short guidelines for their application in order to be easy to use by professionals and teachers.

  • Game developers should offer advice on the use of games and their capabilities.

  • Teachers and professionals working with children with ASD should be provided easily accessible training courses in the application of serious games and the opportunities they offer.

  • Balance: for the successful and purposeful creation of a game, a particularly crucial element is for the designer to achieve a balance between the fun element and the main goal of the game, which by no means should be just having fun. On the one hand, this means that the entertainment element of the game should be available, but it should not be predominant. Also, its exclusion in an attempt to achieve the main goal of the game is not justified, whether the latter is for training or treatment, or a combination of both, etc. The game should stimulate engagement, but it should definitely not deprive the player of the pleasure of playing the game itself, which is a means by which a specific goal can be achieved.

  • The parents of children with ASD and the users themselves should also be involved in the game development in order to check their expectations in terms of content, elements of entertainment, types of rewards and incentives for correct answers/actions, level of language complexity in the instructions, etc.

  • When testing serious games for assessment and treatment in children with ASD as well as for training in basic knowledge and skills, a greater emphasis should be placed on the subjective assessment by parents and children of the usability of the game, its complexity, entertainment value and emotional effect.

  • In developing games oriented towards social skills training in order to compensate for some of the difficulties children with ASD face, more attention should be paid to the specificity of the symptoms of ASD and professionals who are familiar with the functioning of children with ASD should be involved in the development of such games.

  • One should remember that when using therapeutic interventions, the main element is positive stimulation, providing fun and a pleasant experience. Games should bring not only positive results in terms of training and compensation of symptoms, but also a fun and enjoyable experiences for children and their parents

7.6 Conclusion

This article reviews previous studies, as well as developed and published serious games for children with ASD over the period 2006–2021. In order to avoid certain cognitive and behavioral stereotypes of players, such as repeated behavior patterns, parents, therapists, and instructors must be able to customize serious games for children with ASD. In addition, these specialized games provide tools for data analysis or visualization, taking into account the progress and development of the child's skills based on pre-set traceable parameters. Asperger [54] noted that the special interests and skills of people with ASD can reveal much about the nature of their condition as well as their potential achievements. He shared his observations that the conventional approach to learning is not applicable to them, as they create their knowledge and understanding through their own experience, accumulating practical knowledge and reflections. In this sense, video games are extremely attractive to many young people with ASD, who feel comfortable and at ease in the game reality. Previous research has shown the effectiveness of games in helping children with ASD to express their feelings, recognize their partner's emotions, and improve their level of engagement with others. Of course, games should not be considered a substitute for standard interventions, nor are they expected to be a panacea, but their presence in the daily lives of children with ASD certainly contributes to overcoming some of the many difficulties these children face.

The main purpose of the review was to assist the reader in understanding the various aspects of the possible positive impacts of using smart telephones, mobile apps, and computer games to practice different social skills as a therapy tool for youngsters with ASD. There has been much research and discussion on the development of skills of youngsters with ASD, including their ability for independent social interaction and performance. More research and testing are required to achieve an improved understanding of the use of technology in improving the social skills of children with ASD.