Keywords

1 Introduction

Emotions play an essential part in our everyday social interactions as human beings, reason why being capable of identifying them is so important. According to Paul Ekman, humans have six basic emotions: happiness, anger, surprise, sadness, fear and disgust [1] – and the ability to understand and express them starts developing from birth.

Children with Autism Spectrum Disorder (ASD) have difficulties to identify and replicate emotions, as well as interpreting and controlling them. Opposite to the vast majority of babies who can understand these facial expressions by 12 months of age, these individuals have impairments in developing emotional responses and only by 5–7 years of age are they able to recognize happy and sad emotions. This difficulty is present in their life even as adults [2].

The recognition of emotions improves the social relation between children with and without ASD [3,4,5]. Tanaka et al. [6] reinforce this position, explaining that ASD is characterized by difficulties in terms of socio emotional reciprocity and that success in social interactions goes through the ability to recognize and interpret facial emotions in social context.

Mobile applications have already proved to be a successful aid for therapists and teachers in a learning environment, by facilitating the intake of information by individuals with ASD. Many studies using serious games have already been done, exploring different purposes such as education and therapy [7].

Serious games is the term associated with games that move beyond entertainment to deliver engaging interactive media to support learning [7]. Their focus is to facilitate the learning of important topics by making the entire process more appealing and fun. This way, the user is willingly engaged in an activity that they enjoy and the assimilation of knowledge is not so much a burden but more something that happens naturally.

This work is inserted in a larger project with the purpose of using robots and serious games to improve the social life of children with ASD. This way, a playware was included in the project, which has the function of motivating the user to engage in the activities that the serious game has to offer. The goal of the work presented in this paper is to evaluate the acceptance of the developed application (serious game and playware object) by the target group.

This article is divided in five sections: Sect. 2 addresses the materials and methods utilized, serious game is presented in Sect. 3, the obtained results are discussed in Sects. 4 and 5 ends with the conclusions.

2 Materials and Methods

The serious game is divided into three different activities. The first one is the replication/imitation of emotions by the user; the second is centered on the recognition of emotions and the last activity is a story mode with fifteen different scenarios, each with its own narrative. These stories were already validated by the work developed by Costa [8]. The main resource that was utilized during the development of this project is Unity. Unity is a common game engine that supports both 2D and 3D game development. It also has a free version for students and a large online community with members that are mutually helpful.

Playware is the term attributed to the use of intelligent technology that aims at producing playful experiences through the combined use of both hardware and software [9]. Following these lines, this project integrates a peripheral that makes use of a microcontroller and six buttons (each with a different emoji) to send the user inputs to the software by Bluetooth (Fig. 1).

Fig. 1.
figure 1

Playware object developed to control the game

Full size image

3 The Serious Game

The serious game has a simple and intuitive design. Due to its objective and target audience, the chosen language is Portuguese. The main menu allows the user to select the desired game mode or to access the options panel which has the possibility to regulate the output volume, start the Bluetooth connection with the playware object and to check the scoreboard.

The scoreboard registers the current activity that is being played, followed by the facial expression displayed, the answer provided by the user and finally how long it took the user to provide that answer, in seconds.

Furthermore, the application utilizes audio not only to tell the different narratives during story mode, but also as a positive or negative feedback depending on the user’s answer, as well to explain how to play each activity.

The three game activities implemented in the serious game are as follow:

  • Imitation: Initially, it is displayed a screen in which the user can listen to the game mode’s instructions on how to play it and upon any button press the activity begins. The user is supposed to be accompanied by a therapist or a teacher which has the function to check if the children is performing the activity correctly and register which emotion he/she is displaying. It also has the task to motivating the user to participate in the activity.

  • Recognition: As in the previous game mode, it starts with a screen which allows the user to listen to the instructions of how to play. In this case the therapist only has the role of being a motivating agent and ensuring that the user is playing correctly. The user is engaging with the playware and after seeing the emotion displayed on the screen, he/she is free to choose the button that more accurately matches it, after stating what emotion it is.

  • Storytelling: After listening to the game mode’s instructions the activity begins. For each scenario a unique narrative associated with it is played, ending with a question that prompts the user to provide an answer. This answer is the emotion that the main character would feel in the situation that was described. The child must select the corresponding emoji face in the playware object that reflects the emotional state of the avatar.

4 Results

Several tests were performed in the elementary school with typically developing children and children with ASD. The ethical concerns associated with this type of study were previously ensured: a protocol was formed between the parts, elementary school and university, and the informed consents were signed by the parents/tutors of the children that participated in the studies.

In the following sub-sections are presented the results gathered from the different phases of testing and validation, regarding the second activity mode – recognition of emotions.

4.1 Avatars Validation

After creating the different avatars representing each of the six facial expressions there was a need to validate the suitability and relevance of those images before implementing them in the application. For this purpose, an online questionnaire was developed and presented to different groups of teenagers and professionals with experience in aiding students with ASD, with ages ranging between 17 and 58 years old. The volunteers were asked to answer 6 pairs of questions (previously validated by an specialist in the field of ASD), each pair being composed by a question where they had to label the emotion that the avatar was portraying, followed by a second question where they were asked to specify how well that emotion was being represented by the avatar, in a scale of 1 to 5.

In total, the form had 114 submissions and each image had an accuracy of over 90% regarding the emotion that it was trying to portrait. These numbers correspond to fear – 94.7%, surprise – 93%, neutral – 95.6%, happiness – 99.1%, sadness – 98.2% and anger – 99.1%.

4.2 Tests with Typically Developing Children

The next step was to test the activity with typically developing children. The target group was constituted of six children and each played once. They were asked to look at the avatars and after stating what emotion it was representing they would choose the emoji they considered to more accurately represent it (recognition game mode). The accuracy of their answers was 94.4%. These children showed no difficulty playing the game or interacting with the playware object and stated that it was enjoyable, and they would do it again.

4.3 Tests with Children with ASD

The recognition activity was performed with children with ASD. The goal was to test if the children were able to interact with the playware object in a friendly way and could understand the game activity. The success of the activity was measured in terms of response time and number of correct answers. These two indicators allow extrapolating how the child is accepting the game scenario.

The test group was composed of six children but only five ended up performing the activity for more than one session, since one of them could not concentrate in the game and did not participate at all.

From the other five subjects (referred as child S1 to S5 in the following), with ages ranging between six to ten years old, the game data was gathered regarding if the answer was correct or not and the time they took to give that answer. Table 1 displays the average response time and the percentage of correct answers obtained per session and per child.

Table 1. Data gathered from the testing with children with ASD (S – Subject; AT – Average Time; AA – Answer Accuracy)
Full size table

Even though our main objective during this last testing phase was to find out how engaging this application was for children with ASD, it is also possible to notice improvements in their answers during the sessions. It is worth noting that S4 was very young and showed a significant attention deficit, which led him to ignore the game and focus on the avatars on the screen. S5 demonstrated a lot of interest but still had many difficulties interpreting emotions.

Everyone interacted well with the application (except S4), performing the tasks accordingly to the instructions (later, many were able to do so even without the therapist’s support) and expressed their fondness for the game.

5 Conclusions

This paper presents the application developed using a serious game and playware object to promote social interaction with children with ASD. The main goal of this study was to develop and validate an application intended to contribute to the development of pedagogical resources to be used by professionals and families in the support of children with ASD, and its impact validated in future work.

The system was first tested with typically developing children in order to evaluate the application constraints. Then, four sessions were followed with six children with ASD. The purpose of the game and the instructions on how to play it were very well assimilated and the playware object alongside the serious game revealed itself to be a very attractive and intuitive tool to interact with. The first tests with children with ASD allowed to infer that using serious games with playware object as intermediate in the interaction may be an adequate tool. In fact, children react positively, giving the answers by pressing the buttons. In general, the children improved their success rate when interacting with the game.

Future work involves further tests that are mandatory to validate if the skill (emotional behavior) was in fact acquired, with more children with ASD and a higher number of sessions, covering all the game activities. This would allow to extrapolate more accurate and reliable results regarding the use of playware objects with serious games in the intervention sessions with children with ASD.