Keywords

1 Introduction

In any communication, humans generally express their intents effortlessly. Conversely, automatic understanding of social signals is a very difficult task for some individuals, especially for children with Autism Spectrum Disorder (ASD) [1]. Nowadays, distinct technological strategies have been used to try to mitigate the emotion recognition impairments that usually individuals with ASD present, mainly through the use of Objects based on Playware Technology (OPT) and assistive robots [2, 3].

Playware is defined as intelligent technology for children’s play and playful experiences for the user [2]. Henrik Lund suggested the term “playware” as a combination of intelligent hardware and software that aims at producing play and playful experiences among users [2]. There have been few related works in the field of OPT. The work developed by Henrik Lund [2] consisted in designing interactive tiles as a modular robotic playware with the goal of being flexible in both set-up and activity building for the end-user, allowing easy creation of games. A set of experiments was performed with a group of 7 children with ASD. The authors concluded that the results provided by the research offers an interesting novel research direction to investigate playware as playful tools for cognitive challenged children, giving the children a playful experience and automatically investigate the playful interaction to provide insight (and possible a diagnosis).

Concerning assistive technologies, assistive robots can be an exceptional tool for interacting with children with ASD. Research with assistive robots have showed that, in general, individuals with ASD express elevated interest while interacting with robots [4]. The research in this area have moved to using facial expressive robots with humanoid design, since it can promise a great potential for generalisation, especially in tasks of imitation and emotion recognition which can be harder if the robot does not present a human form [5,6,7].

Following this trend, the present work proposes a new approach of using both technologies (OPT and assistive robots) with the goal of promoting social interaction with children with ASD. None of the related works in the literature, to the authors’ knowledge, present a similar approach. Therefore, the present work consists in the development of an OPT to be used as an add-on to the human-robot interaction with children with ASD in emotion recognition activities. In order to evaluate the proposed approach, two pilot studies were conducted one with typically developing children and other with children with ASD. The purpose of these pilot studies was to evaluate both the game scenario and the OPT rather than to quantify and evaluate the performance of the child. The present paper is organized as follow: Sect. 2 presents the proposed approach; Sect. 3 shows and discusses the results obtained; the conclusions and future work are addressed in Sect. 4.

2 Developed Framework

The framework, depicted in Fig. 1, is composed of a humanoid robot capable of displaying facial expressions, a computer, and a new OPT called PlayCube. The Zeno R50 RoboKind humanoid child-like robot ZECA is a robotic platform that has 34 degrees of freedom. The robot is capable of expressing facial cues thanks to the servo motors mounted on its face and a special material, Frubber, which looks and feels like human skin, being a major feature that distinguishes Zeno R50 from other robots. Concerning the PlayCube, the present design approach consisted in developing an OPT that can offer a tangible experience and adapt to different games scenarios, as well as to provide immediate feedback. The concept of tangible interaction refers to enabled technological objects that can be physically manipulated [8]. A two-way Bluetooth communication protocol was developed to allow communication between the robot and the PlayCube.

Fig. 1.
figure 1

Experimental setup. Starting from the left bottom: PlayCube (the final prototype), computer, and the humanoid robot.

Full size image

The developed device, PlayCube (7 cm × 7 cm × 7 cm), has an OLED RGB display, Inertial Measurement Unit (IMU), a small development board (ESP32) that already has built-in Bluetooth and Wi-Fi communication, an RGB LED ring, a Linear Resonant Actuator (LRA), and a Li-Po battery. Additionally, the top face of the cube is a touch sensitive surface. Thus, interacting with the PlayCube just means, touching the physical object and manipulating it via natural gestures (e.g. rotation, shake, tilt, among others).

In order to design the play experience in all its fullness, feedback is a key feature in guiding the children through the play activity. Furthermore, the immediate feedback feature can be a very important factor specially when designing OPT devices for children with impairments [9]. Additionally, the type of feedback must be configurable for different children as some types of feedback can be unenjoyable for some individuals, e.g., in the case of children with ASD, in general, a sound feedback can be unpleasant for them [1].

Following this idea, both the humanoid robot and the PlayCube offer immediate feedback to the children actions. For ensuring immediate feedback for the PlayCube, a ring with a total of sixteen multicolour and equal spaced LEDs is used. A haptic driver is used to enable haptic control of an LRA actuator. These actuators can provide haptic and/or visual feedback to the user. Additionally, the display can also provide visual feedback. Concerning the robot feedback, the reinforcement that is given is based on a previous study [7] and consists in a combination of verbal, movement, and sound reinforcements (e.g. the robot says “Congratulations!” while waving its arms in the air).

3 Results and Discussion

In order to evaluate the present approach, two studies were conducted in a school environment. The goal of the pilot study with typically developing children was to detect the system constraints in an intervention session. Concerning the pilot study with children with ASD, the main goal was to verify if the system can implement a procedure that makes the children able to interact in a comfortable and natural way. The experiments were performed individually in a triadic setup, i.e., child-robot-researcher with a duration of five minutes. The activity played was the recognize game scenario where ZECA randomly performs a facial expression and its associated gestures, representing one of the five basic emotions (happiness, sadness, anger, surprise, and afraid), plus neutral. After, ZECA asks the children to identify the performed emotion. Then, the children have to manipulate the cube by tilting it back or forward in order to scroll through the facial expressions displayed on the cube. When the child selects an answer, by touching the top touch sensitive surface of the cube, ZECA verifies if the answer is correct and prompts a reinforcement accordingly to the correctness of the answer. Simultaneously, the cube provides visual and/or haptic feedback accordingly to the child’s answer. As quantitative measures, the number of right/wrong and no answer was quantified as well as the children mean response time, in seconds, and standard deviation (SD). It is worth to point out that if the response time exceeded 60 s, the child’s answer was accepted as not answering the robot prompt.

Since the work presents studies involving typically developing children and children with ASD, the following issues were ensured to meet the ethical concerns: the school which participated in the studies established a protocol with the research group and informed consents were signed by the parents/tutors of the children that participated in the studies.

3.1 Pilot Study with Typically Developing Children

A set of preliminary experiments were carried out involving eight children aged between six and seven years old. The results obtained with the eight typically developing children are presented in Table 1.

Table 1. Children answers to the robot prompts where C1 to C8 is the number of the child.
Full size table

Children 1, 4, 6, 7, and 8 showed an overall better performance. Children 2, 3 and 5 manifested more difficulties. Nevertheless, it is interesting to notice that the children answered the robot prompts giving a strong indication that in general the participants understood the game, and consequently interact with the robot by successfully manipulating the OPT (PlayCube). Furthermore, the mean response time for the unsuccessful answers was higher when comparing this value for the successful answers – 42.79(8.73) and 37.66(2.70) seconds, respectively. This higher value might be related to the children thinking and considering all options that they have available. Additionally, the children response time to the robot prompts decreased along the session – from 46.38(8.36) to 39.09(8.42) – indicating that the children responded faster to the robot prompts and were able to manipulate the OPT.

3.2 Pilot Study with Children with ASD

A pilot study with three high-functioning children with ASD (two females and one male) aged between 6 and 9 years old was carried out during four sessions in a school environment with the goal of evaluating the suitability and comprehension of the game scenario and the OPT. Analyzing each participant performance (Table 2), it is possible to conclude that child A performance improved much more along the four sessions when compared with the other participants. Furthermore, the mean response time of child A decreased along the session which can indicate that she understood the activity and how to manipulate the OPT in order to answer the robot prompts.

Table 2. Children answers and mean response time to the robot prompts during the four sessions (SD): RT – response time (s), RA – right answer, WA – wrong answer, NA – no answer.
Full size table

Concerning child B, she does not present an evolutive pattern in the first two sessions. However, in the last two the number of wrong answers decreased, maintaining the number of right answers. It is also worth to point out that the response time, for this participant, is longer. The results from the last two sessions suggests that this participant may need more sessions, which is consistent with the difficulties that these children present when discovering new activities or interests [1]. Even though child C had more difficulties answering the prompts, his performance improved in the last session and the child mean response time also decreased. When comparing the children mean response time along the four sessions and the number of total prompts (Table 2), it is possible to observe that in general the number of prompts increased along the four sessions as the children mean response time decreased along the sessions, considering that the session time is always five minutes. Additionally, it is worth to mention that none of the children that participated in this study abandoned the game. All children repeatedly touched gently the robot and in general manipulated correctly the OPT when prompted by the robot. Furthermore, as they answered the robot prompt they were particularly attentive to the cube feedback, lights and the images displayed in the screen for correct and incorrect answers.

4 Conclusions and Future Work

Recently, researchers are using technological tools such as OPT and assistive robots to try to mitigate the emotion recognition impairments that individuals with ASD present. Thus, the present work proposes a novel and hybrid approach for robot-assisted play. It combines the use of a OPT and a humanoid robot capable of displaying facial expressions with a serious game focused on improving the emotion recognition skills of children with ASD.

By analysing the results of the typically developing children it is possible to conclude that they interacted/responded well to the robot and understood the mechanics of the OPT and the game. One of the system constraints detected was the placement of the robot support that caused the children to always have to look up towards the robot face. In the pilot study with children with ASD this was corrected by placing the robot in a similar height to the children face. In general, the children with ASD reacted positively to the activity which can indicate that the developed approach allowed the children to interact in a comfortable and natural way with the system.

Future work includes further development and improvement of this approach. Additionally, a study will be conducted with a larger sample of children with ASD, aiming to understand if and how the presented hybrid approach can be used as a valuable tool to develop skills of emotional labelling by children with ASD.