Keywords

1 Introduction

Autism Spectrum Disorder (ASD) is a developmental disability featuring social communication impairments, restricted interests and repetitive behaviors [4]. Based on the National Autistic Society (NAS, 2004), autistic individuals have impaired abilities in social interaction, social communication and imagination. Meanwhile, autism also brings inconvenience and lost to their families. One study [14] described how challenging life can be for a family having a child with autism. Therefore, autism becomes an important topic for the researchers from different areas. Prior researchers found social development delay of the autistic individuals results from emotional impairment, since they had difficulties to recognize social cues [2]. Frith [6] found autistic individuals feel it is difficult to synthesize information in a coherent way. Although researchers did not make an agreement on the causes of ASD, some studies [3, 10, 13] confirmed behavioral improvements of the autistic children through interventions. Based on the causes of autism and evidences that some interventions do work, we designed a curriculum to teach autistic children emotional concepts. The general method is to create a learning environment for autistic children to build connections between different modalities of emotional concepts, such as facial expression, body language and story scenario (Fig. 1).

Fig. 1.
figure 1

Research conceptual framework

Full size image

The research goal is to design robot mediated instructional activities for autistic children to learn emotional concepts, and validate this design. We have two research questions: (1) Is a robot able to make body movements with emotions embedded which are identifiable by general public? General public is defined here as the regular people with the age ranging from 18 and 60. (2) Is robot mediated instruction feasible and usable for teaching autistic children basic emotional concepts? The first question builds a base for the second one. Since only when the robot could express emotions accurately, they could be used to teach ASD children emotional concepts. We are the first team proposed the use of humanoid robots to teach autistic children emotional concepts in this study. We designed and developed a set of robot body movements with 4 basic emotions and a set of learning activities which could be used by teachers and therapists to teach autistic children emotional concepts. Based on the feedback from the related subject matter experts, the proposed instruction is engaging, and has potential to teach emotional concepts. With adaption, it can be applied to teach a broad spectrum of autistic individuals.

2 Related Works

Prior researchers tried different interventions to facilitate autistic children’s social development, such as human interventions, photographs, and video games [2, 9]. Recently, robots emerged as a new tool. Compared with other interventions, robots have several advantages. First, autistic children prefer more controlled environments. Meanwhile, human’s social behavior is complex and subtle, which is difficult for autistic children to follow [10]. It will be easier to create a controlled and simplified environment by using robots. Second, the learning objectives here are emotional concepts and the final goal is to communicate with real human. Thus, humanoid robots are better than other interventions, since robots could create a more human-like conversational environment.

Some researchers used robots to help autistic children improve their social skills [3, 10, 13]. Robins et al. [10] developed a social robot KASPAR. After intervention, autistic children could gaze, touch robots and co-present children, or even communicate with human by making the robots moving. Wainer et al. [13] used the same robot KASPAR to enhance autistic children’s collaboration skills by having them play video games with robots. Costa et al. [3] added more sensors on KASPAR and they found autistic children showed more appropriate social-physical interaction with co-present human after interventions. Social robots are also used to communicate, display and recognize the emotions, develop social competencies, and maintain social relationships [5]. These studies still have limitations [1], since they often provided free form activities and exposed participants in an environment where they have a robot. Usually there is a small sample size and results with a large variation. Therefore, it is difficult to make a final conclusion for intervention effectiveness. Besides, there is few previous studies targeting autistic individual’s emotional concepts learning and using robot body movements to help create emotional connections.

3 Study Design

3.1 Design Process

Ros et al. [12] shed lights on our study design. They used the framework of creative dance to help children build connections between the concepts in subject and body movement areas. It enhanced learner’s understanding of subject concepts. Their instructional activities had 4 steps, including Warm-up, Exploration, Creation, and Performance and Appreciation. We imitated this framework to design learning activity sequence. The learners should be diagnosed at a medium level of autism and have basic language skills. A robot named Bio will guide autistic children to learn 4 basic emotion concepts (Happiness, Sadness, Anger, and Fear).

We designed a 4-week long learning activity. One day in each week, ASD children will take a 20-min learning session (Fig. 2). The sessions in different weeks will follow the same structure (Fig. 3) and have three roles, autistic children (learners), a robot (tutor) and a human facilitator. First, all of the children will take a pretest through listening to short stories with emotions embedded. Then they will watch two robot movements and identify which movement better matches the emotion expressed by each story. The result will be used to measure their current performance and improvement.

Fig. 2.
figure 2

(designed based on creative dance [12])

Course structure chart

Full size image
Fig. 3.
figure 3

(designed based on creative dance [12])

Module structure chart

Full size image

In the warm up module, Bio will give a self-introduction, and briefly talk about the purpose of the class. Then, Bio will guide the autistic children to do basic body movements slowly. In the mini dance show and emotion identify module, Bio will perform a movement sequence with a specific emotion, for example, Happiness. Then, the facilitator will offer learners 6 facial cards that corresponds to 6 facial expressions (Fig. 4). After that, the participants should pick up one card to match the emotion in the robot body movement. If they select a correct card, Bio will say “Good job!”. Else, they should try again. Bio will tell them the correct answer when they answer incorrectly more than two times. In the mini storytelling and dance improvisation, Bio will tell a story which could express one special emotion, while the facilitator will show a story card to help understand the story. Then the participants will be required to perform a dance based on their understanding of the story. If the dance is incorrect, the robot will prompt the participants to dance again. In the end, the robot will perform one correct movement. In the Goodbye module, the robot will confirm participants’ performance.

Fig. 4.
figure 4

(source: the Grimace Project)

Facial expression cards

Full size image

3.2 Technical Requirements and Development Process

To design a robot tutor, which is required to speak specified sentences and make simple body movements to express specific emotions, we investigated robot dance videos [11], 3-D animation [8], and emotional robots [15] to extract behavioral features to express specific emotions. Then, we developed 4 robot body movements on a miniature-humanoid robot platform DARwIn-OP from ROBOTIS which has 20 degrees of freedom and a speaker, and used their Roboplus Motion software to design each movements to express Happiness, Sadness, Anger and Fear (Fig. 5). By integrating the control sequence, motion files, and pre-recorded sound files into a C++ code, human facilitator can press keyboard and the robot will speak and do the corresponding movement. We integrated these body movements and designed a set of learning activities. This process was demonstrated and recorded in the video below (Fig. 6).

Fig. 5.
figure 5

(demo videos available at https://goo.gl/zNH2po)

Screen shots from left to right in time sequence for the four robot body movements. (Color figure online)

Full size image
Fig. 6.
figure 6

(a demo video available at https://goo.gl/pDBeM1)

Instructional modules and process simulation

Full size image

4 Data Collection

This study used an unique mixed method to collect data from the subject matter experts before we implement the design among the autistic children. To protect the end users we will iterate the design based on the feedback we get from stakeholders, which is a typical method [7, 12]. First, we conducted an online survey to find whether the robot body movements developed could express emotions that could be identified by the general public. We sent the survey to the students and faculty in the researchers’ department and to their friends and families around the world. The participants watched the videos of the 4 robot body movements, and for each movement they selected one emotion which they think can fit the movement best. The reasons of their answers were also collected. The recognition rates and comments were used to measure and improve the design. Second, We conducted in-person interviews with 4 subject matter experts, including two researchers in the autism area (P1, P3), one industry designer who designs mobile applications for autistic children (P2), and one parent of the autistic children (P4). The purpose was to find out how is the idea of using humanoid robots to teach autistic children, as well as the feasibility and usability of the instruction. In the interview, they rated the simplicity of robot body movements design for autistic children, the feasibility and usability of the learning modules in a range from 1–5 scores (score 5 being simplest/most feasible/most usable, score 1 being least simple/least feasible/least usable). They also commented about the reasons of their answers.

5 Results and Discussion

5.1 Results of the Survey for Robot Body Movements Design

From the survey, we got 32 valid responses (\(n=32\)). They come from the countries in Asia, North America and South America, and represent different cultural background, with an age ranging from 18 to 60. The recognition rate is 94 % for Happiness, 91 % for Sadness, 81 % for Anger, and 78 % for Fear (Fig. 7). The survey reveals that Happiness is the easiest to recognize, while Fear is the most difficult. The key comments are summarized in Table 1.

Fig. 7.
figure 7

General public recognition rate

Full size image
Table 1. General public’s main comments on the 4 body movements design
Full size table

Happiness and Sadness got higher scores because of the clear design features. For Happiness, the robot body is open, positive and free with hands waving and eyes looking up. For Sadness, robot is bent down with banging head against hands and touching his eyes. However, there are some features making it confusing, such as the fast moving arms which make people also feel the robot is angry. For Anger, the arms are in a defensive position and eyes are looking into the air, while a few people feel it is bragging. Last, covering the eyes, defending itself make it fear. However, shielding its face makes people feel it is surprised. Thus, emotional expression is subtle which makes people have different perceptions. Overall we got high recognition rates, but more efforts will be needed to make the robot body expression more general.

5.2 Results of the Interviews for Module Instructional Design

From the interviews, we got 4 valid responses (\(n=4\)). P1 and P3 have 20 and 18 years of research experiences. P2 has 4 years of experiences on program design for autistic people, while P4 has been a mother of an autistic child for 7 years. All participants confirmed the instruction is fun and engaging, and the idea of using humanoid robot to teach emotional concepts is appropriate. It is a good way for learners to imitate emotional body movements and learn how to express emotions. The body movements are very representative of the emotions. The reward system is clear and easy to understand.

For the simplicity of the movements, the participants provided rating scores (ranging from 1–5) and comments. The features of Happiness are clear for identification (\(M=4.0\)). But, P2 thought Happiness might be understood by different children as different meanings. Sadness is simpler (\(M=4.75\)), but the movement is a little long and difficult for ASD children to follow. P4 also suggested making it a smaller gesture to match the real emotional expression in daily life. For Anger (\(M=4.38\)), P4 mentioned her son did the same gesture to express the same emotion recently. However, P1 thought it might be a little exaggerated. For Fear (\(M=4.12\)), they thought it looks like more coward or startled.

For the module design, the participants rated scores for the feasibility and usability ranging from 1–5, and also provided their comments on the reasons. The Warm-up module is rated with \(M=4.25\), for feasibility and \(M=4.5\), for usability, the Mini dance module rated with \(M=3.25\), for feasibility and \(M=3.5\) for usability, and the Mini storytelling module is rated with average \(M=3.63\), for both feasibility and usability. For the instructional design, the participants have several suggestions: (1) Tailor the content. It is currently designed for high functioning autistic individuals. This framework could be adapted for every ASD individuals. It can teach low functioning kids basic emotions and high functioning autistic adults subtler emotions. P2 suggested designing 4 or 5 levels of content for novices and find out where they are. It can start from less choices (less facial cards). Then change the content based on students’ performance. For the child who has better language skills, they can handle 6 facial expression cards. (2) Disconnection. Subject matter experts found there is disconnection between the displayed teaching of the emotions and the facial cards. Prerequisite trials should be added to verify whether learners have symbolic understanding first. Then add direct instruction to connect emotion, motion, and cards. (3) Follow up and transfer. It is suggested to add modules for learning generalization and transfer, such as taking Bio out to see if the learning can transfer. Currently, the emotion expressed is not exact like that in the real life, since expression in the real life is subtler. The design should use smaller motions to express emotion in real life.

The participants also provided suggestions toward the technical issues. For example, the wording of the verbal instruction is suggested to be short, brief, simple and concrete. To attract learners, we can design more interactions, for example, let learners operate the robot. It is better to keep the color of the head light the same or pair a color with an emotion. It is also suggested to simplify the facial cards. Some experts suggested using photograph of real children’s face, since it is easier for children to understand. Last, the quality of the speech is not good and need to improve.

Overall we find using humanoid robots for teaching is engaging but the instructional design needs to be improved. Tailoring the content for different autistic learners is strongly recommended. For some emotions in robot body movements, different experts have some different perceptions, which is consistent with the literature statements. But generally they made an agreement on the perceptions of the most movements we developed. There are still some aspects where they did not agree with each other, for example, using photos of real facial expression card or using simplified graphs. This might need further research. There is also inconsistency between the number of facial cards (6) and the types of emotion concepts (4), because we want to test this design on a higher difficulty level initially. They will be changed and kept consistent when we test it with the autistic children finally.

6 Conclusion and Future Work

Using humanoid robots with dancing capabilities to teach autistic children emotional concepts was proposed in this preliminary study for the first time. A set of learning activities were developed for teachers and therapists to teach autistic children. Results of the survey and interviews shows the emotions in the robot body movements we developed could be identified by the general public. The subject matter experts confirmed the proposed robot mediated instruction under the creative dance framework is appropriate and engaging. They also offered valuable feedback, such as tailoring the content to be applied to a broad spectrum of autistic individuals. Considering the robot’s capability, it also might be used for teaching dance for general public people. In the future we will integrate these feedback, improve the body movements and instructional design, and test the refined curriculum among autistic children.