1 Introduction

Language development is strictly correlated with cognitive development especially in the early stages of human life. According to Piaget, in preoperational stage corresponding to 2–7 years, knowledge is represented by language, mental imagery, and symbolic thought [1]. In case of communication problems such as hearing disabilities and autism spectrum disorder (ASD), sign language (SL) plays an important role as an alternative way of communication. Sign languages are visual languages composed by different sets and combinations of hand and upper-torso movements with various facial gestures. Learning sign language is of vital importance for hearing-impaired children from very early age, in order to interact with other individuals, and the efforts also support their cognitive development [2].

Game playing is another principle element for children’s intellectual development, improving their social and cognitive skills, which are necessary to communicate with other individuals [3]. Playing a game is an important activity for children both for their development and creativity [4]. The importance of the play activity in the children’s development and socialization is emphasized in several studies [5–7].

Based on this motivation, we started a project called “Robotic Sign Language Tutor” to develop interaction games with assistive social robots as a part of sign language education. The project focuses on using the assistive robot, which can recognize children’s actions and in return generate actions and feedback based on sign language within an interaction game scenario. We employ child-sized humanoid robots with high degrees of freedom (DOF) in arms and fingers in sign language based interaction games [8–12].

In this paper, we specifically investigate the impact of the embodiment of the assistant robot within the game. A web based game and its tablet version incorporating the video recordings of the real robots are tested with a group of adults, and children with and without hearing impairment and sign language acquaintance. The results are compared to that of the experiments with the physical robot platform employed within the project.

The reminder of this paper is organized as follows: The relevant studies are summarized in Sect. 2. The research questions and humanoid robots used in the framework of the games are presented, and consequently the interactive game designs are detailed in Sect. 3. The experiments and the results are discussed in Sect. 4. Finally, Sect. 5 concludes the paper with future works.

2 Related Work

2.1 Embodiment in Human Robot/Agent Interaction

In this work, we are interested in investigating the effect of different embodiments (physical robot, video of the robot via tablet and video of the robot in a web-based game) of an assistive robot using the subjective and objective evaluations of the participants within an interactive HRI study.

The “physical embodiment” of a robot can be expressed as the bodily presence with actual physical shape by having embedded sensors and motors [13]. Researchers have conducted studies regarding embodiment in HRI [13–15] and robots are employed in various levels of embodiment in different studies [16–18]. Dautenhahn et al. proposed minimal definition of embodiment that can be applied across animals and artifacts, and suggested relevant concepts and heuristics, which can contribute to studies of degrees of embodiment of robots interacting with social environments [18]. The study of Bartneck et al. investigated the effects of embodiment of a robot and their results showed that physical embodiment promotes social interaction [19]. In his recent study on the difference of embodiment of copresent robots, telepresent robots and virtual agents, Li also investigated if there was any change in the human behavior when the robot was telepresent or co-present [20]. The author reported that the majority of the studies in the literature favored a co-present robot more than a telepresent robot.

There are other studies investigating the benefit of using a physically embodied robot as a therapy robot. Wainer et al.  [21] stated that physical embodiment is crucial especially for therapy robots. The initial results in their study presented that the physical embodiment was preferred in task-oriented setting. They also stated that the presence of robot affected the user’s enjoyment. It is important to note that the participants of these experiments were adults. The study in [22] presented an experiment with children to investigate the embodiment effect on the user enjoyment when playing chess. Their results also presented that participants remarked physically embodied robots more enjoyable than virtually embodied agents. The study of Lee et al. [13] presented the results of two experiments, which examined the effects of embodiment and tactile communication. They claimed that physical embodiment is important in social interactions between robots and human. Physically embodied robots are able to interact with individuals using their sensory inputs and by processing these inputs, they can instantaneously generate auditory, visual and/or tactile outputs (feedback). Also it is expected that a social robot, in order to be able to engage in a playful interaction with a child, would require a certain degree of embodiment [23].

2.2 Sign Language

A sign language tutoring system should include a robust sign/gesture recognition and sign realization modules. A survey conducted by Parton [24] summarizes different researches for recognition, translation and generation of sign language, mainly for American Sign Language (ASL). Some researchers have studied the gesture recognition for various sign languages by using different methods [25–27]. Sign realization studies are comparably fewer than sign recognition studies because realization requires two human-like hands and an upper body for both manual (hand configuration, orientation, placement or movement) and non-manual (posture of upper torso, head orientation, facial expression, and gaze direction) components [28]. There are studies for realizing sign language by developing robotic hands [29, 30] as well as utilizing avatar based sign language tutoring [31, 32].

The idea of using humanoid robots within different interaction scenarios especially games with humanoid robots are also studied. There are several user studies presented on non-gesture communication through imitation based interaction games with humanoid robots and human participants [33–37]. In our previous studies, we also designed different interaction games with humanoid robots including physical embodiment and web-based applications [8–11].

3 Interactive Signing Games

This study explores the impact of embodiment of the humanoid robots in teaching sign languages to children with verbal communication impairments. The humanoid robots are employed within interaction games using some selected words from Turkish Sign Language (TSL) repository. In this article, we will create experiments to verify a number of hypotheses, which we believe are important for researchers, who design and implement games for interactions with disabled people.

3.1 Robotic Platforms

We have two different robot platforms with distinct differences in appearance and physical capabilities which are available for sign language tutoring. The reader should notice that the comparison of robot tutor versus human tutor or virtual human tutor (avatar) is out of the scope of this study. The comparison of two robot platforms was discussed in [12] in details. The discussion involving interaction games within sign language tutoring was presented in [8] in details.

3.1.1 Robovie R3 Humanoid Robot

The robot used in the experiments is a specially modified version of Robovie R3 humanoid robot. Standard R3 platform is 1.08 m tall, weighs 35 kg and has 17 DOF (2*arms*4, neck*3, 2*eyes*2, wheels*2). Our version of R3 robot has 12 additional DOF in wrists and fingers, 29 DOF in total. Besides, it has a Light Emitting Diode (LED) mouth to express gestures better. As it has five fingered hands, it is easier to implement accurate signs with fingers moving independently. It has a small platform on the chest, which is used to integrate an ASUS RGB-D camera for gesture recognition. This camera can be replaced with a touch pad tablet according to the scenario of the game. Since R3 is a child-sized humanoid robot, it is especially convenient to be utilized in interactive games designed for children since with the current form of the robot’s embodiment, children consider it as a peer [36].

3.1.2 Nao H25 Humanoid Robot

The Nao H25 robot has a height of 0.57 m, a weight of 4.5 kg, and is a system with 25 DOF, two cameras, sonar sensors, and force sensitive resistors. Furthermore, Nao robot provides two loudspeakers and programmable LEDs around the eyes. In this study, eye LEDs are used for giving nonverbal feedback to children. The Nao H25 robots have hands, and three dependent and movable fingers to implement most sign language words. They are suitable to be used in interaction games due to their expressive face, small size, compact shape and toy-like appearance.

3.2 Interactive Sign Language Game Scenarios

In our experiments, participants played two interaction games with different embodiment conditions: (1) physical and (2) virtual. Both games are based on the generation of selected signs by the robots. The participants were asked to recognize the signs and to reply the robot in return. In the virtual game, both robot platforms are employed, whereas, in the case of physical robots, only R3 is used due to time and space constraints (tests with physical robots take longer time and require special laboratory setup). Readers could refer to [12] for a detailed comparison of physically embodied Nao and R3 robots.

The signs used in both games are selected from the most frequently used, daily signs. Besides, the kinematic and physical limitations of both robot platforms are considered in the selection of signs. The realization of signs are based on the visual TSL dictionary [38]. In the virtual game scenario, the recorded videos of the robots performing selected signs and of the human signer from the online TSL dictionary [39] are used. After collecting information from the interviews done with sign language tutors and teachers, the scenarios are designed. The final experimental setups are shaped after a series of user tests with adults and children having typical hearing (non-disabled) and hearing impairment. These earlier user tests gave us insight about:

  • the selection of the robotic platform to be used for the teaching of the signs [12],

  • the number of signs that could/should be taught/revised in one game session [10],

  • the maximum number of participants, who could/should interact with the robot [12] in one session (found out to be 1–3 participants),

  • the adjustment of the robot’s feedback, whether it should be vocal/motional as signs or facial expressions, to the participants about their performance throughout the game [40].

Especially in [40], we have tested the game using a physical R3 robot with hearing-impaired children, who were fluent in TSL and compared the results with the ones obtained from the tests conducted with plain videos of R3. The results were promising, therefore in this study we attempted to test the embodiment effect with a virtual signing game application. We intend to explore if there is any difference in the outcome, if we present the videos in a game-like framework and increase the level of interaction by giving feedbacks implemented into the game structure. Games are detailed separately in the following subsections.

3.2.1 Sign Language Game with Physically Embodied Robots

The interactive game had three phases: introduction, game play, and testing. The introduction and testing phases were done in groups of three participants, due to time and place restrictions. The play phase enabled the participant to have a one-to-one interaction with the robot. The first phase (introduction) acted as a familiarization phase for the participants fluent in sign language and as a training phase for the ones with beginners level acquaintance. Most of the participants were already fluent in TSL, but a familiarization session was still performed to show them how the robot generates these signs (it is not 100 % similar to human due to kinematic constraints). In this phase, participants showed colored flashcards of signs to the robot, and robot generated the matching sign as in Fig. 1.

Fig. 1
figure 1

Setup for the experiments with physically embodied robots

Full size image

In the second phase (play) each participant played with the robot alone, one-to-one. Here the robot generated the signs randomly, and the participant showed the matching card to the robot. Consequently, the robot gave non-verbal feedback (smiling face or neutral face) in return [40].

In the third phase, a group of participants were seated facing towards the robot, apart from each other and were asked to fill out a paper based test, in which the answers were represented with the same icons on the flashcards. The reason of virtually depicted test was that the writing performance of the hearing-impaired children is in general poor due to the differences of written spoken language and sign language. Detailed information about this game could be found in [40].

3.2.2 Sign Language Game with Virtual Robots

In this scenario, participants played with virtually embodied robots Nao H25 and Robovie R3. In the virtual game scenario, two different sign game applications have been developed.

First application consists of a beginner version of the game. This application includes little number of signs and a training session with the aim of teaching signs from TSL (Fig. 2). It also gathers the subjective evaluations of the user for different robot platforms used to generate the signs within the project. This game is designed to be played by the users with no prior sign language experience or just beginners level sign language knowledge, especially children of early age group.

Fig. 2
figure 2

Sign language game for beginners. Training session consists of three videos per sign (Human videos are taken from http://www.cmpe.boun.edu.tr/tid/, whereas the robot videos are created by us.)

Full size image

The second application was for the users with advanced sign language knowledge. This application was used not to teach signs, but to verify the recognition rate of the signs generated by different robot platforms. This game had no training session and consisted of more questions. The virtual game was very similar to the last phase (testing) of the game with the physical robot. The flashcards/visual icons used in the game were replaced with the multiple choices for the tablet/web based test, where instead of watching the robot generating the signs, the participant watched the video of the physical robot’s signing. Likewise, instead of showing the cards to the robots, or crossing them on the paper based test, the participant clicked the matching icon. Participants were not allowed for repeats. The videos lasted for 4.5 s in average.

The virtual game also had three steps. In the first step of the game, the application collected user’s personal information such as name, gender and age. Gender selection was done with a boy or girl figure. After the user entered her/his information, the game started. In the beginner version of the game only five signs were employed. Every sign was demonstrated by the human signer, R3 and Nao robots respectively. At the end of the videos for each sign, the participant was asked to evaluate, which robot performed most similar to the human signer by clicking on the robot icons. The aim of this phase was to evaluate the robots while teaching the participants the selected signs. After this training part, the participant was tested with robot videos for every sign in the training session. The robotic platform used for signing was randomly chosen for each sign and the user took part in a multiple-choice test by watching the robot video. The options were displayed as the icons and the names of the signs. The participant was expected to click to the icon of the related sign. After the test for all signs was finished, the application asked the participant, which robot the participant liked more.

The advanced version of the game was particularly appropriate for the participants with sign language knowledge. The motivation for incorporating this version into the sign language game framework was because we wanted to verify the success of different robot platforms in generating the signs form TSL and evaluating their effect in assisting sign language tutoring. In the advanced difficulty, different from the beginner version of the game, the participant was able to choose which robotic platform (R3 or Nao) he/she wanted to play the game with. In the advanced version, the game consisted of ten questions. For every question, the user was able to select one of the three options or intentionally choose not to answer the question and pass to the next one (Fig. 3).

Fig. 3
figure 3

Sign language game for advanced users

Full size image

In both beginner and advanced level of games, a screen appeared to demonstrate the performance of the participant. As a last step of the game, a log file was created on the local storage of the mobile device including user information and game data for each level. The data was compared with the correct answers once the game was over and was also mailed to the evaluators. More detailed information about this virtual game could be found in [41].

4 Experiments and Results of Interaction Games

In case the participant has prior sign language skills (beginner or advanced level), the tests cannot be carried out fairly because the participant already has an idea of the human signing and will evaluate the signs of the robots with a bias. However, the participants without sign language information might evaluate the performance of the robots more objectively compared to the human signer when asked the question “which robot signs more similar to the human signer?”. Based on these, we will come up with several hypotheses (H) to be verified and research questions (RQ) to be answered in two separate experiments.

4.1 Experiments with Participants Having No Acquaintance of TSL

4.1.1 Research Question

RQ1: Will there be a significant impact of the participant’s age on the subjective and objective assessment of the signing ability of the robot platforms in comparison to human signer, if the participants have no a priori sign language acquaintance?

4.1.2 Participants and Experimental Setup

In order to assess whether the the signing quality of different robot platforms is similar to that of the human signer, a set of tests was performed with two groups of participants, who had neither any prior knowledge of TSL, nor any experience with the robots:

  • Adults First group was composed of 16 adults with typical hearing (no hearing impairment) from computer engineering and related background. The participants were all male and had an average age of \(28.2\pm 3.5\).

  • Children Second group was composed of 23 children (16 girls, 7 boys) with typical hearing from the age range 9–12 (average age: \(10.04 \pm 1.0\)).

Both groups were tested using a basic sign set consisting of five signs (meaning “mother”, “spring”, “big”, “table”, “black”) within the framework of the game using virtual robots, preceded by a short training session.

4.1.3 Results

Although none of them had any prior sign language education, the adults recognized all the signs correctly (Table 1), whereas the children’s number of the correct recognition were less.

Table 1 Results in the virtual game for beginners with adults and children
Full size table

Inspired by the high recognition rates of the signs, the results of adults and children were evaluated regarding their test score. The test score for each participant was computed by the number of the correctly recognized signs divided by the total number of signs. The mean test score for the adults was 100 %, since they answered all the questions correctly, on the other hand the mean score for the children was 78.26 \(\pm \) 16.96. A t test performed on the adults and children’s scores showed that there was a significant trend in the scores of the given sample (\(t(37) = 5.105 , \;p=0.0001 \;with p<0.001\)).

The mean response time to each question (sign) for the adults was \(6.29\pm 1.74\) seconds whereas it was \(5.93 \pm 1.25\) s for the children. Mean values and standard deviations of participant’s response time are displayed in Table 2 with the results of t-tests. The p-values revealed that there was no difference on the response time regarding their age difference.

Table 2 Mean and standard deviations for each sign, and results of t tests
Full size table

As to the participant’s preference of the robot platform, the adult participants reported that they found R3’s signing more similar to human signer than Nao robot, except one sign “table”. According to the post-test surveys 63 % of the adult participants preferred R3’s signing more than Nao (37 %). The children also stated at the end of the game session that they had found R3 more likeable than Nao (65.22 % R3, 34.78 % Nao). The results of the post-test survey also indicated that the children had liked R3’s signing most (66.95 % R3, 33.05 % Nao). Analyzing in details, the children also liked Nao’s signing of “table” more than R3, again similar to adults’ results.

It is important to note that both group of participants favored R3’s ability to imitate a human signer for the four same signs, however they found Nao’s performance more similar to a human signer while performing the sign “table”. This is due to the difference of hand motion. Nao’s hands are not capable of generating complex hand motions like R3, therefore the difference between R3 and human is more than Nao. The preference of the participants and the similarity in both groups can be seen in Fig. 4.

Fig. 4
figure 4

The preference of the children and adults with respect to the ability of both robots to imitate human signer. The black and dark gray bars indicate that R3 is favored by the children and adults, respectively, whereas the light gray and white bars represent that the children and adults have fancied the Nao robot better

Full size image

The results about the preference of participants supported our hypothesis on the evaluation of the high similarity of the signing of the different robot platforms to that of the human signer (answer to RQ1).

4.2 Experiments with Participants Having Acquaintance of TSL

4.2.1 Research Questions

In addition to RQ1, we expect to find answers to the following questions in the studies performed with hearing-impaired children:

  • RQ2: Will there be any significant disparity in the recognition rate of the signs mainly caused by the physical differences of the two robot platforms in the game involving the “virtual embodiments”? The motivation behind this question is that the observable physical differences between the robots might be minimized, when the videos of the robots are shown to the participants. Therefore, we expect to see no effect of the robot platform on the recognition rate either.

  • H1: Usage of physically embodied robot will increase the recognition rate of children compared to the rates obtained when virtually embodied robots are used.

  • RQ3: Will the level of sign acquaintance of the children have a significant impact on H1, namely will the effect of physical embodiment of the robot change based on the children’s signing skills?

4.2.2 Participants and Experimental Setup

After the pilot studies with the adults and children without any acquaintance of TSL, the web version of the virtual game and the interaction game played in the presence of physical R3 robot were tested with 31 hearing-impaired children at the Cognitive and Social Robotics (CSR) Laboratory in Istanbul Technical University.

Two groups of hearing-impaired children having different age and prior TSL knowledge were tested with both setups; i.e., the setup with physical R3 robot and a web version of virtual game with R3 and Nao robots. The profile details of each group are as follows:

  • Group 1 The first group was composed of 21 hearing-impaired children (11 girls and 10 boys), mostly with cochlear implant from the age range of 7–11 (average age: 8.71) with beginners level sign information. They were able to hear and communicate verbally, though limited. The participation to the tests was not obligatory, it was based on the volunteering principle. Therefore, 18 children among them preferred to play with virtual Nao, whereas 16 children preferred to play with virtual R3. There were 13 children who had volunteered to play with both robots virtually and 10 of them had volunteered to be tested with the physical R3 robot priorly. And also, a 16-year-old girl with additional mental disorders took part in the tests performed with this group. Her results were promising and she was motivated to play with the virtual and physical robot but her results were not included in the analysis.

  • Group 2 The second group was composed of ten children (five girls and five boys) of 9–16 years (average 11.8). They were advanced level sign users, and had severe hearing impairments. These children had a chance to play with both game setups (physical and web-based) and five of them had also the chance to play the virtual game with both robots: In summary, six games with R3 and nine games with Nao were played virtually.

All hearing-impaired children played the sign language game for advanced users. In both cases, children had a chance to pass the questions if they were not sure about their answers.

4.2.3 Results

The overall results per sign (percentage of correct answers for each sign) for the virtual game using both robot platforms are displayed in Fig. 5.

Fig. 5
figure 5

Correct rates of the participants in the experiments with the virtual sign game for advanced users. Black and dark gray bars indicate the correct rates of the beginner participants, who played with R3 and Nao, respectively. The light gray and white bars represent the correct rates of the participants with advanced sign language skills, who played with R3 and Nao, respectively

Full size image

The children of beginners level (Group 1), preferred to play with Nao first (12 children played their first game with Nao and 9 with R3). Yet, their average error rates are smaller in the games played with R3 than Nao in the first games (average error rate with \(R3=2.8; Nao=4.7\)), and total error in both games (\(R3=3.1; Nao=4.3\)). This is in accordance with the children’s subjective remarks about the robots, such as “They find Nao cute, as a toy, and R3 is more like a peer or tutor, whose actions are more understandable” [12].

The participants were evaluated by their test score and the comparison between the variables and their effect were performed regarding the computed test score of the participant.

In order to explore if the difference between the virtual robotic platforms had any effect on the participant’s test score, a t-test was performed on the results of the children from the first group. There wasn’t any significant difference in their scores with Nao (\(\mu =57.22, \; \sigma = 23.96 \)) and with R3 (\(\mu =68.12, \; \sigma = 17.21 \)), given that \(t(32) = -1.506, \;p=0.142\). Furthermore, a paired t test was performed on the test scores of the children who had played with both robots virtually. The results were similar to the independent-samples t test (\(t(12) = -1.091 \; p =0.297\)).

The results of the t test performed on the scores of the second group showed also a similar result; the test scores of the children who played with virtual Nao (\(\mu =80, \sigma = 25.5 \)) and the scores of the ones who played with virtual R3 (\(\mu =78.33, \; \sigma = 32.51\)) weren’t influenced by the difference of the robotic platform (\(t(13) = 0.111, p = 0.913\)). The paired t-test performed on the test scores of the children from the second group who had played with both robots virtually; revealed also a similar result to the previous one (\(t(4) = -0.497 \; p =0.645\)).

Both results support our hypothesis that there is no significant difference on the test scores of participants because the difference between the robots is minimized due to the use of their video (answer to RQ2).

Figure 6 focuses on the comparison of two different embodiment conditions: physical and virtual R3 robot. Inspired by the high recognition rates of the children with beginners level knowledge of sign language (Group 1), who also played with physical R3 (mean test score of 97.25 \(\pm \) 7.78), a t-test was performed on the test scores computed based on the nine common signs (“mother”, “to throw”, “spring”, “mountain”, “baby”, “I/me”, “big”, “to come”, “black”; the sign meaning “table” was excluded in the interaction game designed for the physical R3 robot) to see the effect of the physical presence of R3 robot. The results showed that the difference of embodiment had contributed to a significant difference on the recognition rates of the participants (\(t(22)=4.522, \;p=0.00016 \;with\; p<0.001\)) and consequently on the performance of the children with beginners level sign language knowledge, who had both played with physical R3 robot and the game with virtual R3 (where the mean recognition rate for the signs was 72.22 %). These results support our hypothesis on the effect of the physical embodiment (H1) for the first group.

Fig. 6
figure 6

Correct recognition rates of the participants in the experiments with the physical and virtual sign game designed for advanced users. Black and dark gray bars indicate the correct recognition rates of beginner and advanced level participants, who played with physical R3. On the other hand, the light gray and white bars represent the correct recognition rates of the beginner and advanced level participants, who played with virtual R3

Full size image

On the other hand, the test results didn’t reveal a similar difference on the test scores of the children who were fluent in TSL (Group 2) and played both with physical R3 and the game with the virtual R3. The mean test score with physical R3 robot was 93.4 \(\pm \) 7.7, whereas the mean dropped to 78.33 \(\pm \) 32.51 with virtual R3. In order to see whether there was also a significant difference in the recognition performance of the children with advanced knowledge of sign language due to the physical embodiment, another t-test was performed. In the analysis, the results for the sign meaning “to come” was not taken into consideration because the children reported that they had difficulty to match the selected icon to represent this verb with its semantic meaning even though they did not have any difficulty to recognize the demonstrated sign during one-to-one interaction sessions. Therefore, the evaluation of the recognition performance of participants was done with respect to the 8 common signs (“mother”, “to throw”, “spring”, “mountain”, “baby”, “I/me”, “big”, “black”). The t-test performed on the test scores computed based on the recognition success of 8 signs showed that the physical embodiment of the robot hadn’t any significant effect on the participant’s performance compared with the virtual robot (\(t(14)=-1.431, \;p=0.174\)) if the participant was fluent in TSL.

It is noteworthy to point out that the results of the first group support our hypothesis that the physical embodiment is important (H1). And the results of the second group combined with the first one, show that the prior knowledge of sign language has an influence on the significance of physical presence of the robot (answer to RQ3).

4.3 Discussion

At the end of the study, the children who participated in the tests had also shared their opinion on the generation of each sign performed by the robot. The children with advanced sign information reported that they liked the word “I/me” most when it was performed by either one of the robots; the generated sign is depicted in Fig. 7a, c. On the other hand, children who do not know signing liked “table” using Nao and “mother” using R3 the most (Fig. 7b, d). The children of advanced level also preferred to play with Nao first (eight children played their first game with Nao and two children with R3). This is likely because of the fact that during the demo event day, they already played a different game with the physical R3, and wanted to play with a different robot. These children had same average error rate in both robots. Yet their error rate was significantly lower with R3, when they played with physically embodied robot platforms, instead of the virtual robots [12].

Fig. 7
figure 7

Nao and R3 performing the most favored signs by test participants. a Nao signing “I/me”. b Nao signing “table”. c R3 signing “I/me”. d R3 signing “mother”

Full size image

The children, who know signs suggested that the word “table” is not correctly signed by the robots (this specific sign is depicted to be generated variously in different sources), and because R3 has more distinguishable gestures due to its larger hands and independent fingers, this difference becomes more apparent in R3. The children with advanced sign language information were successful in recognizing the signs from the application and reported that they had fun to play with the application.

As a conclusion, children of different levels of hearing impairment and sign language acquaintance were motivated to play both with the physical robot and the virtual robot in the applications. The results seem promising and these applications can be useful to support sign language tutoring in the long term. As stated in [8], children of younger age, without sign language knowledge benefit more from the physical robots than the videos and 2D applications, yet, the use of game environment and robots [10–12] still increase their recognition rate of the signs compared to the standard education material without them [8].

4.4 Limitations of the Experimental Study

As presented in the previous sections, the objective of the study was to explore and evaluate the effect of the robot’s embodiment in SL tutoring with interactive game scenarios. The experiments were performed with two different humanoid robots in different embodiment conditions but the experiments had some limitations summarized as below:

  • Working with hearing-impaired children requires the assistance of sign teachers and/or translators to guide or monitor them during the different phases of the study. Therefore the interaction of the child with the physically embodied robot required the presence of more than one person in the room, which might be resulted in an additional distraction for the child interacting with the robot.

  • Due to the child-like size of the R3 robot (1.08 m and 35 kg), the children had to be present at the university laboratory or their school schedule should be adjusted for our user study by the school management. Due to these restrictions, we couldn’t test our scenarios with a fixed size of participants.

  • The sign language game scenarios with the robots require generally a familiarization with emerging technologies; therefore we had held familiarization session with the robots and on how to use the presented SL application with the children

  • The scenarios also require at least the beginner level knowledge of TSL and a minimum level of literacy to be able to answer the question on the paper-based test.

  • The subset of TSL signs were selected with respect to the kinematic and physical limitations of both robot platforms and considering the difference of the degrees of freedom between 2 robots. We had also taken into consideration the fact that the children’s performance and motivation decrease due to fatigue, and their concentration decreases when the experiment time gets longer, or involves too many words/signs. From our previous experiments, we experienced that for a fruitful and enjoyable experiment with children, therefore we had to use a set composed of 10 TSL signs at maximum, per experiment.

5 Summary and Outlook

In this article, we introduced a framework utilizing gaming concepts to teach children sign language and test their knowledge of sign language. The game was based on an interaction game with two different robot platforms, namely a modified Robovie R3 and a Nao H25 robot. The motivation of this study was to implement social robot peers, which could serve as assistants in sign language tutoring for children with communication impairments.

There were two proposed scenarios with respect to the embodiment of the assistant robot in the game. The first scenario involved an interaction game with the physically embodied robots. In the game with the physical R3 robot, the children used colored flashcards of the signs generated by the robot in order to interact with the robot. In the second scenario, a virtual version of this game was implemented based on videos.

In the first version of the virtual game designed for the children of small ages and adults with no sign language acquaintance, a training session including the videos of a human tutor and both robot platforms generating selected signs from TSL were employed. Then, the recognition performance of the participants using randomly selected robot platforms were evaluated. Participants were also asked to assess, which robot signed most similar to the human signer for each word. The second game was intended for the participants with sign language acquaintance. This game did not have a training session, and comprised twice as much words. In this game, the participant was able to choose the robot platform to play with.

The virtual games were tested with 16 adults and 5 children having no sign information and no hearing problems, as well as two groups of 31 hearing-impaired children with different levels of sign knowledge. Their recognition abilities per sign, per robot platform, and subjective evaluations of the robot’s signing performances are gathered and analyzed. Although the children preferred to play with Nao robot more, their success rate using the R3 robot was higher. The results verified our hypotheses and research questions: (1) the age of the participants, who did not have sign language acquaintance, did not influence their preference about the human or robot signer (RQ1); (2) physical differences of the robot platforms did not matter when their videos appear in virtual signing game (RQ2); (3) physically embodied robots improve the recognition rate of the signs drastically compared to virtually embodied robots (H1); and (4) the level of sign language knowledge changes the fact that the recognition performance gets better with the physically embodied robots in comparison to the virtually embedded robots (RQ3).

As a future work, detailed comparison and analysis of signing performances of the virtual and physical robots will be carried out. These games will be updated according to the evaluations of the participants and will be tested with hearing impaired children from the collaborative schools.