Abstract
Eye contact is among the most substantial yet underrated non-verbal social communicative behaviors. Establishing and maintaining eye contact is quite challenging for children affected by social impairment. As a solution, we propose a virtual reality serious game to help improve eye gaze abilities for these children and facilitate the transfer of the acquired skills to real-world social interactions. The project comprised of the design, implementation and evaluation of a three levels game that emphasizes on different aspects of eye contact. According to the evaluation carried out in this research, a virtual reality gaming context is a promising strategy for gaze development of children with social deficiencies.
Access provided by CONRICYT-eBooks. Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
How many times have arguments occurred because of an unclear text message? A mistakenly perceived as aggressive message can accurately be interpreted as sarcastic in case of the speaker wearing a sarcastic smile on his/her face. That same message may also be interpreted as caring if a deep, long and loving eye gaze with the listener is maintained. People regularly misunderstand our messages if presented only verbally which highlights the significance of non-verbal communication in our daily lives. Non-verbal social communicative behaviors constitute a vital and informative part of a conversation with their different forms of facial expressions, eye contact, body language and gestures.
Among the most important non-verbal communication channels is eye contact. Lack of eye contact namely gaze aversion may robustly obstruct daily communication processes. The adopted definition of eye contact classifies it further into two categories: dyadic eye gaze and triadic eye gaze [13]. Dyadic eye gaze is a term to denote eye to eye or eye to face interactions. Whether triadic eye gaze -which is frequently referred to as joint attention- represents a person’s perception of another person gazing in a specific direction (i.e. in the direction of a certain object). Both forms of eye gaze are vital for social communicative encounters. For instance, dyadic eye gaze is important because looking directly into our interlocutor’s eyes enables us to read their emotions. Consequently, the process of talking to people wearing dark glasses may be quite challenging [1]. Eye to eye gaze is used as a mean of getting feedback during a conversation, and its lack may also be misconstrued by others as inattention. Maintaining strong and stable dyadic eye gaze is a sign of the speaker’s self confidence and increases his/her credibility to his/her listeners. On the other hand, triadic eye gaze is considered as an important factor in understanding other people’s intentions, in other words mind reading [3]. Joint attention provides people with information about their environment, enables them to comprehend references, to point to objects and consequently learn new words [12].
Difficulty in eye contact is a clear sign of social impairment, characteristic of some developmental disorders such as autism spectrum disorders (ASD) [9]. Gaze aversion may also affect neurotypical (NT)Footnote 1 people with a mild degree of social impairment which makes them overly timid or less socially active than others.
Using conventional techniques to improve eye contact for those individuals such as verbal and physical prompting has proven to be ineffective [6]. Virtual reality (VR) serious games that comprise of virtual characters similar to real ones can help individuals with problems of eye contact to enhance it. It will make it more feasible to transfer acquired skills to real-world context afterwards. Implementing these games for children rather than adults may yield to better outcomes since they are still in an elementary stage of social development.
2 Related Work
As contended in [4], video games can support the development of social aptitudes when played in moderation to avoid gaming addiction issues. Henceforth, the field of serious games for social development has been widely explored recently with examples such as “Let’s Build” [5] that aims at teaching children the notions of fair play, collaboration and leadership. Likewise in [4], the researchers modified the “The Elder Scrolls V: Skyrim” which is a previously developed action role playing game to enhance the social skills of autistic children particularly.
Several applications and games have targeted the development of one specific social skill which is eye contact. Samsung along with doctors and psychology professors from two universities in Seoul have developed the “Look at me” app. This app aims at training autistic children on a range of social skills among which is eye contact. Another popular game for helping autistic children establish appropriate eye contact (both dyadic and triadic) is “Ted’s Ice Cream Adventure”. The game constitutes of many levels that revolve around a bear trying to sell ice cream to his clients. The user has to identify when is the bear trying to talk to him/her through his eye gaze direction. In other stages, the user has to recognize where the bear’s gaze is directed, for instance in the direction of a specific object.
In recent years, VR technology has been used to teach children everyday skills by providing them with a safe environment resembling real life [7]. VR allows social interaction with a less amount of stress, anxiety and fears of mistakes as in real life situations [8]. Virtual environments that we expose a child to are more controllable, we can modify the colors, characters, situations and scenarios accordingly to perfectly meet the requirements. Moreover, researchers showed that skills learned in virtual environments can be generalized to the skills in real life. In [7], a desktop VR game was developed in order to train children on mastering the street-crossing skill. The game was then evaluated and 3 of 6 participants showed successful transfer of the virtual street-crossing to actual street-crossing.
Among the experiments where VR technology has been tackled for eye contact improvement is [9, 10]. In the former, an “Oculus Rift” as the VR headset, and a stereo camera were used. A fading prompt approach was employed to draw the attention of autistic children to the instructor’s face and eyes. A virtual image of an interesting colorful object appears on the face of the trainer and it starts fading away until it leaves focus only on his/her face and eyes. The study showed that the fading prompt is an effective mean for intriguing the child and helping him/her with eye contact. In the latter, an eye contact reminder system which reminds people with autism to focus their eyes in the direction of a human speaker was implemented. This device performs a detection of the speaker’s voice and a calculation of the sound direction. A real time message is then displayed on the head mounted display (HMD) to direct the user to look towards the speaker.
A framework implemented in [11] with the objective of developing social attention skills in autistic children uses a HMD to show images from a virtual classroom. This classroom contains a set of 3D virtual characters who are presenting educational material. Only one of these avatars is speaking at a time and will start to vanish if the child diverts their eye gaze from him/her.
“Coffee without Words” is also an example of a commercial VR game developed on “Oculus Rift” for ameliorating the eye contact skill. The virtual scene in this game is a coffee date with a female avatar who keeps moving her eyes in an attempt to encourage the user to initiate eye contact with her. No verbal communication is enclosed in the game.
To this end, existing work highlights the potential of using VR technology for migrating daily skills from a virtual environment to a real-world context. In our work, we have employed techniques similar to those explored in “Ted’s Ice Cream Adventure”. Nevertheless, we made sure to use 3D human avatars instead of 2D animals to help users identify the connection to similar real-world interactions. As discussed above, several existing applications are designed from a purely educational perspective. The absence of the amusement factor makes the application dull for children. Hence, arises the need for gamification to render the system more friendly and agreeable.
Some applications like the eye contact reminder system have illustrated their impracticality since the users need to carry the device around with them constantly. Developing an application/game to act as a training rather than an assisting tool is a substantial aspect considered in our project. Therefore, our work aims at creating a VR game that will be used as a training tool in a joyful way to help children ameliorate their eye contact capability.
Most of the above-mentioned applications that targeted eye contact [9,10,11] have focused primarily on the dyadic eye gaze capability. In our game, we will focus on both classes of eye contact.
It is also worth noting that VR technology has been rarely used in Egypt. This work is among the first formal comprehensive studies in Egypt to harness VR gaming for children afflicted with developmental impairments.
3 System Overview
Our goal in this research is to enhance eye contact for children with social impairment via a VR serious game. This can be a beginning stride to further build up their social communicative abilities. The game consists of levels that target both dyadic and triadic eye gaze independently.
3.1 Collaborative Design of “I-Interact”
Our proposed framework is directed towards children with social deficiencies. Considering the special targeted user group, it was decided to approach community experts. Contact had been established with a local early intervention center “Ebny Association for Special Needs and Autism”, and a workshop structured into two sessions had been organized with them. The first was a brainstorming session with instructors and caregivers of the intended population. Our ideas were discussed with feedback and modifications from their side. Some of the advised design aspects were to use a reward rather than a scoring mechanism to designate a game win. It is more enthusiastic for youngsters to anticipate a celebration scene as a reward on the screen than an increasing score that they seldom comprehend. Also among the decisions that were reached collaboratively was to organize the game in levels with ascending difficulty. The first level is a preliminary introduction to the concept of looking towards faces/eyes. Moreover, each level is further subdivided into three stages with the first one as a primitive trial that introduces the level flow stream. The initial target age was 3 to 9 years. However, since social deficiencies are commonly accompanied by intellectual disabilities, older children with lower intellectual abilities may benefit from the game too.
During the second session, an informal test was conducted on a first prototype that has been implemented for one stage of the first level of our proposed game (the current platform includes three levels with three stages each). The objective of this test was to foresee the viability of our system by assessing the children acceptance and delight during the VR game experience. Three children participated in this preliminary study each with low, moderate and high computer aptitudes respectively. The game fascinated the three users equally. The participant with low computer abilities was not capable of playing the game, whereas the other two children could finish the level successively without external aid.
3.2 Game Flow Description
In order to guarantee the simplicity of our game “I-interact”, prompts were limited, verbal, and in Egyptian Arabic dialect. The purpose was to dispense with the challenges that a child may confront in case he/she has other learning deficiencies (i.e. struggles to read or struggles to understand foreign languages). Virtual characters changed recurrently throughout the levels to enable the children to generalize the interactive scenarios to all human figures.
The game consists of three levels with three stages each. Both first and second levels focus on dyadic eye gaze, while the third is more concerned with triadic eye gaze. The levels were implemented in order to have a gradual impact. Therefore, the first level aims at just attracting the child’s attention towards a human face and eyes without being explicitly prompted to look towards any of them. In the second level, the child is implicitly solicited to look into the eyes of a human character in an attempt to teach him/her to initiate eye gaze and maintain it for a few seconds. The purpose of the third level is to get the child trained on triadic eye gaze by enabling him/her to identify where the character’s eye gaze is directed. The surrounding virtual environment is exceptionally primitive to avert confusion in both first and second levels. However, in the third level several distracting factors are added to render the level more challenging: two additional human characters and some furniture in the background. Scenes of some of the levels stages are shown on Fig. 1.
The levels are played in a sequential manner. Nevertheless, a player can exit a level in the middle and choose to start another one.
Below is a detailed description of each level:
Level 1. Consists of a female character who articulates a verbal request to the child. She asks him/her to remove the butterflies scattered around her face to prevent her distraction. The child performs the task by clicking on the butterflies.
The first level consists of three stages:
-
Stage 1: six butterflies are organized in a roundabout shape around the young lady’s face. The user has to get rid of them all in any order to enable the transfer to the next stage. This scene is presented on Fig. 1a.
-
Stage 2: six butterflies appear consecutively on the girl’s face. A new butterfly does not show up until the user disposes of the past one. In this phase, butterflies are nearer to the face than in the first stage.
-
Stage 3: similar to stage 2, yet the butterflies are smaller and closer to the eyes as depicted on Fig. 1b.
Level 2. Consists of a male character who asks the child about the color of his eyes which keep changing after each correct response. Eye colors alternate randomly between brown, green and blue. The user is stuck at the same scene until a correct answer is provided. The child performs the task by clicking on the correct color choice.
The second level consists of three stages with six trials each:
-
Stage 1: only one color option is provided which is eventually the right answer as shown on Fig. 1c.
-
Stage 2: the user has to select the correct option from two available color choices.
-
Stage 3: the user has to select the correct option from the three available color choices.
Level 3. Consists of a female character who asks the child to identify and catch the butterfly that she is looking towards. Similarly to level 2, the user is stuck at the same scene until a right answer is given. Similarly to level 1, the child performs the task by clicking on the designated butterfly.
The third level consists of three stages with six trials each:
-
Stage 1: As depicted on Fig. 1d, in this stage only one butterfly is present in the scene which is eventually the right answer.
-
Stage 2: the user has to catch one of two butterflies by distinguishing which one the character is looking at.
-
Stage 3: like stage 2, however, the two butterflies are closer to each other as displayed on Fig. 1e.
Every level stage ends with the celebration scene represented on Fig. 2. The scene is unified through the game and is accompanied by cheering and clapping sounds.
4 System Evaluation
With a specific end goal to get an accurate impression of our game, we evaluated it through two user studies. The first aimed to measure the effectiveness of the game in enhancing the eye contact skill. The second was performed to measure engagement levels in immersive VR gaming using HMD for people with disabilities. The motivation for this was that immersive VR technology using HMD has rarely been used in Egypt for people with disabilities and developmental impairments. Therefore, it would be beneficial to gauge their acceptance of it which will help predicting the effectiveness of the technology in the future. For consistency and center-related recruitment reasons, all selected users for this second study were autistic.
Our samples for both studies consisted of four users each. They were all recruited through the Egyptian association “Ebny Association for Special Needs and Autism”. The average age for the first study was approximately 8 years, as opposed to an average age of 14.5 years for the second study. The reason why the average age for the second study is higher is that it is not encouraged to use HMD for children under 13 years old. The majority of our selected users were autistic, thus, most of them were males. This is because more males are diagnosed with autism than females with a ratio of 4:1 [14]. A detailed presentation of our recruited samples is shown in Tables 1 and 2.
4.1 Experiments
Eye Contact Improvement Test was organized in six sessions over a period of 3 weeks. Each session was of a duration 15–20 min. In order to quantify eye contact improvement, pre- and post tests were performed to assess eye contact levels. The pre- test was performed in the first session before they started playing the game, whereas the post test was performed after they played the game for the last time in the last session. The pre- and post tests were identical and were inherited from standard tests implemented to gauge eye contact levels such as the one in [2]. The procedures for both tests started by the child’s teacher calling his/her name and identifying if the child exhibited the convenient eye contact by looking directly into the instructor’s eyes. This trial is repeated for ten consecutive times, one trial for every ten seconds. The instructor recorded the results by writing a positive or a negative sign in front of each trial on the corresponding test paper.
The flow of the six sessions went similarly afterwards. Each child played the levels of the game which were loaded successively. Initially, the children were intended to play only the first two levels, since our eye contact assessment is used for testing dyadic eye gaze only. However, we allowed the children to play the third level or even sometimes play the three levels twice in the same session. This was only in the event that they were exceptionally intrigued and did not have any desire to leave the game.
Their teachers were present by their sides amid all sessions to provide additional assistance if necessary. The game was played on a touch screen notebook powered by an Intel i7 core processor and with embedded speakers for the sound effects.
Immersive VR Engagement Test was a one session test for each participant. The duration of each session was 20–30 min length. In order to assess participants engagement and delight in the immersive VR technology, each session was partitioned into two sections. During each section, the user played one of the two versions of the game. The first one was played on a touch screen notebook powered by Intel i7 core processor with embedded speakers. The second was played on a HTC Vive HMD, its controllers and a pair of stereo speakers that were used for sound generation. All users were accompanied by their instructors throughout the session to provide additional assistance if necessary. After each part of the session, a hard copy of a standardized survey derived from [15] was filled by the teacher. The test questionnaire evaluated the overall engagement of the user during the corresponding game version through nine points with five available Likert scale responses.
4.2 Results
Results of the eye contact improvement test are depicted on Fig. 3 which presents the number of positive eye contact trials before and after the game intervention.
The results show that all participants have showed an improvement in eye contact with the exception of participant P14 whose eye contact behavior did not change. Furthermore, with respect to other participants, P13 specifically has shown a clearly significant improvement (+4 in comparison to +2). This can be attributed to the fact that P13 is the only participant with no disabilities, rather only social behavior problems. This might make it easier to alter her social conduct in case she was sufficiently intrigued which she clearly was.
Immersive VR technology using HMD has yielded to a slight positive effect only in a single case of the four participants in the immersive VR engagement study. It was clearly noticed that the reason behind that is that they did not feel in full control while using the HMD controllers. Results for this study are shown in details on Fig. 4.
4.3 Discussion
As previously discussed in Sect. 4.2, the eye contact improvement study yielded positive results. Due to our target group being extremely special, recruitment of the sample was quite challenging. It was noticed that in order for a participant to play the game, it is preferable for him/her to have at least moderate computer skills. Children with social impairment that have moderate/high computer abilities rarely exhibit problems in the eye contact skill. Therefore, our sample was exceptionally small. Two participants started the study, nonetheless, they dropped out early. One of them had ADHDFootnote 2 and thus was never able to fully concentrate on the game. The second had very low computer aptitudes and was not under any condition pulled in by the game. It is important to note that only one user (P11) had basic computer skills and yet has completed the study with positive outcomes. For this particular user, progress in his engagement level was noticed in each session.
Although no study was carried out to test the transfer of the eye contact behavior to real-world context, it is anticipated that our framework can likewise perform satisfactorily in this regard. P11 kept kissing the virtual characters each time they show up on screen. When asked, his teacher affirmed that this is his behavior in real life towards individuals he likes. The same participant was constantly waiting for the celebration scene and used to get energized, cheer and applaud along when it begins on the screen. This also confirms the convenience of the proposed design aspect to utilize a reward rather than a scoring system to denote a game win.
The results for the immersive VR engagement study were not quite positive though. P23 expressed unequivocally that he truly enjoyed the Dektop VR version versus the immersive VR adaptation. All users in this study were chosen to have an explicit interest in technology and a range of moderate to high computer skills to be capable to manage the HMD and the controllers. Nevertheless, three out of the four users did not feel in full control while playing the game. They additionally often got bored when playing the immersive VR version because it took them so long to perform the required tasks. This suggests that immersive VR gaming for people with disabilities may still need a lot of training with them before we can see positive outcomes out of it.
5 Conclusion and Future Work
The motivation behind this project was to develop a VR serious game to enhance eye contact capacities for children with social deficiencies. To achieve our objective, we conducted a workshop with specialists in the field and then developed a three levels game that handles different aspects of eye contact. In order to evaluate the adequacy of our platform, we directed a user study for the targeted children. Eye contact levels were assessed before and after six sessions of playing the game. The results of this test guarantee the viability of our framework for the purpose it was created for. In the future, longitudinal studies that test the migration of the acquired eye contact capabilities to real life communication encounters can be conducted. It is important to note that our user study focused on testing the improvement of dyadic eye gaze only, future user studies can additionally assess the progress in triadic eye gaze.
A second user study was performed to investigate the potential of immersive VR technology in the participants overall engagement in the game. This study did not render positive outcomes which is ascribed to the lack of control that the users exhibited amid the immersive VR experience. Better results can be achieved in the future by implementing different adaptations of the game that demand less control driven input. Providing more training for people with disabilities on the VR headset and it controllers will also prove fruitful on the long run.
Notes
- 1.
A term used to denote people who are not on the autism spectrum.
- 2.
Attention Deficit Hyperactivity Disorder which is a disorder characterized by symptoms such as inattentiveness, hyperactivity and impulsiveness.
References
Swami, V., Furnham, A.: The Psychology of Physical Attraction. Routledge/Taylor & Francis Group, Abingdon (2008)
Thompson, N., Campling, J.: Non-verbal communication. In: Campling, J. (ed.) People Skills, pp. 93–103. Palgrave, London (1996). https://doi.org/10.1007/978-1-349-13737-4_10
Baron-Cohen, S.: Mindblindness: An Essay on Autism and Theory of Mind. MIT Press, Cambrige (1995)
Gay, V., Leijdekkers, P., Pooley, A.: Building social awareness for teens and young adults with autism via gamification. In: Marsh, T., Ma, M., Oliveira, M.F., Baalsrud Hauge, J., Göbel, S. (eds.) JCSG 2016. LNCS, vol. 9894, pp. 116–127. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45841-0_10
Lim-Fei, V., Woo, H.M., Lee, M.Y.: Serious games to develop social and emotional learning in students. In: Marsh, T., Ma, M., Oliveira, M.F., Baalsrud Hauge, J., Göbel, S. (eds.) JCSG 2016. LNCS, vol. 9894, pp. 3–12. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45841-0_1
Carbone, V.J., O’Brien, L., Sweeney-Kerwin, E.J., Albert, K.M.: Teaching eye contact to children with autism: a conceptual analysis and single case study. Educ. Treat. Child. 36(2), 139–159 (2013)
Josman, N., Ben-Chaim, H.M., Friedrich, S., Weiss, P.L.: Effectiveness of virtual reality for teaching street-crossing skills to children and adolescents with autism. Int. J. Disabil. Hum. Dev. 7(1), 49–56 (2008)
Kandalaft, M.R., Didehbani, N., Krawczyk, D.C., Allen, T.T., Chapman, S.B.: Virtual reality social cognition training for young adults with high-functioning autism. J. Autism Dev. Disord. 43(1), 34–44 (2013)
Wang, X., Desalvo, N., Gao, Z., Zhao, X., Lerman, D.C., Gnawali, O., Shi, W.: Eye contact conditioning in autistic children using virtual reality technology. In: Cipresso, P., Matic, A., Lopez, G. (eds.) MindCare 2014. LNICSSITE, vol. 100, pp. 79–89. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11564-1_9
Wang, X., Desalvo, N., Zhao, X., Feng, T., Loveland, K.A., Shi, W., Gnawali, O.: Eye contact reminder system for people with autism. In: 2014 6th International Conference on Mobile Computing, Applications and Services (MobiCASE), pp. 160–163. IEEE (2014)
Jarrold, W., Mundy, P., Gwaltney, M., Bailenson, J., Hatt, N., McIntyre, N., Kim, K., Solomon, M., Novotny, S., Swain, L.: Social attention in a virtual public speaking task in higher functioning children with autism. Autism Res. 6(5), 393–410 (2013)
Podrouzek, W., Furrow, D.: Preschoolers’ use of eye contact while speaking: the influence of sex, age, and conversational partner. J. Psycholinguist. Res. 17(2), 89–98 (1988)
Leekam, S.R., Ramsden, C.A.: Dyadic orienting and joint attention in preschool children with autism. J. Autism Dev. Disord. 36(2), 185–197 (2016)
Bellani, M., Fornasari, L., Chittaro, L., Brambilla, P.: Virtual reality in autism: state of the art. Epidemiol. Psychiatr. Sci. 20(03), 235–238 (2011)
Pearce, J.M., Ainley, M., Howard, S.: The ebb and flow of online learning. Comput. Hum. Behav. 21(5), 745–771 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Elgarf, M., Abdennadher, S., Elshahawy, M. (2017). I-Interact: A Virtual Reality Serious Game for Eye Contact Improvement for Children with Social Impairment. In: Alcañiz, M., Göbel, S., Ma, M., Fradinho Oliveira, M., Baalsrud Hauge, J., Marsh, T. (eds) Serious Games. JCSG 2017. Lecture Notes in Computer Science(), vol 10622. Springer, Cham. https://doi.org/10.1007/978-3-319-70111-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-70111-0_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70110-3
Online ISBN: 978-3-319-70111-0
eBook Packages: Computer ScienceComputer Science (R0)