Abstract
Recent work by researchers in China has demonstrated the potential for collaborative play as an educational tool for children on the Autism Spectrum Disorder (ASD). In this paper, researchers in the United States and China investigate the potential for accessible interface design and learning by students with ASD. Through the use of known tools and interventions, an educational protocol has been designed to evaluate two applications. The pilot studies, including experimental design and outcomes, are presented here, and provide a solid foundation for comparative assessment of mid-air finger-gesture interaction as well as hand-gesture interaction. Early results in China are promising, based on experiences in the United States.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Our understanding of the effectiveness of using games enabled by a portable marker less finger and hand position capture device (e.g. the Leap Motion Controller) to train children with Autism Spectrum Disorder (ASD) fine motor and motor gesture skills is very limited due to few empirical studies. The study described in this paper is believed to be one of the first to shed light on the suitability and degree of effectiveness of such a training approach to improve children’s motor impairments.
ASD is a neurodevelopmental disorder affecting 1 in 68 children in the US [1]; it is characterized by such core impairments as repetitive and constrained behaviors and lack of social communication knowledge and skills [1]. Of note, many children with autism demonstrate severe impairment with reciprocal interaction significantly impacting their joint attention [2] and collaborative interaction and play [3] which in turn decreases their ability to participate in meaningful learning, social and play interactions [4].
In this paper, we report on two pilot studies conducted in the classroom of a large Children’s Educational Development Center in a southern Chinese city for children with ASD. To the best of our knowledge, it is one of the first attempts to study the acceptability, usability and possible benefits of finger- and hand-gesture enabled interactions for special education.
2 Related Work
Few previous empirical studies have provided us with sufficient understanding of both the acceptability and usability of the finger- and hand-gesture based motionless games/applications for individuals with ASD. However, two indirect lines of prior research are relevant to our current study.
Recent studies revealed that the under-explored motor abnormalities are also prevalent in ASD [5] and are believed to interfere with the development of adaptive skills [6, 7]. The motor abnormalities, known as associated symptoms [8] may include problems with gross and fine motor skills [9, 34], skilled motor gestures [6, 10] and motor learning patterns [11]. However, children with autism often demonstrate strong visual perceptual abilities for objects as they have been found to be drawn to consistency of visual stimuli (both static and dynamic), matching and discriminating salient features [12–15, 33], while conversely there have been impairments noted in the visual perception abilities for facial features [14–16].
Although a wide variety of games and applications have been developed to improve individuals’ social communication, eye-hand coordination and cognitive skills etc., fewer games and applications target the individual’s motor impairments.
2.1 Benefits of Motion-Based Games and Applications
Several recent studies on the possible advantages motion-based touchless games provide for children with ASD [17–20] have been found. In particular, these studies revealed that the whole-body interaction enabled by Kinect games can enhance children’s motor skills and social behaviors [19, 20, 35], reduce their distress and increase emotion-releasing abilities [17]. Custom-made Kinect games can further be used for attention and cognitive skills training [18]. The field studies conducted in [17] indicate that motion-based gaming can promote cognitive learning and attentional skills, supporting both selective and sustained attention.
2.2 Use of Gesture Games and Applications for Fine Motor Skills Training
In addition to Kinect, Leap Motion, an increasingly popular touchless hand and finger motion tracking device, has been examined to demonstrate its accuracy in measuring hand movement in pointing task among healthy adults [21]. Due to its extreme portability, Leap Motion offers a greater advantage than Kinect for training on fine motor skills, such as that needed by individuals with ASD. A full understanding of the benefits of motion-based games (including both whole-body and hand-finger gesture-based interactions) requires examining that of the latter. However, little previous empirical evidence directly addresses the issue, especially in China where technology-based application use is rarely used in both the school and at home [22].
Following this path of research, we have seen so far one attempt which includes a series of Leap Motion-based games to train children’s fine motor skills empirically studied in a special education center in Beijing [23]. Results showed that the ‘innovative’ interaction poses bigger challenges—children have difficulty adjusting their hand gestures in a 3D space.
3 Experimental Design and Assessment
3.1 Study One: Drawing Game
The Game.
A simple drawing game was implemented using C#. Players can use both hand or finger gestures to draw on the canvas (Fig. 1). In order to assess children’s attention and engagement level and their emotional state, we made the game rules very simple by supporting only two simple actions—drawing through hand-finger air gesture (depicted by the boy in Fig. 2) and cleaning via a circle gesture.
Participant.
Five pairs of boys and their family members were recruited by the Center to participate in this game. (In the Center, close to 90 % of the children are boys). The boys participating in this study have comparable (similar) intelligence and abilities (M = 4.8, SD = 1.8), referred to as C1 (the oldest, 8 years old), C2, C3, C4 and C5 respectively.
Experimental Conditions.
One purpose of this experiment was to examine whether finger-gesture based manipulation would bring more fun and usability (in terms of ease of use) to the children. Therefore, we tested the game in the school with which children feel familiar and comfortable. The game was projected onto a wall via a projector; and to avoid children’s curiosity towards the tiny Leap Motion controller, we ‘hid’ the controller into a box leaving the interaction area open (as depicted in Fig. 2). Each family member/child pair was given a maximum 15 min of play in order not to fatigue the children [17, 18] due to their young age.
Measures.
Qualitative data, such as the level of enjoyment and engagement, and the degree of attention on screen objects were collected through observations, commonly seen in the literature [17, 24, 25]. These variables were among the behavioral and emotional signals examined in previous research [17].
Experiment Procedures.
Before the experiment, the head teacher was given the Chinese translation of the game including a short description of how to play it. Parents were informed of the purpose of the experiment, experimental procedures and their rights. Each child entered the testing room with their parents (except for one with his grandmother, C3). The developer first introduced the Leap Motion device, and demonstrated how to draw with two finger-gestures. Then, the parent/grandmother-child pair interact with the game together. After the testing was over, structured interviews based on the one developed in [26] for a Chinese family were conducted at the end of the play session. The aim was to obtain information on the child’s behaviors and preferences, and phone- and computer-game uses at home.
Results and Discussions.
In contrast to previous findings that children were mostly reluctant to touch a singing plant [22], both children and their family members showed high engagement in the drawing game; their attention level was very high during the entire game-playing session. All children showed high levels of sustained attention, and most of the time when they were playing the game with the help of the parents, their eyes remained focused towards the visual stimulus in the game. Both children and parent/grandma’s emotional states were enthusiastic. Structured interview with all family members indicate the high satisfaction of the family with the game, especially regarding the Leap Motion device since it was their initial encounter with such a ‘magic’ controller. Overall, for all of the parents, it was the very first time they were aware of the existence of such a device and they all expressed amazement at such an innovative way of user interaction. While all children had been given opportunities to play with mobile phones at home, their family members claimed that none of the games and applications are specifically designed for their children; in fact, according to the teachers at the Center, this was the first-ever such educational application. An earlier game, the Yuudee (Little Rain Drop), launched in May 2015 [27], lacks the personalization mechanism where teachers can adaptively change the contents based on children’s responses and conditions, therefore, it has not gained popularity among special education schools in China. With this said, the teachers at this Center were very open to our drawing game. However, it was observed that the children had difficulty manipulating and using their fingers; we designed a second more portable web application which utilizes hand-gesture.
3.2 Study Two: Zoo and House Game Study
The Application.
The improved version was implemented using JavaScript and can be accessed at: http://www.tmywk.com/ as a web-enabled application (a Leap Motion gesture controller is required). Currently, there are two available scenes: a zoo and a house. When the child interacts with one screen object (an image), its Chinese name (a word) will be shown accompanied by a sound file to spell out its name (see Fig. 3). This learning process is commonly used in the famous Lovass model of Applied Behavior Analysis (ABA) in which individuals with ASD are required to pair a word with a corresponding image (through the use of familiar materials and intermittent rewards with an aim to enhance children’s vocal language understandings). Since it is well known that children with ASD frequently experience sensory overload and may be averse to excessively loud and high pitched sound [28], the auditory sound was presented at a moderate volume, and the tones fall into the frequency range of human speech. None of the children in our experiment demonstrated any observable startle response; instead, the children’s attention to the screen objects was continuously dedicated. Teachers also voiced no complaints about the sound and the tones.
Participants.
Five boys with the same intelligence and abilities participated in this study (M = 6.3, SD = 2.4), referred to as C1, C2, C3, C4 and C5 respectively; four boys were reported to have interest in playing computer games (C1, C2, C3, and C5). Their head teacher recommended the children because they were informed of the nature of the game beforehand (as required by the Autism Center).
Experimental Conditions.
One of the purposes of this second experiment was to examine whether hand-gesture based manipulation would bring more fun and usability (in terms of ease of use) to the child, therefore, we tested the game in the school again with which children feel familiar and comfortable.
Measures.
Qualitative data included the level of enjoyment and engagement, the degree of attention on screen objects, and additionally, the degree of independent manipulation by children were collected through observations (commonly seen in the literature [12, 17, 24, 25]). These variables were among the behavioral and emotional signals examined in previous research [17]. Although the conclusions derived from these data cannot provide a convincing picture of the design patterns for such a game, they are sufficient to inform the designers of whether the new design managed to serve its intended goals.
Experiment Procedures.
Each child entered the testing room with either the home-room teacher or the head-teacher. The teachers had been informed earlier about the nature of the experiments (with a Chinese translation of our game). The game designer provided brief instructions to the children on how to play the game through a short demonstration (lasting less than one minute). After the instruction, the children were given the time to play the game. Prior to this experiment, and unlike some other testing in which a given time is allocated, Chinese children had seldom been exposed to such gesture-based game and technology-enabled games before [22]. This is only the second time such a gesture-based application has ever tested in the ten-year period that the Center has been educating children. Therefore, free-play was administered in order to assess the usability of the application. After each play round, teachers were interviewed to provide their perspectives on the children’s noticeable behaviors and the overall benefits of the game.
Results and Discussions.
All the children showed high enthusiasm while playing the game, and with very minimum training needed (see Fig. 4 on one testing moment). Among the five boys participating in the game, three older boys C3, C4 and C5, were extremely attentive to the game, played it with ease, and showed high sustained attention [17]. Among the qualitative data collected through the analysis of the videos and observations during the testing, the level of enjoyment and engagement, the degree of attention on screen objects and the degree of independent manipulation were all very high. Compared to the similar observations in Study One, the children needed less assistance to activate the device and play the game. Since the game was also projected into the wall, two younger boys C1 and C2, demonstrated high interest during game-play as they were observed to stare at the wall display instead of the computer screen. Of note, child C4 stopped working in the application when it was time for him to attend a class; however he later went back on his own to the game during the class break. According to the homeroom teacher this was the first time he had independently initiated participation in an activity.
In summary, all the children were able to show sustained attention to the on-screen stimuli. They demonstrated high skills in mimicking the developer’s demonstration. The results lead us to believe that the usability of such a hand-motion based touchless application is very satisfactory. To our surprise, the teachers’ feedback and reactions in Study Two were much more positive when compared to those from Study One: they revealed that although it is the first time they had ever seen such a type of game tested in their center, it was beyond their expectations to observe the children’s reactions to the application as well as their quick and deep immersion in it.
Unlike previous studies [17, 18] examining the learning benefits of such motion-based touchless games relative to children’s attentional skills, our work attempts to understand the acceptability of such games (Study One and Two) and the learnability and usability of it (Study Two). It is unclear what design rules would best be formulated in both gesture-based and multi-modal interactive applications since some cultural issues unique to this population have contributed to the overall acceptance of these applications. In addition, children seldom receive strengthened in-home intensive training in parallel to their school-based ABA training programs. Chinese parents largely rely on the Autism Center to undertake all the interventions. However, our two experiments have drawn attention from both the teachers and parents who have encouraged us on in further development of such games.
4 Lessons Learned
Our current understanding of the possible benefits of motion-based application/games for individuals with ASD are extremely limited, partly due to the small number of empirical studies, particularly in China. Hence, overall, the two studies conducted and reported here offer a valuable first glimpse into the acceptability and usability of motion-based touchless games for Chinese children with ASD.
Results of our two pilot studies reveal practices and perspectives on the use of finger- and hand-gesture enabled interactions that add depth to our previous understanding of such use. In particular, while mid-air finger-gesture interaction is challenging for younger children with ASD, hand-gesture interaction offers more fun and playability in the learning process. However, we expect some resistance from parents and special education teachers who may remain suspicious of the use of such games/applications.
5 Conclusion
The research presented here provided an opportunity to learn about how cause and effect contributes to motor skill acquisition and learning. In particular, the researchers observed that when the child is engaged in a motivating task he will practice the required movements, and through this practice develop motor control accompanied by true learning to build a foundation and provide carryover into more complex movement patterns. Additionally, and perhaps more importantly, when the parent/caregiver observes the child’s ability to activate and sustain engagement in a task, the parent is more likely to carry over the practice at home further solidifying the child’s skill development. This training protocol is beneficial for both the teachers and the parent/caregiver, as they recognize the ease of use when provided with the initial instructions, reinforced with observation of the child’s focused and sustained engagement in the activating the game and attending to the visual stimulus that results from the child’s motor movement (either the full hand gesture or the more refined 1 or 2 finger gesture).
Teachers can build on the visual motor abilities the children gain through the use of the games provided here to integrate learning concepts (examples include matching colors/shapes; categorization of images and finally matching words to pictures to promote reading and language acquisition). Finally, children can develop social skills through turn taking and collaborative play with the computer games. Developmentally the child is given the opportunity to simultaneously link the kinesthetic and visual systems together. For example, the child will move his hand or fingers and watch the visual stimulus that results from the movement. This is foundational for numerous necessary eye hand coordination tasks for daily living skills (buttoning a jacket, opening containers/book bags, folding clothes) and school/learning skills (using scissors, arts and crafts, drawing, letter formation and handwriting). In the United States, occupational therapists who work with children with ASD, use play and school based activities to promote the children’s bilateral hand coordination and eye hand coordination (visual motor integration). For example, for handwriting, children must hold the paper with one hand and write with the other. Foundational eye hand coordination activities, such as the finger- and hand-gesture enabled interactions with fun and engaging computer games and programs, provides children the opportunity to develop these abilities. Additionally, multiple studies have reported the positive impact of the use of a multi-sensory approach to teach children with ASD and to promote social interactions and engagement [29–31]. Through the use of motion based visual interactive computer games and programs, the child with ASD benefits from a multisensory approach to learning (the motor act of activating the device and the viewing the visual stimulus).
For a nation where even the prevalence study has only started two years ago [32] and government-funded special education is both expensive and limited, in terms of both quality and quantity, affordable and portable technological solutions such as those outlined here could offer great helps to families living with ASD. We hope our study sheds light on the degree of benefits of such hand and figure motion control games for Chinese children with ASD, and also provides preliminary data to call for more HCI community engagement in such research in China.
References
American Psychiatric Association. DSM-V: Diagnostic and statistical manual of mental disorders, 5th Edition. American Psychiatric Publishing, Section 299.00 (2013)
Kasari, C., Freeman, S., Paparella, T.: Joint attention and symbolic play in young children with autism: A randomized controlled intervention study. J. Child Psychol. Psychiatry 47(6), 611–620 (2006)
Machalicek, W., Shogren, K., Lang, R., Rispoli, M., O’Reilly, M.F., Franco, J.H., Sigafoos, J.: Increasing play and decreasing the challenging behavior of children with autism during recess with activity schedules and task correspondence training. Res. Autism Spectrum Disord. 3(2), 547–555 (2009)
Case-Smith, J., Arbesman, M.: Evidence-based review of interventions for autism used in or of relevance to occupational therapy. Am. J. Occup. Ther. 62, 416–429 (2008)
Sacrey, L.-A.R., Germani, T., Bryson, S.E., Zwaigenbaum, L.: Reaching and grasping in autism spectrum disorder: a review of recent literature. Front. Neurol. 5(6), 1–12 (2014)
Mostofsky, S.H., Dubey, P., Jerath, V.K., Jansiewicz, E.M., Goldberg, M.C., Denckla, M.B.: Developmental dyspraxia is not limited to imitation in children with autism spectrum disorders. J. Int. Neuropsychol. Soc. 12, 314–326 (2006)
Leary, M.R., Hill, D.A.: Moving on: autism and movement disturbance. Ment. Retard. 34, 39–53 (1996)
Ming, X., Brimacombe, M., Wagner, G.C.: Prevalence of motor impairments in autism spectrum disorders. Brain Dev. 29, 565–570 (2007)
Fournier, K.A., Hass, C.J., Naik, S.K., Lodha, N., Cauraugh, J.H.: Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. J. Autism Dev. Disord. 10, 1227–1240 (2010)
Jones, V., Prior, M.: Motor imitation abilities and neurological signs in autistic children. J. Autism Dev. Disord. 15, 37–46 (1985)
Haswell, C.C., Izawa, J., Dowell, L.R., Mostofsky, S.H., Shadmehr, R.: Representations of internal models of action in the autistic brain. Nat. Neurosci. 12, 970–972 (2009)
Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., Sasson, A.B., Gal, E., Weiss, P.L.: Collaborative puzzle game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS 2009), pp. 197–204. ACM Press (2009)
Grandin, T.: My experiences with visual thinking sensory problems and communication difficulties. Center for the Study of Autism (1996)
Grandin, T.: Visual abilities and sensory differences in a person with autism. Biol. Psychiatry 65(1), 15–16 (2009)
Simmons, D.R., Robertson, A.E., McKay, L.S., Toal, E., McAleer, P., Pollick, F.E.: Vision in autism spectrum disorders. Vision. Res. 49(22), 2705–2739 (2009)
Behrmann, M., Thomas, C., Humphreys, K.: Seeing it differently: visual processing in autism. Trends Cogn. Sci. 10(6), 258–264 (2006)
Bartoli, L., Corradi, C., Garzotto, F., Valoriani, M.: Exploring motion-based touchless games for autistic children’s learning. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC 2013), pp. 102–111. ACM Press (2013)
Bartoli, L., Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M.: Designing and evaluating touchless playful interaction for ASD children. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC 2014), pp. 17–26. ACM Press (2014)
Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M: Motion-based touchless interaction for ASD children: a case study. In: Proceedings of AVI, pp. 117–120. ACM Press (2014)
Ringland, K.E., Zalapa, R. Neal, M, Escobedo, L. Tentori, M. Hayes, G.: SensoryPaint: a natural user interface supporting sensory integration in children with neurodevelopmental disorders. İn: CHI 2014 Extended Abstracts, pp. 1681–1686. ACM Press (2014)
Tung, J.Y., Lulic, T., Gonzalez, D.A., Tran, J., Dickerson, C.R., Roy, E.A.: Evaluation of a portable marker less finger position capture device: accuracy of the leap motion controller in healthy adults. Physiol. Meas. 36(5), 1025–1035 (2015)
Tang, T., Winoto, P., Wang, R.: Having fun over a distance: supporting multiplayer online ball passing using multiple sets of kinect. In: CHI 2015 Extended Abstracts, pp. 1187–1192. ACM Press, New York (2015)
Zhu, G., Cai, S., Ma, Y., Liu, E.: A series of leap motion-based matching games for enhancing the fine motor skills of children with autism. In: 15th International IEEE Conference on Advanced Learning Technologies (ICALT 2015), pp. 430–431. IEEE (2105)
Greis, F., Silva, M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. Comput. Support. Coop. Work 24(2-3), 149–175 (2015)
Goh, W., Shou, W., Tan, J., Lum, G.T.J.: Interaction design patterns for multi-touch tabletop collaborative games. In: CHI 2012 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2012), pp. 141–150, New York, NY, USA (2012)
La Valle, C.: Chinese cultural factors impacting the educational schooling of children with autism in China. DePaul Discoveries, Vol. 2, Issue 1, Article 10 (2013). http://via.library.depaul.edu/depaul-disc/vol2/iss1/10
Wang, S.: In China, the making of an app for autism. The Wall Street Journal, 19 May 2015. http://blogs.wsj.com/chinarealtime/2015/05/19/in-china-the-making-of-an-app-for-autism/
Marco, E., Barett, L., Hinkley, N., Hill, S., Nagarajan, S.S.: Sensory processing in autism: a review of neurophysiologic findings. Pediatr. Res. 69(5 Pt 2), 48R–54R (2011)
Goldsmith, T.R., LeBlanc, L.A.: Use of technology in interventions for children with autism. J. Early Intensive Behav. Interv. 1(2), 166 (2004)
Hayes, G.R., Hirano, S., Marcu, G., Monibi, M., Nguyen, D.H., Yeganyan, M.: Interactive visual supports for children with autism. Pers. Ubiquit. Comput. 14(7), 663–680 (2010)
Silver, M., Oakes, P.: Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 5(3), 299–316 (2001)
Clampton, N.: China moves to tackle autism with first study of prevalence. South China Morning Post (2013)
Grandin, T.: Thinking in Pictures, Expanded Edition: My Life with Autism. Vintage, New York (2008)
Frith, C., Law, J.: Cognitive and physiological processes underlying drawing skills. Leonardo 28(3), 203–205 (1995)
Harper, R., Mentis, H.: The mocking gaze: the social organization of kinect use. In: Proceedings of ACM CSCW 2013, pp. 167–180. ACM Press, New York. (2013)
Acknowledgements
We acknowledge Kean University’s financial support to Tiffany Tang under Wenzhou Kean University’s Student Partnering with Faculty (SPF) Research Program. The authors are grateful to children and their families at Wenzhou XingLe Children’s Educational Development Center for participating in the experiments; thanks also go to Tina Xiaoting Fu, and Esther Mingyue Tang for developing the two games and their efforts during both the experiment and interviews as well as Leila Zeqiang Huang and Relic Yongfu Wang for their helps during the experiment.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Tang, T.Y., Falzarano, M., Morreale, P.A. (2016). Engaging Chinese Children with Autism to Interact with Portable Hand- and Finger-Gesture Based Applications: Experiment and Reflections. In: Zaphiris, P., Ioannou, A. (eds) Learning and Collaboration Technologies. LCT 2016. Lecture Notes in Computer Science(), vol 9753. Springer, Cham. https://doi.org/10.1007/978-3-319-39483-1_51
Download citation
DOI: https://doi.org/10.1007/978-3-319-39483-1_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-39482-4
Online ISBN: 978-3-319-39483-1
eBook Packages: Computer ScienceComputer Science (R0)