Keywords

1 Technology-Based Intervention in Autism Spectrum Disorder

1.1 Learners with Minimally-Verbal Autism and Their Needs

Individuals with autism spectrum disorder (ASD) experience a severe delay or atypical development in the areas of language and communication (American Psychiatric Association 2013). Worldwide, 1 in 160 children is estimated to have ASD (Elsabbagh et al. 2012), and it has been estimated that between 30% and 50% of children with ASD are non-verbal or minimally verbal (Tager-Flusberg and Kasari 2013). Students with ASD who are nonverbal, or minimally verbal can face significant challenges to academic growth because speaking and listening are key components of literacy measures in K-12 education that require developmentally appropriate and socially expected levels of proficiency in all four literacy modalities across several content areas. Limited communication skills also pose repercussions beyond the academic setting by predisposing these students to increased risks of depression, lowered sense of well-being, lack of autonomy and self-determination in the workplace, and in society at large (Taylor et al. 2015; Scott et al. 2018; Kirby et al. 2016; Henninger and Taylor 2012; Chiang et al. 2011; Hotulainen et al. 2010; and Friedman et al. 2019).

The legal and social impetus for providing evidence-based practices to help this student population succeed is clear. However, very few studies have examined “the progress or lack thereof for minimally verbal children with ASD” in response to interventions (Bak et al. 2017). This area of research has been intriguing legislators, educators, clinicians, and researchers since Kanner’s identification of autism in 1943. Improving educational results for children with disabilities, including minimally verbal children with ASD, is an essential component of policies across national boundaries and jurisdictions since an ever growing corpus of research shows that educational attainment is linked to improved chances for equity of opportunity, full participation in society, independent living, and economic self-sufficiency for individuals with disabilities (World Health Organization 2014a, 2014b; American Speech Language Hearing Association 2016; U.S. Department of Education 2004). Clinical interventions to improve educational outcomes for minimally verbal children with ASD are critical for meeting these goals.

1.2 Intervention Through Augmentative and Alternative Communication

In clinical and educational practice, the most immediate need for learners with severe, minimally verbal autism is to establish functional communication to meet basic wants and needs and to participate in daily life. Augmentative and Alternative Communication (AAC) interventions have been developed and implemented to address this most pressing need. AAC approaches range from low-technology paper-based communication boards (Ratusnik and Ratusnik 1974) to the more current and widely used high-technology strategies including speech generating devices (SGDs) and tablets such as the iPad with corresponding tablet applications (Wendt et al. 2019a, 2019b; Waddington et al. 2014).

The growing ubiquity of smartphone and tablet usage in society at large has prompted much interest in the deployment of this convenient and affordable technology due to its social validity. This normalization of usage fosters potential for increased therapeutic applications to enhance communication and to also ideally increase the natural speech production of minimally verbal children (McNaughton and Light 2013; Schäfer et al. 2016).

1.3 New Opportunities and Challenges Through Mobile Technologies

The introduction of the Apple iPad in 2010 had a significant impact on the range of tools available for AAC intervention. Established dedicated devices, that had a rather high price point and were limited to communication purposes, lost their appeal as the go-to solution for minimally verbal children with autism. The iPad provided a more flexible platform to run a variety of AAC apps to go beyond the functions that most SGDs were able to supply. iPad-based AAC interventions helped to save costs while increasing motivation and social appeal (Wendt 2014).

This affinity for technology has been further explored in the systematic literature review conducted by Valencia et al. (2019). The authors point to a growing body of research documenting benefits of technology-based interventions in educational settings, with about 25% of included studies targeting communication and language skills as primary learning outcomes.

The literature on technology usage to teach students with ASD suggests the need for future research into user experience and effects of gamification on learning to validate this mode of intervention (Valencia et al. 2019). The need for further validation of, and research in, technology-based interventions is also highlighted in the meta-analysis by Watkins and colleagues (2019), examining best practices for students with ASD in inclusive educational settings. Nonverbal, or minimally verbal students with ASD in particular are an underrepresented population in the intervention literature, for example composing only 4% of included participants in the meta-analysis by Watkins et al. (2019).

1.4 SPEAKall! for AAC Intervention in Autism Spectrum Disorder

To create an autism-specific AAC solution that directly targets critical speech and language milestones while being sensitive to the cognitive and sensory processing characteristics of learners with autism, the UCF Research Lab on AAC in Autism developed a suite of mobile AAC applications including SPEAKall!® and SPEAKall! Lite®.

The original SPEAKall! application was designed for the iPad® platform to target AAC intervention outcomes such as functional communication skills, natural speech production, and social-pragmatic behaviors. SPEAKall! creates a very intuitive user experience for the non-verbal learner (see Fig. 1). A selection portion at the top of the screen provides graphic symbols and photographs which can be moved via drag and drop to a storyboarding strip at the bottom to construct sentences. The drag and drop gesture can be substituted by a simple touch gesture for learners with motor control difficulties. Caretakers can take their own photos to extend the existing bank of symbols and to create relevant and recognizable content that reflects the user’s everyday life. Once a user combines the symbols into a sentence, the activation of the “Speak All” icon will speak the sentence out loud. Voice-output is provided via pre-recorded speech from caretakers or by using Apple’s synthetic Siri voice.

Fig. 1.
figure 1

(Courtesy of the UCF Research Lab on AAC in Autism).

An example for a communication display on the SPEAKall! application to teach functional communication skills.

Full size image

To explore the full functionality of iOS-based platforms and to fill the gap on feasible AAC apps for smaller screen devices, the research lab developed an iPhone/iPod-based version, SPEAKall! Lite (see Fig. 2). The purpose of SPEAKall! Lite is to provide a readily available communication tool for individuals and their caretakers. Because many families affected by autism or developmental disability may not have immediate access to an iPad, but may already possess an iPhone, SPEAKall! Lite has the potential to be an instantly available solution that practitioners and caretakers can use to establish basic but crucial communication skills and assess whether or not the learner may benefit from a mobile technology solution.

Fig. 2.
figure 2

(Courtesy of the UCF Research Lab on AAC in Autism).

An example for a communication display to teach basic wants and needs with the iPhone application “SPEAKall! Lite©”.

Full size image

From an early intervention perspective, AAC experiences should be provided to learners with severe speech and language delay as early as 2 years of age (Davidoff 2017). At such a young age, however, many young infants, especially when affected by developmental delay or disorders, do not possess fully developed fine motor skills to operate mobile devices with strong proficiency. Particularly for learners with autism, it has been well documented that about 80% are affected by some difficulty in fine or gross motor function (Hilton et al. 2012; Isenhower et al. 2012). Often times, additional training becomes necessary to teach the proper motor access strategies for the user to properly operate applications on mobile devices.

1.5 Gamification to Enhance Learner Experience

Gamification is a design approach for optimized human computer interaction that holds great promise to facilitate skill acquisition on mobile devices used in ASD.

Originally, gamification was defined as the “application of game-like accelerated user interface design to make electronic transactions both fast and enjoyable” (Marczewski 2015). The approach has grown in popularity because it enables skill development and retention. By providing incentives to users and motivating them to strive towards rewards throughout their user experience, users tend to spend more time interacting with the technology. Essentially, gamification uses the most prominent elements of games and combines these elements with nongame components and contexts to motivate users to increase target behaviors (Goethe 2019). It enhances the level of user engagement and leads users to make the most out of their natural desires including learning, socializing, achievement, mastery, and status. Training skills and behaviors that originally appeared hard, tedious, and boring can become a fun and motivating exercise (Hamari 2017).

Of particular interest for computer based ASD intervention is the idea of ‘serious game design’, an approach to boost the effect of these interventions, facilitate acquisition and generalization of targeted skills to everyday environments (Whyte et al. 2015). Serious games incorporate pedagogical principles and motivational aspects of game design to optimize the process of skill acquisition while also exploring the benefits of motivational design components to create the best possible learning experiences for the user. Serious games have shown promise in enhancing the skill repertoire related to emotion recognition in youth with ASD (Tang et al. 2018) and the potential to improve a variety of ASD interventions (Boucenna et al. 2014; Grossard et al. 2017).

1.6 Project Goals

Our project goals were to use the approach of serious game design to develop a series of gesture-based games that directly target the difficulties in learning complex and multi-step motor skills to access content on mobile AAC devices. The envisioned outcome of engaging with these games was to enhance motor functioning of the young child or first time AAC user to maximize operational competence in navigating touchscreen-based content. With better gesture and operating skills, children can get started earlier on mobile devices and benefit from SPEAKall! and similar AAC applications to the full extent. The purpose for the resulting application was to train users in major iOS gestures through a gaming experience that is engaging, rewarding, and motivating while targeting fine motor control skills.

2 Design and Development Process

2.1 General Requirements for SPEAKplay!

The design and development process was a collaborative effort between a senior design team in software engineering at Rose-Hulman Institute of Technology and the AAC and Autism Research Lab at the University of Central Florida. The larger project team decided to label the new application “SPEAKplay!”. An initial needs assessment revealed the following critical user aspects to address:

  1. 1.

    Children with special needs, in this case with severe communication disorders, have difficulty performing the motor tasks necessary to use touch screen devices. This prevents them from accessing and using learning and therapeutic applications to their full extent.

  2. 2.

    Games should be intuitive.

  3. 3.

    Games should be engaging and motivating.

  4. 4.

    Games should be challenging without causing frustration.

  5. 5.

    Current games are not child friendly.

  6. 6.

    Current games take too long to play.

  7. 7.

    Current games do not offer an option to pause.

  8. 8.

    Existing applications do not take into account that children do not have the ability to configure the application to fit their needs in terms of cognitive, motor, and sensory abilities and characteristics.

Subsequently, the larger project team identified the general design requirements for the SPEAKplay! application:

  1. A.

    SPEAKplay! will train motor capabilities for operating AAC apps on mobile devices; the training will focus on the critical gestures of (i) Directed and Sustained Touch, (ii) Drag and Drop, and (iii) Vertical and Horizontal Swipe.

  2. B.

    Different levels of difficulty are desirable to gradually enhance learner performance.

  3. C.

    The skills should be taught through gamification elements that keep the learner engaged and motivated. This can be accomplished through a series of brief “arcade-style” games.

  4. D.

    The games include “Pop the Bubbles” (for targeted touch), “Fly the Rocket” (for targeted and sustained touch), and “Feed the Dino” (for horizontal and vertical swipes as well as drag and drop).

2.2 General Design and Visualization Process

The general design and visualization process focused on the goal of helping children learn and practice the motor skills necessary to use other applications in the SPEAK*** suite. Game design was the priority, with an admin mode serving as a lightweight method of enhancing basic games on a per user basis. To get started, the UCF research team created some initial game idea sketches. These pen and paper sketches focused on capturing a cohesive theme for each game, defining the gesture skill to practice, and creating a simple, yet engaging, layout and play style to teach the target skill (see Fig. 3).

Fig. 3.
figure 3

(Courtesy of the UCF Research Lab on AAC in Autism).

An initial sketch of the Rocket Game.

Full size image

Afterwards, the Rose-Hulman team created prototypes for the entire application using Figma ©, v 76.4.0 (Field 2019), a collaborative interface design tool (see Fig. 4). Over several weeks of discussion, the team utilized Figma to quickly and easily make app design changes, from overall screen flow to color, sizing, and animation choices.

Fig. 4.
figure 4

(Courtesy of Rose-Hulman Institute of Technology).

Reusable Figma components.

Full size image

Figma’s demo features allow the prototype to be downloaded to a mobile phone and capture the “feel” of clicking through the app (see Fig. 5).

Fig. 5.
figure 5

(Courtesy of the UCF Research Lab on AAC in Autism).

Various screens of the Rocket Game linked to allow application playthrough.

Full size image

A variety of graphic images were incorporated to enhance visual appeal. Most icons came from Flaticon (Freepik Company 2019), a source for thousands of free to use images. The prototype design process resulted in the decision to create a home page with a list of games and a button to access the administrative mode. Games can be launched directly from the home screen and can be paused and stopped during gameplay. The administrative mode was kept minimal, allowing a few settings for each game to be changed on a per user basis. One such setting option per game, is the ability to hide it from the home page’s game list. Once the prototypes were finalized, the team created views of the application using the storyboard on the development environment Xcode, v.11.5 (Apple Inc. 2020). Because iPhone screen sizes vary across device versions, the team programmatically positioned the same icons used in the prototypes to ensure uniform display across iPhone devices.

2.3 Gamification Element “Bubbles”

While the Bubbles Game looks simple, it has the potential to train children to learn two of the most essential touchscreen motor skills: targeted touch as well as touch and hold (a.k.a “press” and “press and hold”; Wrobleski 2010). When the user opens the game, they are faced with a blue background with the silhouette of a bathtub on the bottom. Placed upon this blue background are multicolored bubbles serving as the main touch targets in the game. The bubbles are fixed on the screen, but when tapped or “popped” by the user, disappear from the screen with an auditory “bubble pop” cue. When all of the bubbles are “popped”, a “You win!” dialogue screen appears informing the user that they have completed the Bubbles Game and giving them the chance to return home or play again. Upon pressing “Play again”, the same bathtub background is displayed but with different randomized locations of the bubbles on the screen.

This game is highly customizable, with the size and number of bubbles available to modify through the application’s settings page. The fixed screen size can accommodate more small bubbles onscreen than large bubbles. For this reason, the maximum number of bubbles onscreen ultimately depends on the set size of the screen. In terms of difficulty levels, a Bubbles Game set up with a low number of onscreen bubbles and large bubble size is substantially easier than a Bubbles Game with a high number of onscreen bubbles and small bubble size. Once a user displays proficiency at an easier configuration of the Bubbles Game, the parent or clinician can modify the game to display smaller bubble targets and a larger count of these targets in order to challenge the user. The game also includes pause, stop, and start functionality. At any point in the game, the user can press the pause button in the upper left corner of the screen to pause the game. The game will freeze and a pop-up will display with a “Resume” button allowing the user to resume the game upon clicking. The stop button is located in the upper right corner of the game. A button press will trigger a pop-up window providing “Restart Game” and “Return to Home” options. Selecting the “Restart Game” option will restart the game with a new randomized set of bubbles. Selecting the “Return to Home” option will bring the user back to the application’s home screen (Fig. 6).

Fig. 6.
figure 6

(Courtesy of the UCF Research Lab on AAC in Autism).

The Bubbles Game sequence of events, from start to finish.

Full size image

2.4 Gamification Element “Rocket”

The purpose of the Rocket Game is to provide additional training for targeted touch using a different scenario that mimics a sequence of button presses typically found on mobile AAC applications. The Rocket Game teaches its users to respond to a visual stimulus, the movements of the stars. The stars either pulse, shake or flash. When pulsing, stars repeatedly expand and contract. Shaking consists of the star moving slightly from side to side repeatedly. Flashing means that the star becomes transparent across a gradient and then returns to opaque. Each movement type lasts between 1.3 and 1.6 s, pauses for two seconds, and then repeats the animation until the user selects that star. When the activated star is pressed, the star stops animating, then a new star becomes activated and animates. The rocket icon advances towards the moon. On the last turn, the rocket reaches the moon, and the user wins the game. A popup with an explosion of animated stars appears, with the words “You won!” There is an option to return to the home screen or to repeat the game.

The Rocket Game has three levels of difficulty; four, six, or eight stars (see Fig. 7). Activated stars and animations are always random. Thus, the user will not be forced to continue tapping in the same spot each time and must move to a different one. Randomization of the sequence also makes the game different and unpredictable each time. Each difficulty level places the stars at slightly different places on the screen in order to properly accommodate the varying number of stars. For graphics, as in the Figma mocks, icons from Flaticon were used for the rocket, moon and stars. Star colors were chosen to be distinctive from one another and eye-catching, without being excessive.

Fig. 7.
figure 7

(Courtesy of the UCF Research Lab on AAC in Autism).

Images of each star configuration in the Rocket Game.

Full size image

2.5 Gamification Element “Dino”

The last game in SPEAKplay! is the Dino Game. This game aims to provide practice dragging elements on a phone to on-screen targets. It serves as a precursor to the “Drag & Drop” mode for building sentences within the SPEAKall! application. The user learns to press and hold onto a dinosaur and move it around to food and bones scattered around the screen. To reward and encourage the user to keep playing, the dinosaur dances for a moment as bones are collected before continuing. Once all the food and bones are collected, the user drags the dinosaur to its home to complete the game and reach the win screen. From there, the player returns to the home screen or plays again. When replaying, the food and bones are placed in new locations on the screen to provide a fresh game for the user.

The game has two modes, one with a free moving dino in Fig. 8 and one with a path following dino shown in Fig. 9. For the “free moving dino” mode, food and bones are placed all over the screen and the player moves the dinosaur anywhere on the screen. This mode allows the user to freely test dragging on a touch screen. To increase the difficulty of this mode, there is a setting to increase the amount of food and bones. For the path following dino, the game displays a pathway that leads to the dinosaur’s home. The food and bones are restricted to the path and the user is only able to move the dinosaur along the path. This mode helps the user learn to target a specific direction if they had trouble targeting food and bones in the first mode. The settings include an option to make the path in this mode longer or shorter to adjust the difficulty of the game to the user’s needs.

Fig. 8.
figure 8

(Courtesy of the UCF Research Lab on AAC in Autism).

The Dino Game: “Free Moving” Mode.

Full size image
Fig. 9.
figure 9

(Courtesy of the UCF Research Lab on AAC in Autism).

The Dino Game: “Path” Mode.

Full size image

3 Future Directions and Conclusion

3.1 Clinical Evaluation of SPEAKplay!

Current research efforts in the UCF Research Lab on AAC in Autism aim to generate proof-of-concept for the SPEAKplay! application by generating behavioral data as well as usability data. Behavioral data can be obtained by examining the effects of SPEAKplay! training on enhancing fine motor skills in the context of a single-case experimental design. Single-case research designs are one of the most rigorous methods to examining treatment efficacy and are ranked equally to quasi-experimental group designs in evidence hierarchies for AAC (Schlosser and Raghavendra 2004). These designs are typically examining pre-treatment versus post-treatment performance within a small sample of participants (Kennedy 2005).

Usability data can be generated by asking clinicians and caretakers to provide feedback on the user interface design and gamification elements by quantitative and qualitative survey. For this purpose, the UCF research team designed an initial usability survey containing eleven items. Nine questions required a Likert Scale response of 1 (Strongly Disagree) to 5 (Strongly Agree). Additionally, two open-ended questions were asked regarding needed improvements and desirable new features. The format was loosely based on the Treatment Acceptability Rating Form (TARF) by Reimers and Wacker (1988).

The survey was conducted after respondents had evaluated the SPEAKplay! application through a hands-on test of the latest Beta version with particular instruction to engage with the gamification elements. A total of twelve evaluations were returned by the participating practitioners. The respondents were primarily speech-language pathologists (n = 7, 58%; one person was also certified as an Assistive Technology Provider); the remainder included graduate students in speech-language pathology (n = 3, 25%), university faculty (n = 1, 8%) and a general education teacher (n = 1, 8%).

Results of the Likert Scale items indicate a predominantly “Strongly Agree” evaluation of the application as shown in Table 1. Comments generated through the open-ended questions were used to inform future development (see below).

Table 1. Summary of usability survey – SPEAKplay!
Full size table

3.2 Future Development

During initial usability testing, respondents were asked to indicate elements for improvement of the existing SPEAKplay! application and suggest any additional features that might improve future versions. This input will guide future development efforts. The majority of the respondents’ comments focused on three primary issues:

  1. 1.

    Fine-tuning of the motion abilities for the “dino” icon: Responses indicated a need to modify the sensitivity to detect user movement on the screen to better resemble a Drag and Drop gesture. Future work may focus on creating different maze scenarios with varying levels of difficulty. Potential activities may include further exercises to drag the “dino” icon to specific screen locations to train a variety of slide gestures.

  2. 2.

    Incorporating progress tracking and scoreboard options: Respondents articulated a need to monitor users’skill development. Such progress tracking will allow to better align the difficulty level for the various games with the current capabilities of the user. Adding on scoreboard and progress tracking options within the games can help users to stay engaged and motivated.

  3. 3.

    Expanding the tutorial within the application: This will help practitioners to quickly learn how to set up user profiles and program games within the application. Future tutorials may also include “best practice” lesson plans and ideas for seamlessly incorporating SPEAKplay! into clinical activities.

In addition, the SPEAKplay! interface can be replicated across a variety of mobile devices and platforms beyond the iPhone. This will allow to extend the motor skills training onto those devices that prospective AAC candidates ultimately end up using. Currently, SPEAKplay! supports a limited range of critical gestures (i.e., Directed and Sustained Touch, Drag and Drop, and Vertical/Horizontal Swipe). Future versions may add training for further gestures such as Free Rotate, Pinch, Spread and Flick that are commonly used in mobile applications designed for young children (Abdul-Aziz 2013). Accompanying research will refine the current SPEAKplay! design and validate its effectiveness to enhance motor access skills and operational competence for mobile touch screen technologies.

3.3 Potential for Gamification to Improve Clinical Practice

In sum, the results from the initial usability testing suggest that SPEAKplay! meets an important clinical need and has the potential to enhance the fine motor skills repertoire needed to access and operate mobile touch-screen devices. Training these motor skills in isolation would be a daunting and difficult to accomplish task. Adopting the principles of ‘serious game design’ into the SPEAKplay! interface creates an experience that has the potential to be more motivating and engaging for young users or beginning communicators who are making their first steps on mobile devices.

When designing this application, the UCF and Rose-Hulman research teams tried to adhere to the five serious game principles conceptualized by Whyte et al. (2015): 1) embed learning experiences into a motivating storyline (e.g., “feed the dino”), (2) create short term and long term goals that lead up to the targeted skills (e.g., “pop all these bubbles for a reward”), (3) have consistent rewards and feedback to mold behavior and provide guidance through difficult game levels (e.g., “aim for the winning screen”), (4) allow for individualized levels of difficulty that match the activity to the users current capabilities (e.g., “tap four, six, or eight stars to fire the rocket”), and (5) offer the user different choices to support autonomy (e.g., “chose between Bubble, Rocket, or Dino Games”). The utility of this framework goes beyond targeting motor skills in AAC users with severe autism and can easily be extended to other target skills and other target populations within severe communication disorders.

Serious game design holds a lot of promise to benefit acquisition and generalization of emerging expressive and receptive language skills for AAC users. Oftentimes, AAC intervention sessions are focused on repetitive, basic tasks such as choice making, labeling items, and producing short two- or three-term utterances, all of which could be enhanced through a motivating and engaging gamification experience. Inclusive design practices that are focused on seeking direct input from users and related stakeholders (e.g., Frauenberger et al. 2011, 2013) can help to identify those AAC scenarios and activities most appropriate for a gamification approach. Future research in this direction will help to understand the utility of serious game design in enriching mobile technology AAC interventions.