1 Introduction

As the number of children diagnosed with autism spectrum disorders (ASD) continues to grow [7], we are reminded of the needs of this community. In spite of great strides in early diagnosis and interventions, the outcomes for a majority of children with ASD are still poor, with few able to live independently when they reach adulthood [4, 10, 27]. To answer these needs, many in the computing field have begun conducting research on computer-based interventions for children with ASD.

Our research has concentrated on developing computer-supported activities to enhance the social skills of children with ASD with an emphasis on collaboration, coordination, creativity, compromising one’s interests with the interests of others, and understanding emotions. We have adopted multitouch tablets as a platform to support face–face activities and have developed a toolbox of simple, mistake-free, open-ended applications.

We have developed the activities and applications working with 26 children with ASD (ages 5–14), their teachers, special education staff, and parents. We have also consulted with a local support group for adults with ASD. We have made our software open source and developed it targeting commoditized hardware to maximize access and sustainability.

In this article, we present the activities and applications we developed, and how the children participated in the activities, concentrating on the experiences of 3 children. We then discuss the lessons we learned from the perspective of having designed technologies for typically developing children in the past and having a first experience with children with ASD. Our contribution is a novel use of technology with children with ASD in the form of a toolbox of simple applications and activities for multitouch tablets to enhance social skills, as well as a formative evaluation of our approach.

2 Related literature

According to an estimate by the Centers for Disease Control and Prevention, about 1 in 110 children in the US have an ASD [7]. The key characteristics of ASD are impairments in social interaction, verbal and nonverbal communication, and imaginative ability coupled with restricted and repetitive behaviors, activities, and interests [2].

The types of interventions with the most empirical backing for children with ASD use applied behavior analysis [11], which involves using clear instructions, repetition, practice, and reinforcement and are based on a behaviorist approach [3]. Increased use of this methodology is suggested as being behind the increasingly positive outcomes for adults with ASD who were diagnosed in preschool [10]. While greatly improving the chances of many children with ASD to live independently as adults, current estimates of adults with ASD who live independently are still low. Eaves and Ho [10] classified roughly half as having “poor” outcomes in spite of being diagnosed in preschool, and other estimates put the percentage of adults living independently between 4 and 12% [4, 27]. These results suggest there is a solid set of interventions that are having a positive impact on many children with ASD, but that more need to be done to increase the number of children with ASD who can live independently when they grow up.

2.1 Computer-based interventions

The past 10 years have seen a great increase in the number of computer-based interventions. Closest to our work are applications designed for multitouch displays that encourage social interactions and help children practice social skills. For example, Piper et al. [34] designed a tabletop application for children with ASD in the form of a four-player cooperative game. The authors found the game provided students with an engaging experience for group work, something they usually find challenging. Hendrix et al. [21] studied the design of a tangible tabletop application to engage shy or socially withdrawn children in games by giving those roles that encouraged other children to engage with them in a positive manner with promising results. Gal et al. [15] conducted a 3-week study with 6 children diagnosed with ASD (aged 8–10) using Story Table, software implemented on a Diamond Touch multitouch surface that used enforced collaboration in the context of storytelling. They observed an increase in children’s responses to peers, with more positive affect, and greater likelihood to express emotions. Multitouch tablets have the potential of providing similar advantages to tabletop displays in terms of encouraging pro-social behavior through sharing an interactive surface, while providing advantages in terms of cost, availability, flexibility of use, and mobility. They can also enable additional social behaviors such as passing the device to a partner.

Tangible technologies have also been used to encourage social interactions. For example, Farr et al. [13] used Topobo and LEGO toys with groups of children with ASD and noted the differences in playing styles, finding that Topobo led to more social forms of play. Somewhat related are interventions that use robots to encourage social interactions [14, 36].

Some mobile technologies have been designed to support and encourage communication with others. For example, Madsen et al. [30] use mobile computers loaded with software that automatically classifies emotions on human faces. Tentori and Hayes have also proposed mobile devices to support social activities in school and other controlled environments [38]. Less sophisticated approaches include speech-generating devices (e.g., DynaVox) that use picture communication symbols or text and are very costly but have shown promising results for some children [5].

There are also technologies that blend into children’s everyday environments and support them while not getting in the way of engaging with others. Hayes et al. [20], for example, studied the use of computer-based visual supports for communication, scheduling and recording of images for children with ASD. Hirano et al. [23] expanded on the scheduling system called vSked, which replaces the paper-based visual schedules that are commonly used in classrooms with children with ASD. The goal of the system was to mimic these paper systems while reducing teacher burden and automatically generating records and reports. The system was evaluated in a classroom with 9 children with ASD (aged 8–10) and was well received by teachers, students, and other stakeholders. Other research on embedded or pervasive approaches to support children with autism has involved tools that help keep track of children’s behavior [1, 19, 29, 40] and that encourage certain behaviors [33].

Finally, there has been a wide set of computer interventions that work with traditional desktop or laptop setups and have led to positive results in areas such as building vocabulary, encouraging vocalizations, and learning about appropriate forms of communication [6, 8, 9, 12, 13, 17, 22, 28, 31, 41]. These desktop- or laptop-based technologies are generally intended for individual use, potentially limiting social interactions while they are being used, while in other cases interactions with another person are accomplished through the computer [37].

3 Research objective

Our research objective has been to explore the potential of multitouch tablets to engage children with ASD in social activities and enable them to better collaborate, be creative, express themselves, compromise their interests, and understand emotions.

4 Research setup

4.1 Participants

We iteratively developed multitouch tablet applications and activities by visiting 2 sites on a weekly basis: Hoover Elementary School in Iowa City, Iowa, and the Four Oaks afterschool program for middle school children with Asperger’s syndrome in Cedar Rapids, Iowa. From February to May of 2010, we worked with 16 elementary school children and 10 middle school children over a wide range of the autistic spectrum, from children who communicate mostly with picture cards, to highly functioning children who are fully integrated in classrooms with typically developing peers. In total, we conducted 13 two-hour sessions at each site.

At Hoover Elementary, we collaborated in every session with a staff member from the local Area Education Agency who delivers social skills training on a weekly basis. She provided ideas for activities and helped us refine existing activities on every session. We also had contact in every session with the lead teacher assigned to lower-functioning children at the school and the aides who worked with him (3–4 depending on the day). They helped us by participating in some sessions with children they work with and by providing feedback on children’s specific needs and interests. We also had a meeting with additional teachers and staff (about half a dozen) at the beginning of our activities and had occasional interactions with them throughout our activities.

At Four Oaks, we obtained feedback and ideas from the Program Coordinator, who would observe our activities on every session. In addition, we collaborated with three other staff members who would often participate in our activities, either by observing and providing feedback, or by actively engaging with the children through our activities.

We have also attended meetings of a local support group for parents of children with ASD, meeting with them three times during our research activities. In these meetings, we updated them on our work and received feedback from them. This support group includes about 100 families, with 10–20 people in attendance at a typical meeting. We also attended a meeting of adults with ASD who belong to the Iowa City chapter of the Global and Regional Asperger Syndrome Partnership (GRASP). In the meeting, we presented our work and received feedback from the members present (about a dozen).

4.2 Materials

We decided to use multitouch tablets for our research with children with ASD to take advantage of the strong interest most children with ASD have in computers and combining it with the social dimensions that can be added through the use of a multitouch surface. While other single-display groupware methods could also be made available, there is also evidence that touch screens are easier to use for some children with ASD because they do not force children to associate moving a physical device with moving the cursor on the screen [39, 41].

We used a Dell XT2 multitouch tablet in all our sessions. It has a 12.1″ screen that can detect up to 4 simultaneous touches. It also has a stylus, but the stylus may not be used simultaneously with touch. We developed the software using PyMT [18], a Python-based cross-platform toolkit for developing multitouch applications that we helped develop, which supports other platforms including multitouch tabletops.

4.3 Research activities

We began our activities by working with two girls with ASD in the summer and fall of 2009. Our purpose, based on previous literature, was to design applications that were simple [15, 16], predictable, favored the visual medium, and took advantage of children’s strengths [35]. In addition, we wanted our applications to be mistake-free to reduce frustration (e.g., no error messages, no wrong answers, clear mappings between actions, and system states), and open-ended, to increase flexibility by enabling a variety of activities. Through visits and demos, we discussed the kinds of multitouch applications the girls would be interested in using and asked them to try some available with PyMT. Based on these sessions, we adapted and developed the first versions of the four applications we used during our intensive research activities in the winter and spring of 2010.

We conducted these research activities with the purpose of iteratively improving the applications and developing related activities while conducting a formative evaluation of their impact. We did so in three different settings, two at Hoover Elementary School and one at Four Oaks. In all settings, we videotaped sessions so we could document the evolution of applications and activities, and their impact on children’s behavior.

4.3.1 Sessions with lower-functioning elementary school children

At Hoover Elementary School, we worked with children in two settings. The first was a classroom setting where we would work with 1 or 2 children at a time as the others were conducting activities outside the classroom. In total, we worked with 11 children in this setting, ranging in age from 5 to 9 years old. They were all boys. All these children were lower functioning and were in classrooms with other children with ASD. Counting teachers, aides, and special education staff, there was a 1–1 ratio of adults to children in this setting. We usually worked with about half the children in each of our visits.

In these classrooms, we would typically bring in an application and present an activity to the child or children. We would show them how to interact with the application and then let the child try it. By observing the children, we obtained valuable information to improve both our applications and the activities to conduct with them. When children struggled with our activities, they would quickly show it through disinterest (e.g., standing up and walking to another part of the room) and vocalizations (from grunts to requests for other activities). When they enjoyed the activities, they would let us know by using words (e.g., “I love this”), smiling, giggling, laughing, and showing a great amount of focus and interest in the activity. Even though most of these children tended to speak very little, many were more than willing to make recommendations on improving the applications. Other times, if we wanted to get feedback on specific features, we would ask yes/no questions and write “yes” and “no” on separate sticky notes and let them pick one to tell us their answer. We were pleasantly surprised at the amount of input we were able to gather from children, which ended up having a direct impact on applications and activities. We also received many suggestions from the special education staff and teachers who participated in these sessions.

4.3.2 Sessions with higher-functioning elementary school children

At Hoover Elementary School, we also conducted sessions with higher-functioning children. These sessions occurred during “lunch bunches”, when children with ASD who are integrated in regular classrooms invite 3–5 of their typically developing classmates to have lunch with them in a classroom that is otherwise empty. These “lunch bunches” provide the children with ASD with a chance to socialize with others in a setting that is less distracting than a typical cafeteria. We worked with a “lunch bunch” for second-grade children (7–8 years old), which included 3 children with ASD (2 boys and 1 girl). We also worked with a “lunch bunch” of fifth-grade children (10–11 years old), which included a boy and a girl with ASD.

At these “lunch bunches”, we would usually introduce an activity and have the group of children participate in it. We most often got feedback from the group as opposed to getting feedback from individual children or pairs of children. The children with ASD in these groups were as vocal as their typically developing peers in terms of providing feedback.

4.3.3 Sessions at Four Oaks afterschool program

At the Four Oaks afterschool program, we worked with 10, 12- to 14-year-old children, 2 girls and 8 boys. We usually observed the children taking part in social skills training and then would work with 1–3 children at a time. All the children were diagnosed with Asperger’s syndrome and were able to communicate without much difficulty. They were more likely to suggest specific changes to applications and/or activities than the children at Hoover.

5 Description of applications and related activities

While our research at Hoover Elementary School and Four Oaks concentrated on developing activities, all activities centered around applications, which were also improved in the process. Hence, we organize the descriptions below by application. Table 1 shows a summary of the applications, their related activities, and targeted skills.

Table 1 Summary of applications, their corresponding activities, and targeted skills
Full size table

5.1 Drawing

We have implemented a drawing application that enables children to draw with a stylus and zoom in and out and rotate their drawing using their fingers. Moving fingers closer together makes the drawing smaller. Moving the fingers apart makes the drawing larger. The drawing can be rotated with two fingers by moving finger either up or down with respect to the line formed by the two fingers when they first touch the screen. This set of gestures is commonly referred to as the “pinch” gesture and is already used in other multitouch devices such as the iPhone.

We have used this freehand zoomable drawing application for two types of activities. The first is to enable children to express themselves in ways that were not previously possible. The application enables children to do this through its zooming capabilities. This can lead to children creating something that they want to share with others. It may also enable children to more easily express how they feel. For example, one girl let us know how she felt uncomfortable being observed by drawing two big eyes looking at a girl.

Using the same software, we have also conducted collaborative storytelling activities with the goal of having children with ASD socially engage with other children (both typically developing and diagnosed with ASD). For this activity, we have a group of children sit around a table (we have done this with groups of 2 to 6 children). We then ask one child to begin telling a story by drawing something on the touch screen. The child then passes the tablet to the next child, who gets to continue the story by adding his/her own drawing. If it is a group of two, he or she then passes it to the first child; otherwise, he or she passes it to the next child. In some cases, we have had members of the research team and other adults participate in the activity as well.

This activity leads to high-quality social interactions between the children. Both children with ASD and typically developing children show great interest in how the story will turn out, asking their peers about what they are drawing and what will happen next. It is also an exercise in compromising their interests with those of others. For example, in a session with a group of elementary school children (two with ASD, three typically developing), the two children with ASD added to the story an element related to their strong interests: one drew a dinosaur, the other a frog. However, they had to see and accept their interests as being part of a larger story where they interacted with a number of other things that do not necessarily interest them.

5.2 Music authoring

We have developed a music authoring application that turns the screen into a harp-like device. Children are able to select tiles set up in rows and columns that represent individual notes. Each column’s notes enable children to select from low (at the bottom of the screen) to high notes (on the top). The columns are played in sequence, with the column currently being played turning green. Children can also adjust the tempo. Figure 1 shows an elementary school boy diagnosed with ASD playing with the application.

Fig. 1
figure 1

Composing music

Full size image

The application provides children a chance to create something they can share with others. It is also an activity that enables them to have fun with something that does not involve one of their specific interests. We have also tried using this application with multiple children simultaneously, but it seems to work best when children work on it individually. Multitouch displays still help if an adult needs to explain to children how to use the application. The application may also provide a window into children’s minds, as many of the children with ASD (not all), often try to form specific visual patterns when creating music.

As an alternative, we have used this application for collaborative music authoring. In a similar way to the way we use the Drawing tool for collaborative storytelling, we ask children around a table to add a few notes and then pass the multitouch tablet to the next child. This activity has proved quite engaging and entertaining for the children. On one occasion during a “lunch bunch” session, and more recently again at Four Oaks, we ended up with all the children dancing to the music they made together. We expect that enabling children diagnosed with ASD to enjoy this type of social experiences is likely to help them appreciate social interactions and be more willing and open to experience them.

5.3 Untangle

This application presents children with a visual puzzle to solve. The puzzle consists of a set of circles, each connected to 2 other circles by lines. To solve the puzzle, none of the lines should overlap. We have the option of presenting 10–25 circles in the puzzle, with more circles increasing the difficulty of the puzzle. We can also make half the circles one color and half another color to provide an activity that requires greater coordination if the children are assigned one color of circle to move. The goal of this application is to encourage communication, collaboration, coordination, and visuospatial thinking. Figure 2 shows two children collaborating while using Untangle.

Fig. 2
figure 2

Two children collaborating on Untangle

Full size image

We are very encouraged by what we have observed when children use this application in pairs or in larger groups. It naturally encourages collaboration and coordination without requiring it. Both children with ASD and typically developing children enjoy playing with it, and most of the time accept and even appreciate solving the puzzle together with other children. Even when playing on a 12.1″ tablet screen, we have had groups of up to 5 children simultaneously work on the application. Using Untangle, we have been able to successfully engage children from kindergarten to middle school children in collaborative activities.

Because moving a circle changes the overall puzzle, children have to coordinate with each other what they are doing. This has led in several occasions to children with ASD making suggestions to the other children. The puzzles also led the children with ASD to share in the joy of solving a puzzle with their peers.

5.4 Photogoo

Photogoo enables children to distort an image by dragging their fingers on the touch screen. It also supports drawing on the picture by using a stylus. We have conducted two types of activities with this software, both involving faces.

The first activity is about exploring emotions. This activity has been particularly useful with lower-functioning children who are working on recognizing emotions on faces. With Photogoo, they can modify the faces of their favorite cartoon characters to express specific emotions. They can do this by dragging the ends of the mouth, changing the size of the eyes, and so forth.

Figure 3 is from a session with a 9-year-old boy who primarily communicates through picture cards. The images show a sequence in which a special education aide asked him to modify the face of a cartoon character to show the following emotions: sadness, happiness, fear, and surprise.

Fig. 3
figure 3

Making a cartoon character look sad, happy, scared, and surprised

Full size image

We later expanded the exploring emotions activity by presenting children with a scenario and asking them to show us how the character or person in the picture would feel. This encouraged children to think about the feelings of another first and then find a way to show those feelings on their face.

6 Case studies

Below, we discuss the experiences of three children with our activities to provide a sense of how they evolved in their participation based on a review of video from all the sessions we worked with them. We do not use the children’s real names, but use their real gender and age. Every child we worked with had unique needs and abilities. The three children below had only “typical” experiences in that they tended to prefer one activity over others and evolved in their behavior over time with mostly positive outcomes.

6.1 Beth

Beth was 13 years old and diagnosed with Asperger’s syndrome when we conducted research with her at Four Oaks. Outside of participation in our activities, she was always very quiet and tended to be sad or upset, complaining about headaches or things that happened earlier in the day. She was very hesitant to participate in our activities most of the semester. However, toward the end, she made a very positive turn, finding the collaborative storytelling activity fun and enjoying its social aspects. Below, we present a brief overview of how she participated in our research activities.

We worked with Beth for the first time on February 15, 2010, when she used Untangle and Photogoo with Jane and then used the Music authoring application and the Drawing application by herself. She was mostly quiet, but surprised us with her artistic abilities and the way she expressed herself when she drew a girl starting with details, then moving to the whole, with two eyes looking at it (see Fig. 4). This was consistent with the way children with ASD process information, favoring local over holistic processing [32]. Her drawing was of such quality and the process she followed was so unusual that we showed it to two Arts faculty at our university who were also impressed by her ability to draw and surprised by the way in which she drew, moving from the details to the whole.

Fig. 4
figure 4

On the top, Beth draws squiggles as she works on the details of her drawing. On the bottom, she reveals the figure she was drawing by zooming out and adds 2 eyes looking at it

Full size image

Beth declined to participate in other sessions until March 22, when she joined two boys playing the music application. While she did a good job giving up her turn, she sighed loudly when the other boys took too long a turn. Then, they played with the Drawing application, which she really enjoyed because it plays to her drawing strengths. She made a detailed drawing and explained what it was when she was done. However, she still did not engage in the collaborative aspects of the storytelling activity. Instead of building on the story, she drew something independent of the story. She did provide common-sense suggestions for improvements to the Drawing application (e.g., adding colors, saving drawings). She also vocalized one of the things she liked about it: “when you zoom in there are so many gaps you can fill in”.

Beth again declined to participate in activities until May 3 when she had her first highly positive social interactions as part of our activities. She participated in a collaborative storytelling activity with a boy, passing the tablet back and forth several times. In this activity, she described what she drew while laughing and made great use of both her artistic skills and the zooming capabilities of the application. She was amused by how the collaborative story evolved, saying things like “that is absolutely funny!” She said she found it fun to hide drawings within drawings. A week later, she had a similar experience with the same activity. The interesting thing in this case is that she was complaining about a headache and having a rough day as she began the activity, but very quickly started laughing and enjoying herself, and stayed in a much better mood after participating in the activity. In both sessions, she followed the pattern of beginning to draw with squiggles that together form an object and then zooming out to make a different object out of her original drawing.

With Beth, we see an example of how our activities can make a positive difference for a child, helping her express her feelings, enjoy social activities, and feel better about herself. Beth also highlights the importance of our toolbox approach: only the collaborative storytelling activity seemed to help her, likely because it played to her strengths.

6.2 Jane

Jane was 14 years old and diagnosed with Asperger’s syndrome when we conducted research activities with her at Four Oaks. While Jane was always happy to participate in our activities, at the beginning of the semester she would very quickly lose interest. It would not be unusual for her to participate for less than a minute and then say she was done. By the end of the semester though, she became more comfortable with the activities and found ways to enjoy the social interactions with other children. During the last session, her mother had to plead with her to go home, as she was engaged in explaining to another child how to play with the Untangle application.

As mentioned earlier, during our first session at Four Oaks, Jane played with Beth, starting a bit of a pattern for her interactions where she tended to dominate collaborative activities and lose interest quickly. Although the novelty of our technology kept her involved for several minutes during this first session, she still left early, with Beth continuing to use two more applications.

The following week, she used the Drawing application by herself and did a few small drawings, quickly moving from one to the next. In all, she only used the application for a bit over 2 min.

The following 3 weeks, she participated in activities with boys from the program. She made attempts at being social, sometimes following suggestions from partners, asking about their feelings, and seeking commonality with them (e.g., noting that she and her partner are both left-handed). However, she seemed very anxious, quickly wanting to move to the next activity or leave. She also had difficulty giving up her turn.

The March 29 session was critical for Jane to build confidence using one of the applications. She played Untangle by herself and not only enjoyed solving puzzles, but improved significantly. This became very useful a week later when she solved puzzles collaboratively with a boy using the two-color version of Untangle. She used social language saying “I’m sorry” when getting in the way, asking the other boy to move specific circles, accepting suggestions from the boy and seeking confirmation saying “move this one?” Together, they quickly solved the most difficult puzzle.

During the last two sessions, Jane showed great improvement in her social skills. On May 3rd, she participated in activities with another boy. They began by playing Untangle together, discussing what strategy to pursue, which Jane enjoyed very much, making suggestions and asking for feedback (e.g., “where should this one go?”), and encouraging the other child (e.g., “we almost got it [NAME OF BOY]!”). This transferred to using other applications, where Jane for the first time truly participated in creating a visual story in a collaborative fashion with someone else, as they built on each other’s drawings. She also showed a tremendous change in the amount of attention she paid to the tasks and her partner and had no interest in stopping the activity. During our last session, Jane showed a new boy in her program how to solve puzzles with Untangle and was so interested in the activity that she made her mother wait for several minutes before leaving.

Jane shows the importance of learning about these interventions longitudinally. Had we only worked with Jane during a couple of visits, she would have never developed comfort with using the technology, which ended up having a very positive impact in her social interactions with other children. We also saw again the importance of having a toolbox of activities instead of just one computer intervention. For Jane, the activity that made the difference was solving puzzles with Untangle, something she enjoyed doing by herself and with others, and which triggered pro-social behaviors that went beyond the activity she most enjoyed.

6.3 Robert

Robert was 9 years old when we conducted research activities with him at Hoover Elementary School. He was diagnosed with Autism and was perhaps the lowest-functioning child we worked with. He was able to sometimes communicate using single words, but relied mainly on picture cards for communication. He also went through a regressive period in his behavior during our research activities, which was reflected in the way he used our applications. Robert provides an example of the challenges and potential rewards that should be expected when working with lower-functioning children with ASD.

On February 10, we saw Robert first using one finger to manipulate a picture in Photogoo. He behaved very repetitively, making an action upon the picture and then requesting a new picture. A special education teacher then asked him to play with the picture three times before he could receive a new one, which he complied with, but he kept doing the same action on the photograph (stretching it out), each time using broader motor motions. As he worked with other photographs, his motions got broader, although at one point the special education teacher modeled an action and he copied it a few times.

We saw Robert again 2 weeks later and learned that he had not stopped telling parents and his closest aide about Photogoo. Having his most trusted aide present, he made small incremental motions instead of broad strokes. He then managed to follow the prompts of his aide and modified the face of a cartoon character to show certain emotions, as shown in the description of Photogoo. Using a different picture, he then followed some prompts and also modified the picture to show emotions unprompted.

The next time we saw Robert, without his close aide, he reverted back to his earlier broad, repetitive motions when using Photogoo, in spite of prompts by us and another aide. We continued to see the same pattern in the following visit and a teacher told us they had observed him regress in general during the past few weeks.

For the last two sessions, we had made a user interface change to Photogoo. In the original version, a tap of the stylus on the screen allowed children to select a new picture. At the request of other children, we added the ability to draw on the picture with the stylus, which meant that we had to provide a new way of selecting a new picture. This change greatly frustrated Robert, who kept trying to use the previously available interaction even though we told him many times that it would not work.

Robert highlights the opportunities and challenges of working with lower-functioning children. On the positive side, we were able to find an application that greatly appealed to him and enabled him to explore emotions, something very important for lower-functioning children with ASD. On the negative side, we learned about the challenges of working with children who often take two steps forward and one step back in development. We also learned about the impact of user interface changes in children who are very rigid in their approach to the world.

7 Feedback from children

We specifically asked the children in the “lunch bunches” at Hoover Elementary School, as well as the children at Four Oaks to provide us with specific feedback on our applications and activities during our last session.

At Hoover Elementary’s “lunch bunches”, we went around the table asking the children to tell us likes and dislikes for the Drawing, Music, and Untangle applications.

The children emphasized their enjoyment of the social aspects of all three applications. For example, for Untangle they “liked working together”. For the Drawing application, they liked the collaborative storytelling activity as they mentioned that it gets more interesting as more people add story components. They also liked zooming in and out and that “you can make anything, even crazy stuff”. Similarly, for the Music application, they said that the music created together with friends was the best part.

They had a constructive approach to dislikes, as they concentrated on suggesting features to expand the functionality of each application.

At Four Oaks, we asked the children to write on sticky notes three things they liked, three things they did not like, and three things they would like to change about each application. We grouped the sticky notes in an affinity diagram and discussed with the children what they said.

Again, children made it clear they enjoyed the social aspects of applications. For Untangle, they liked that they could “work with someone else to solve it”. For Photogoo, they said they particularly enjoyed modifying their own faces, or those of friends. For the Music application, they liked the sound of the music they created with friends.

Similarly to the children at Hoover, the dislikes and suggestions concentrated mainly on proposing new features to be added to the applications, such as color to the Drawing application, or more instruments for the Music application.

8 Lessons learned

Below we discuss the lessons, we have learned from our research with children with ASD. Many of these re-enforce what others have observed in the past. We believe it is important to present them though given the high variability between children with ASD and the small number of children who have participated in individual projects. In addition, we present these from the perspective of having worked on designing technologies for typically developing children for more than 10 years in projects such as KidStory and the International Children’s Digital Library [24, 25] and learning about the impact of technologies in school settings [26].

8.1 General observations

8.1.1 Technologies can help us learn more about children with ASD, how their minds work, and how they relate to the World

Seeing the different patterns, children with ASD follow when using technologies as compared with typically developing children can help us learn about how they process information, and how they see the world. We observed Beth using the Drawing application draw the details first and then zoom out to draw the “big picture”—an opposite pattern from the typically developing child’s process of outlining a drawing, then zooming into fill in the details. As we mentioned earlier, this may reflect the way some children with ASD process information differently from typically developing peers, paying greater attention to details instead of the whole [32]. We also observed children with ASD systematically create geometric patterns in the Music Authoring application more often than typically developing peers.

Nonverbal children may express their thoughts and emotions that they may otherwise have a difficult time sharing, through the use of technologies. For example, the Drawing application and the storytelling activities we pursued with it were very useful in learning about children’s interests. Some children would usually draw the same type of item (e.g., a dinosaur, a frog, a mummy). Others would usually stay within a particular topic (e.g., video games, cars).

Some of our activities also helped us learn about children’s perceptions of others. With Beth, we learned about how she felt about being observed when she drew two large eyes looking at a person during her first session with us. Other children found a way to tell us about other people’s feelings when using Photogoo. For example, when modifying the faces of members of the research team, some children would draw what they thought we were thinking on our forehead.

8.1.2 Technology may be enough of an incentive to improve the quality of social interactions

In earlier studies, Tartaro and Cassell [37] found that children with ASD had more quality communications with an embodied conversational agent than with another child. We have some indications that this may also happen when children interact with each other in the context of using a technology that they are both interested in using and find entertaining. In spite of the variability between children, interest in computers seems to be consistent. When we asked a group of eight higher-functioning middle school–aged children an open-ended question about their interests, all spontaneously answered some variant of “computers.” Though social interaction is initially frightening, challenging, and undesirable, when paired with technology aimed at enhancing social skills, the process might be a reward within itself. With the multitouch technology we have been using, the computer itself becomes the recipient of the participants’ focus, providing a more structured narrative for the interaction patterns, rather than the usual open-endedness of social interaction, which perhaps reduces the anxiety of initiating social interaction. Also worth noting is that when we asked children to give us specific feedback on what they liked about our activities, they consistently mentioned the social aspects of the activities (e.g., “working together”).

8.1.3 Create safe spaces in which children can explore

Because many of these children dislike unknown outcomes, it is important for them to feel comfortable with a technology before they are asked to do more challenging tasks. With one higher-functioning child, we noticed his hesitancy in interacting with our applications, though he paid attention when his peers played. However, when he was given the opportunity to play Untangle, an application in which he naturally excelled, he readily accepted. Whether this was due to his desire for perfection or an unwillingness to explore and possibly fail in front of others, it might benefit children like him to give them a safe space in which they do not feel judged to first experiment with a technology until reaching a level of personal comfort. We saw something similar with Jane having to build up her confidence with Untangle before she could engage in positive social interactions with other children.

8.2 Design methods

8.2.1 Design activities along with the technology

We have observed that the activities that we asked children to conduct with technology have a significant effect on the social outcomes. Because we want our technologies to be used at schools and homes without the intervention of researchers, it is crucial that we provide specific activities so teachers know how to make the most of the technologies. Providing the technology without social supports is unlikely to lead to gains in social skills.

While this may not be as necessary for behaviorist-inspired technologies, we believe it is of utmost importance for technologies that allow children to express themselves in an open-ended way or interact with each other socially. For this reason, when designing and evaluating technologies for children with ASD, the activities need to be designed and evaluated together with the technologies.

8.3 Evaluation

8.3.1 Important to know how children behave in contexts without technology

Evaluation of behavior must be set against a baseline to see how ASD children behave in contexts without technology. What may seem like little engagement in a technology-related activity may actually be a great improvement compared with what happens in other contexts. What may seem like a typical use of social skills when technology is involved or a poor allocation of attention may be a tremendous improvement of the social skills and focus of attention used in other contexts.

8.3.2 Important to have a baseline with typically developing children

On the other hand, what first appears to be an “ASD behavior” might actually be a behavior seen with typically developing children as well. With our music application, each touch to a tile on the screen activates a new note. When the ASD children interact with the application, they often do not try to systematically create a musical piece, but instead try to select all the tiles, which creates a loud banging noise, or they press many random tiles quickly. We attributed this to being an ASD behavior, but later observed this phenomenon in a group of typically developing children of a similar age. This example highlights the importance of having a baseline of non-ASD children to set behavior against.

9 Future work

During the 2010–2011 school year, we are conducting a more formal evaluation of the use of our applications at Hoover Elementary School and Four Oaks. Tablets with our applications are available full time to teachers and staff, enabling us to better establish the impact of more intensive, longer-term use. We are using standard questionnaires to assess the impact of conducting activities with our applications on the social skills of children both at school and at home. We are also focusing our video recordings of sessions on children’s faces, to be able to better study facial expressions and eye contact and compare children’s behavior when participating in our activities to their behavior when participating in other activities.

We have also made our applications freely available at http://www.openautismsoftware.org, together with guides to their related activities. We have had more than 400 downloads of our software so far, and visitors to our website have come from 62 countries.

Our long-term plans are for this effort to be the beginning of an open-source development community dedicated to providing solutions for children with ASD. We believe the autism community is a particularly good fit for such an effort because of the prevalence of parents of children with ASD in engineering and computing fields [42] and their great dedication to their children. We also think that this approach will provide gains in terms of access and sustainability. In particular, we hope it will make it more likely for efforts from academia and even individual parents to have a greater impact and continue even if the originators of these efforts have moved on to other endeavors.

10 Conclusion

We have presented our experiences conducting activities with multitouch tablet applications designed to promote collaboration, coordination, creativity, compromising one’s interests with the interests of others, and emotion understanding in children with ASD. In doing so, we explored a new platform to approach children with ASD in multitouch tablets. This is particularly relevant given the recent surge in the use of tablet computers such as the iPad. We took a different approach in providing many simple, open-ended, mistake-free applications instead of concentrating on one, and in focusing on activities as opposed to technology. The results so far are encouraging, with evidence that activities with our applications can lead to pro-social behavior, enabling children with ASD to enjoy social activities, develop appropriate social skills and express themselves. We hope this work inspires others to continue making use of tablets and other novel technologies to support and encourage face–face social activities for children with ASD.