Keywords

1 Introduction

Given its inherently social and creative nature, storytelling appears to be an important activity to support children’s skill development [5]. In her survey on interactive digital storytelling for children [8], Garzotto gives an overview of systems and technologies. The use of virtual intelligent characters for storytelling, relying on advanced planning techniques [16], has been an important contribution to reach outstanding levels of both narrative generation and character representation. Supporting intelligent characters beyond the computer screen and exploring opportunities to make interactive storytelling more engaging for children [1] has been gaining more interest, offering collaborative storytelling in which multiple users can participate in the activity, and including tangible objects to build a more physical storytelling environment [15]. On the move towards this physicality, which would be appropriate for mediated technology with children, these tangible objects are often toys rather than characters capable of acting autonomously, although some related work explores the use of robots as characters. Combining tangible features with virtual intelligent characters is a promising line of research that is worth exploring, as it would allow us to pick features from both realities.

The research project coBOTnity aims to make a first step towards combining physical and virtual story characters by delivering moveable touchscreen-based bots, called surfacebots, with the purpose of supporting social and collaborative playful interaction around storytelling activities with children. Rather than a system unfolding a pre-scripted story autonomously, we understand the storytelling process as a creative and dynamic activity, in which children should be able to use the mediated technology as a means of expression and collaboration. The role that surfacebots can have is an interesting issue to study due to the multiple capacities that could be implemented. The surfacebots are envisioned as expressive agents (through the display of the touch screen) that can move around in physical space (e.g., on a table). They can be controlled remotely by the child to enact parts of the story. However, being expressive agents they could also proactively contribute to the story, e.g. by expressively showing their appraisal of story events, by autonomously moving to salient places in the story space, or by otherwise responding to the story as it unfolds similar to virtual intelligent characters in screen-based interactive storytelling systems [3]. Before implementing such interactive storytelling system with intelligent surfacebots, we have developed a working prototype with some basic features to tentatively see how children might use a surfacebot in their storytelling. It allows children to use a surfacebot character controlled remotely from a tablet to express part of the story they ideate. We carried out an informal exploratory study, observing how children interacted naturally with the prototype in order to assess which technological components are most promising to be further developed.

In this paper, we discuss this study and its implications for future work. Section 2 presents some of the related work. Section 3 introduces the working prototype that was tested. Section 4 summarizes the goals, observations and remarks from the tests. Section 5 outlines immediate future work to develop the prototype further.

2 Related Work

Here we review some of the related work on playful technology with a focus on social collaboration and storytelling involving physically embodied characters or robots. Garzotto [8] gives an overview of enabling technologies and approaches to interactive storytelling for children. A strand of work focuses on how to empower children as interactive storytellers, and especially how to make the activity collaborative (e.g. [13, 20]).

Some of the related work on tangible interaction in storytelling specifically involves tabletops and related tangible objects (e.g. [2, 4, 21]). In this segment, the Interactive Storyteller described in Alofs et al. [1] has a unique position because it integrates tangible objects representing intelligent story characters as agents capable of autonomous interaction on the tabletop. This kind of intelligence is missing in previous tabletop systems. One of the limitations of the Interactive Storyteller was, however, that the tangibles could not be actuated by the software, leading to problems of consistency between the digital and the physical representation. This problem could be solved by using robots to represent the characters.

With a different concept of tangible characters, the work in progress by Wang et al. presents Cartoon [23], a device with movable limbs which supports the recording of movement in a similar way to kinetic memory. It is used as a stand for paper or card-based images placed on top. Children can create moving creatures, plan the movement of the robotic bases, and put them to walk while telling stories.

The use of robots is also explored to either represent story characters, act as storytellers, or participate as companions in the activity. Examples of robots used as characters include GENTORO [19] and RoboTale [12]. In the first, a simple radio-controlled turtle toy is controlled by children using handheld projectors. It just follows the path indicated by the user without giving any affective responses or smart behavior. RoboTale combines a child-controlled robot with physical and virtual objects for collaborative storytelling. The control is mediated by tangible cards, which are recognized by the interactive tabletop on which the activity is taking place. In [17], stories are composed of programmed behaviors for Pleo, a dinosaur robot. It stands on a tabletop which tracks fiducial markers corresponding to sequences of behaviors indicated on a tablet PC by children.

Fridin [7] presented KindSAR, a system that uses a Nao humanoid robot acting as an embodied interactive storyteller, assisting educational staff by telling prerecorded stories, and playing a social role in supporting children’s literacy development and knowledge acquisition. A social robotic learning companion in a storytelling activity with children is presented in [10]. A tablet is used to display characters, and the child and the robot (tele-operated by the experimenter) engage in a turn-taking activity, exploring the range and kind of vocabulary being used [24]. Finally, Leite et al. [11] use two MyKeepon robots to explore the development of children’s social skills when interacting with fully scripted interactive narratives. They let children use tablets to choose the actions of a robot at specific moments, allowing them to see the effects of their choices. The experiment resulted in higher story recall when children interacted alone rather than in groups.

The related work shows that storytelling systems for children are starting to involve some active objects or robotic technologies. Different embodiments for robots have been explored so far, each with a different range of capabilities according to Milgram’s reality-virtuality continuum discussed in [6]. For instance, some have specific features such as dance or emotions, while others cannot deliver custom visual feedback but are able to use their arms to implement advanced gestures. Yet others cannot move at all, or only in a slow or clumsy way. So far, tangibles and robots have been mostly used in storytelling as either fully controlled, expressionless characters or storytellers that show some limited expressive behavior. In contrast, in our ongoing project we strive to develop physically present characters that are expressive and can provide some autonomous input in a storytelling activity.

Given the lack of affordable robotic characters, we take advantage of the widespread availability of tablets in learning contexts to integrate an extensible virtual agent on a tablet with a compact robotic base, with the aim of supporting future exploration of collaborative storytelling activities with robots. The following sections present a first informal evaluation of a working prototype of these ‘surfacebots’ to guide the further development of such an embodiment for children’s play.

3 Overview of the Working Prototype

We have developed a functional prototype to start exploring how children interact with a surfacebot while using it to tell stories. The choice of the components pursues a balance in features and costs with regard to other robotic platforms, while allowing children to interact through touch input remotely with the tablets available at school. The prototype consists of a robot-tablet (surfacebot) that embodies a character (see Fig. 1). Using a moving tablet to embody a character allows us to show a digital representation of the character’s facial expressions and intentions on screen while also allowing it to move around in the physical play area. However, before testing more advanced interactive features with children (e.g. self-driven and autonomous behavior in the surfacebot), we have focused on a limited version of a surfacebot character.

Fig. 1.
figure 1

Examples of screens on the surfacebot representing a character. Story assets are shown in the bottom left corner of the screen.

Full size image

The character representation can be changed dynamically by touching the surfacebot screen (see Fig. 1 for some examples) and the surfacebot can be driven around on the tabletop by using an additional control tablet. Images of story assets, mainly understood as props, can be displayed on the surfacebot screen by selecting the corresponding items from the control tablet (see Fig. 2-a,b) as a means of facilitating children’s expression. In the test described below we also provided physical versions (cards) of the story assets (see Fig. 2-c). The current prototype is implemented in Android running on 7-inch tablets. The surfacebot tablet communicates via Bluetooth with a Zumo Robot by Pololu hidden in the case to drive the movement. All tablets, either surfacebots or control, are connected via Wi-Fi through a wireless router by relying on an overlay network based on rosjava. It makes the prototype expandable, supporting multiple devices communicating commands and therefore sharing the state of interactions.

Fig. 2.
figure 2

Example of screens on the control tablet showing virtual story assets (a,b); physical cards showing the same assets (c); remote control (d).

Full size image

4 Exploratory Child-Surfacebot Interaction Study

We carried out an informal study of children’s interaction with a surfacebot prototype to inform future development. Inspired by the most relevant aspects from the related work discussed in Sect. 2, we focused on the following aspects:

  • Storytelling: What kind of stories do the children create with the surfacebot, and how structured are they?

  • Use of assets: How do children use the offered physical and virtual story assets? Do we see indications that they feel the need for more or different assets?

  • Character embodiment: Do we see indications that children see the surfacebot more like a character rather than a moving tablet used to play? Do we see indications for autonomous capacities it should have as a character?

  • Movement control: How well do children control the surfacebot’s movement? To what extent should the surfacebot move autonomously?

4.1 Context and Participants

A total of 22 children (12 male, 10 female) from 6 to 12 participated in our exploratory tests. Their parents or legal tutors provided an informed consent to participate. For logistic reasons, 13 children from 10 to 12 year-old participated on day 1 (average 10.92 y.o.), whereas 9 children from 6 to 8 participated on day 2 (average 6.7 y.o). Their interaction with the surfacebots was one among several interactive play activities organized by the university in collaboration with visiting schools. For this reason, we followed an observational and exploratory approach rather than an experimental one, as children could decide to join our activity whenever they wished, in groups or individually, and without possibility to control for study variables. This gives the exploration an “in the wild” nature, and we relied on annotations and video recordings as a means to confirm our observations.

In our setting, we had one surfacebot and a control tablet on some tables that were arranged together to set up a flat surface. Besides the virtual story assets which children could use during their interaction we also included physical cards representing assets. The assets depicted locations (e.g., island, planet), objects (e.g., treasure map, rocket) and characters (e.g., pirate, alien) in a pirate and space domain. There were some extra assets dealing with emotions and some related to general commonplace objects (e.g. tree, ball, etc.). The physical assets did not entail any virtual feedback on the robot, but allowed us to observe how children naturally use this format and assess to which extent a mechanism should be considered to take them as interactive input.

When children joined the activity, they were explained about the robot, the controls, and assets, and they were asked to create a story with the elements given.

The older children formed a total of 4 groups, ranging from 1 to 4 members, with a mean size of 3. The younger children formed 7 groups. Four of these were actually single individuals, resulting in a mean group size of 1.57. The time spent on the activity was higher in the case of younger children. In particular, young children spent on average 7.51 min versus 5.5 min for the older groups. These numbers are informative as can be a consequence of mixed reasons (group size, age, friendship, genders, etc.) given the exploratory nature of the sessions.

4.2 Observations and Remarks

Stories

Stories took place in the context of Pirates and Space, in line with the provided assets. It was out of scope in the exploratory sessions to assess the quality of the stories told by children. However, we still made some observations on the ability of children to structure the stories being told with the surfacebot. Some groups provided high story coherence, having links between events and what they decided to happen next [22]. For example, one of the stories involved a pirate finding a boat, using it to sail to an island guided by a treasure map, and finding an empty treasure chest that made him angry. Some children had difficulties to show this kind of logically connected events. In that case, they told a story with more unconnected events, moving the robot around and picking up things. Thus, including a strategy to support the development of structure and understanding would be beneficial. For instance, by enabling the robot to give clearer feedback on consequences of actions and new states, children could be guided towards the narration of more goal-directed action sequences. Although we did not observe a clear relation between children’s age and story coherence, we surmise that such support would be most beneficial for younger children who are still developing their storytelling abilities [18].

Use of assets

Both virtual and physical assets were used by the children during the storytelling activity. We observed that the physical cards were used in many cases to make sense of the space represented on the tabletop, positioning objects for the surfacebot character to find and locations to visit and thus essentially designing, on the fly, their own story world. For example, there was one episode in which the pirate found a rocket and used it to fly to Saturn to pick up the treasure; they placed a Saturn card considerably further away from where they were unfolding the story. Locations could be fairly abstract; a card with stars was used as “sky”, and was referred to as a location to fly to. Cards depicting characters were sometimes used to expand the cast of characters in the story. Thus, using dolls as additional characters can be interesting.

Children had expectations of the system giving some feedback or response related to the assets. For example, when using cards placed on the tabletop, they expressed their expectation of the robot saying something related to them when driving nearby. We also observed some children showing physical cards to the robot, expecting recognition and response. When using the virtual assets, some expected sounds in addition to the visualization. One child compensated for the lack of sound from the cannon by making the sounds himself. We also observed some cases in which the children incorporated imaginary assets in their story (e.g. “the prison was demolished with a hammer”, neither the prison nor hammer was provided), or adapted existing assets to their needs (e.g. “Airplane! Can’t find an airplane. Oh then, I’ll use a rocket”). We observed that with the physical cards, children sometimes combined assets (e.g. an alien with a pirate hat, or a boat full of pirates by joining all the pirate character cards). This kind of composition was not possible with the virtual assets, although some tried to sequence the virtual assets on screen. This triggers an important design issue, which is how to enable virtual compositions to allow children to express their ideas quickly.

Children used the offered assets as essential input for their stories, but they did not limit themselves to objects or props. Those assets representing inner states (e.g. being angry or happy) were used in both formats. Assets also triggered some reasoning. For instance, when they were discussing on where to place the Earth but using the “Mars” card for that, one of the children said “This cannot be Earth!” Thus, assets could and must be designed for educational purposes to trigger learning on new objects or concepts to leverage all the potential that an activity like this can bring.

Character embodiment

The surfacebot represented a simple character by means of a moveable tablet, capable of showing images but without any intelligent or advanced behavior. Still, children treated the character as being alive or animated, beyond a simple remotely controlled car. Observations supporting this idea are, for example, children pretending to give the robot a hug, or speaking “come to daddy”.

Children could change the character. This did not only happen at the beginning to establish the main character, but also during the story to pick another character or to just change the expression of the character (e.g. the friendly pirate was changed into an angry pirate, when he did not find money in the treasure chest).

As mentioned earlier, the children expected the character to give some feedback such as sounds, in addition to visual representations. One suggestion by children was that the robot should say something when it arrives at a card or should say what it finds. Another suggestion was to have the robot ask the user to do things or give assignments. During some sessions we explored how intervening on the story by having the character make some suggestion would work. We used a second control tablet, without children being aware of it, to visualize on the robot the asset that we wanted them to use. For example, we visualized the Earth to suggest that next action could be bringing the character to Earth. This is a feature that some groups appreciated, reinforcing in their comments that they liked that “the pirate had an idea”. However, children often failed to notice the robot’s suggestion, which indicates that additional and different feedback (sound plus animation) is needed to inform children that there is something that the character wants them to do.

Movement control

From the children’s comments immediately after the sessions as well as from our observations, we can state that driving the character is fun. Children engaged in driving the robot and it was one of the features they loved. However, we must be aware of some potential negative points. We believe that controlling the robot based on interactive low-level commands (i.e. go forward, turn left, etc.) can take more cognitive resources than desirable and therefore could have a negative impact on the storytelling process. In this sense, a control with high level commands (i.e. “go to here”, and it goes) should contribute to children having a better focus on storytelling process. None of the children suggested that the surfacebot should move autonomously; presumably because they had too much fun driving it.

5 Conclusions and Future Work

The test sessions produced a number of observations and remarks on different aspects that we have summarized above. These do not only serve as input to further develop the surfacebot-based storytelling system in our project, but would be applicable to expand similar interactive tabletop settings (e.g. [12, 17]) involving a robot and tablets as distributed controllers.

In response to our goal questions, we summarize the main points as follows. Having a surfacebot that can be controlled can help to organize events with some spatial and temporal coherence, but more support in the form of feedback must be included to scaffold the development of structure. Children managed to use assets in different ways in their stories, including more abstract concepts. We consider that virtual assets can be improved by defining feedback and robot responses, while specific tangibles or projected images on the tabletop can help in making sense of space in the story. Physical cards supported collaboration between children.

The surfacebot was treated more like an embodied character despite being remotely controlled without advanced autonomous digital features. However, children did expect to get more feedback from the robot, and they also had a positive reaction to the character proposing next steps, which is an interesting intelligent feature to be provided. We noticed that having low-level commands for movements, while it is fun, can diminish the purpose of the storytelling activity. Thus, autonomous behavior can be implemented to support the movement between locations in a higher level control. We believe that balancing or raising the interest of children in storytelling elements rather than driving the robot is important, and we must therefore ensure that driving the robot without creating a story is not possible nor takes too much cognitive effort. Towards the implementation of these improvements, we have started by integrating some functionality of the ASAP realizer [9, 14] into the surfacebot to support animated and expressive agents, as a way to improve the character embodiment and visual feedback. We envisage that one way to tackle the fun of driving while ensuring focus on the storytelling process would be to use a location camera-based tracker relying on attached fiducial markers, so that high-level commands can be used to get the robot moving among locations. Finally, we are working on an algorithm to allow the surfacebot to make suggestions and give feedback to children based on logically connected action sequences.

With these improvements, we aim to design and carry out experiments to explore scenarios involving multiple surfacebots, investigating the usage and management of such affordable and programmable robotic characters in storytelling activities with several children.