Building Narrative Experiences for Children Through Real Time Media Manipulation: POGO World

Abstract

POGO world was designed between 1997 and 2002. Returning now, 15 years later, what meaning could it have? What is being done today in the high-tech development laboratories that think and design the classes of the future? What educational philosophies are these developments based on? How are educational environments designed? What are the resources for prototyping cyber-physical systems?

FormalPara Author’s Note, Funology 2

• POGO World Fifteen Years Later

• Antonio Rizzo, Francoise Decortis and Edith Ackermann

To return fifteen years later on POGO designed between 1997 and 2002, what meaning can it have? Answering this question calls for more questions: what is being done today in the high-tech development laboratories that think and design the class of the future? What educational philosophies are these developments based on? How are the educational environments designed? What are the resources for prototyping cyber-physical systems? Since the last century, activities that are considered important for children have changed radically (Yarosh et al. 2011). Yarosh and colleagues provide an overview of these dimensions and the values promoted by the international community of researchers and designers (e.g. “Interaction Design for Children”, IDC), which have guided and led the design of technologies for children since the 2000s. They point to five orientations that have remained immutable throughout the last ten years of the IDC community: (1) a contribution to the social, intellectual and creative development of children; (2) a vision of children as active agents in the appropriation of technical devices, which requires involving children in the design and evaluation of these devices; (3) anchoring in the perspective of constructivism and constructionism; (4) a preference for a new digital ecology in which traditional devices interact with new ones; and (5) the creation of gateways between physics and digital.

We suggest that the EU POGO project has been at the heart of the movement identified by Yarosh and his colleagues, and which, before his time, is an integrative (ecological), reflective (theoretical foundation), experiential (GUI to NUI), and methodological (empirical, grounded in field situations) example of this movement.

POGO has ventured on the path of augmented spaces, communicating tools, intuitive interfaces, seamless integration of the digital and the physical, making use of of the most advanced technologies, based on a vision of the future that integrates an aesthetic of innovative interaction modalities. POGO was a research project based on an educational philosophy rooted in the work of psychologists such as Bruner, Vygotsky, Piaget and Dewey, whose contribution has been to refocus education on building knowledge and know-how while they are engaged, with others, in projects that interest and involve them. From the perspective of these current dimensions, let us come to the eight reasons that underlie the relevance to revisit POGO fifteen years later.

• 1. The first reason why it is pertinent to return to POGO is above all theoretical and rooted in the very principle of the evolution of human cognition, both cultural and “cumulative”. What distinguishes human culture from other species is its cumulative nature (Dean et al. 2012). This “cumulative” concept reflects our tendency as a species to re-use and to improve the artifacts that have been produced by our ancestors and by our peers. In designing POGO, we focused on successful practices in schools rather than those that presented problems, focusing on what works well rather than stopping at what does not work, as is often the case. This was also the key for integrating the digital with the physical resources (see below).

• 2. POGO, was built to be integrated, and to interact, with the most effective and common tools made available to children in their school environments. The idea was to conceive an environment that does not create a break with existing tools, but rather to make these new tools integrate into the classroom and be used concomitantly with the old ones, while offering new opportunities for interaction. The idea was not to substitute one tool for another but to produce a “seamless” world between existing and new tools, between so-called “physical” tools and so-called “new” digital technologies.

• 3. With POGO, we designed tools that would merge artistic practices such as drawing, painting, and sequential audiovisual art. We tried to exploit the work of Scott McCloud on the basic principles of sequential art by exploring the concept of the ellipse, the “two-box” sequence that creates a progression of time and movement. The camera, one of POGO’s tools, was originally conceived and designed as a tool that allows children to capture a variety of viewpoints, not just the point of view of the person holding it in their hand.

• 4. POGO was built according to the principle of construction/deconstruction/reconstruction: POGO constitute a family of objects (those they can bring, those they can conceive). Children and teachers can build by themselves, assigning functions and meanings to them, playing and “pretending”. In POGO, children had the opportunity to do, undo, and redo: they could try things, observe together constructed things, deconstruct, start over, transform, add pieces, remove them, all with great ease. The concept of “Bricolage” as presented by Levi-Strauss, which we can be translated as “Tinkering” (as developed by the Exploratorium Museum in San Francisco), refers to these relationships.

• 5. POGO was both a performing stage and a creation and editing workshop: a commonplace of creation and design, where each child, by creating and conceiving, brings a small stone to the building which sets itself up collectively. In designing POGO, we anticipated the importance of an environment in which children could speak about a hundred languages (cfr. Loris Malaguzzi), while appreciating the possibilities offered by each media: mixing performance and narration, use layer and level techniques, write, recite by adding voices, annotate an image or text, and so on.

• 6. Designing POGO has contributed to a vision of the learners as intelligent subjects that act to develop and transform the world in which they grow, a vision rooted in the work of Bruner, Piaget, and Vygotsky. In a synthetic way, let us retain here the dimensions of inter-subjectivity and the discursive and dialogic spirit. Piaget argued that role-taking and point-of-view skills, assembled in a cooperative spirit, not only make culture and language possible, but promote reasoning. Bruner (1996) further argues that, “we live most of our life in a world constructed according to the rules and devices of the narrative.” This is why educational initiatives should promote the creation of a “metacognitive sensitivity” to give children an awareness of what the constructions and constraints of narrative impose on the reality they create.

• 7. Designing for the child in the perspective of long-term transformations is an essential issue. Sustainability and affordability are central here. We failed to create a sustainable environment. The possibility of observing the use of POGO in the long term was not possible, which we regretted. Our target was the school but we did not take into account the process of introducing POGO world in the school. An assessment made by McKinsey determined it was impracticable to create a commercial product.

• 8. POGO was designed by a multidisciplinary team of ergonomists, designers, psychologist, software and hardware engineers. The project was an opportunity for extensive production of mock-ups and prototypes. Yet the prototyping tools available at that time to embody in interactive artifacts the concepts and the envisioned scenarios were really inadequate. The hardest challenge was to merge the computational resources dedicated to interact with the physical world (e.g. microcontrollers, sensors and actuators) with the resources for producing complex elaboration and manipulation of the content (e.g. multimedia processing with full fledged computers). This brought some of us to engage in the design of new prototyping tools that would make physical computing simpler to afford and to use in interaction design projects (Rizzo et al. 2016).

• 2003 Chapter

1 Introduction

POGO world is an information technology environment to support the development of narrative competence in children. When we first started our user study for the design of POGO world we were conscious that the development of educational technologies calls for new interaction design approaches to overcome the limitations of current personal computer-based metaphors and paradigms. To help understand the limits and constraints of computer based technology for mediating educational activities and to have sound empirical evidence to share within our multidisciplinary design team, we carried out a longitudinal study in two European schools; one located in Siena, Italy the other in Brussels, Belgium. In this study, we observed and described more then 30 narrative activities, and we discovered that even though in one of the schools advanced digital technologies were available, and although there were several activities that included the use of computers, none of the narrative practices involving such equipment were perceived by the teachers as successful. Moreover, the teachers considered the introduction of computers into the activities that were successful a potentially disruptive factor that would prevent cooperation. The teachers supported their claim with evidence drawn from their own experience (UniSiena and UniLiegi 1999). Thus one of the main challenges we faced in designing a new system for interactive story building was to envision a new form of interaction that encourages creativity and cooperation and that did not jeopardise successful pedagogical activities currently used in the schools.

In the following we describe a general model of successful narrative activities that we developed from the field study in the two schools. We then present the POGO tools in some detail in order to briefly illustrate how the proposed tools embody the concept of situated editing as a metaphor to mediate interaction between children and the POGO world. Finally we present a summary of the tests carried out in the schools with the last version of the POGO world prototype. This provides some empirical evidence to show that POGO world does not jeopardize successful narrative practices, but empowers it.

2 Narrative Activity in Classroom

Through our observation of narrative activities in the classroom, we found that the cycle of creative imagination proposed by Vygotsky (1998) as a psychological process to account for the creation of knowledge occurring in the zone of proximal development, could be used to represent the different chronological and structural phases of a narrative activity. The cycle of creative imagination has four phases namely, exploration, inspiration, production and sharing and describes how the individual experiences the external world, elaborates the impressions received, assembles them in a novel way and shares this production with others. The narrative activities at school included a focus on all of these phases, often in linear sequences, sometimes with small loops or repetition, sometimes with a leap.

Exploration

This consists of the interactions with the real world, which can be either direct or mediated by social relations. All narrative activities that we observed are rooted in the child’s experience. The narratives represent things the child has seen, heard, touched or encountered in the museum, the forest, the seashore or even in the classroom but with the support of objects, instruments and people that create an event. This means that the teacher initially focuses on the sensory experiences of the child, which subsequently constitute the starting point for the theme and for the ideas. At this stage, the child uses instruments appropriate for exploration (e.g. dip net, shovel, microscope, points of view, etc.) and handles various materials (e.g. earth, shells, sand, photos, objects, etc.).

Inspiration

This is a phase of reflection and analysis on the experience had during the exploration. The child is encouraged to think about the previous experience, discuss it and sort out the elements they gathered. The teacher supports the child in the analytic process and in the discussion of choices. Individual writing, drawing or group discussion usually supports this phase. In any case there is a moment when their ideas and thoughts are externalised in more or less lasting way.

Production

This corresponds to the recombination of the elements dissociated and transformed during the previous phase. In other words, production is the moment where children, based on selections and choices of elements made previously, produce new content usually through a great variety of media. During this phase, the teacher’s role is to supervise the organization of narrative content, as well as to ensure conformity with standard rules of story construction. The teacher also makes sure that the text is coherent and sufficiently rich. During this phase, the children mainly use their notebooks, pens and pencils for illustrations, cardboards, puppets, posters and bricolage sets.

Sharing

This is the phase in which children’s externalised productions start to exist in their social world. Children present the result of their production and verify the effects of this production on the others (e.g., children, teachers, parents). We observed that to conclude the activity, the teachers propose a moment of exchange and sharing of the narratives produced. In conventional activities, this is the phase when the teacher concentrates on the presentation of the final product created by the child. The most important document in this phase of the activity is the Story Notebook, which contains the final product: text and drawing. In other cases the sharing phase is a full-scale performance of groups of children or of the whole class.

The pedagogical activities observed in the schools were modelled as Narrative Activity Models (NAM) and were used in order to define the users’ requirements and to relate them to the POGO concept and enabling technologies. A set of requirements were produced both to assess and refine the design concepts proposed by the industrial designer of Domus Academy and Philips Design and to foster further concepts to be expressed in mock-ups. One of the key concepts for the interaction design was the one of situated editing supported by “invisible computing” (Norman 1998), that is, to allow a seamless integration of the physical and virtual world through intuitive interaction modalities.

3 POGO World

From the NAM it was clear that in the school environment children use a number of tools to construct narratives. In referring to the phases that constitute the activity, we ascertained that the exploration phase is usually carried out in groups. Afterwards each child independently carries out the process of creating the narrative. The graphic illustration of the story is also done individually, during or after the verbal description. In other words, in conventional activities story making is mainly an individual undertaking. Nevertheless, we also observed several activities where all the children of the class or of sub-groups worked together to create a single story. The patterns of cooperation vary on a case by case basis, but generally the phases of group activity (choice of story subject and story line, etc.) alternate with individual creation (drawing, inventing dialogs, etc.), where each child makes his/her personal contribution to the construction of the story and the same tools are used to support both individual and group activities in a seamless way.

Most of the personal computer technology available in the school does not fit with this articulated way of carrying out narrative activities; the technology is just out of the loop.

3.1 The POGO Tools

The POGO environment can be thought of as a virtual story world, accessible through a number of interactive physical tools distributed in the environment. The active tools are the main interface to the narrative process. The functionality of the tools spans many areas from gestural (live performances), visual (manipulation of images and drawings) and aural (sounds and atmospheres), to manipulative (physical feedback, kinematics) and material (surface and texture, weight etc.). Although the system is computer-based, the standard computer interface of keyboard, screen and mouse has been replaced with a far more intuitive one. The interaction is very simple so that children can begin to play with no need for instruction.

The system has a number of tools that support the process we call situated editing. Raw non-digital media elements (e.g. drawings, sounds) can be converted into digital assets using tools for rich asset creation. These digital assets are stored on physical media carriers and can be used in tools that support story telling. With these tools assets come alive on a big projection screen, sound system, paper cardboard, paper sticks etc. The system provides tools to capture the creative end-results and share it with others using the internet in movies or in digital or paper based storyboards. POGO has been developed in a modular way allowing parts of the system to be re-used and combined for different purposes. The following tools compose the POGO world:

The Beamer is a threshold tool that connects the real and the virtual environment by allowing the passage of physical things into the virtual story world. The Beamer captures new story elements such as real world objects (including the children) or live video. It has a base unit integrating a horizontal LCD screen with a pressure sensitive touch panel, a video camera, a card reader, and a composition area (Fig. 1). Drawings and objects can be positioned in the composition area, and collages can be created there. The captured image can also be edited in the same composition area. The camera can be used to capture these elements and also as a simple, live video mode where the images are directly projected onto the walls.

Fig. 1

The Beamer

The Cards are media for exchanging story elements such as sounds, pictures and video clips. They are a ‘memory’ for story elements that can be associated to real-world objects by physically attaching the card on drawings, clay models or toys. Cards contain a unique ID tag and are used as physical pointers to virtual story world elements. Whenever the card is activated (e.g. inserted in the silver mat) it displays the corresponding image or sound. When the children pop a story card in the slot on the table and press the record button, the pictures are stored on the card. If a child places a card in one of the pockets on the side of the silver mat, the pictures on the card are displayed as a background. If a child puts the story card into one of the Mumbos, whatever is on the card is shown on the mat in front of the background.

The Mumbos are tools to control foreground elements on the screen. Through the Mumbos, images can be animated (moved) and modified. For example, if the Mumbo is rolled, the image stored in the card contained in the Mumbo moves in the direction of the roll.

The Camera tool allows to record live video which can be stored in cards and displayed together with the other elements and characters. A controller allows the image size on the screen to be adjusted and photos to be taken which can then be inserted in the background (Fig. 2).

Fig. 2

Mumbo and camera tools

The Settings, comprise a silver mat surrounded by leather cushions and various tools. The mat is a screen on which to project images. However, it is also possible to project images anywhere in the physical environment, including onto the children’s body (Fig. 3). The Background Composer is inserted into the setting. This allows up to three cards to be inserted to create a hierarchical background. Dropping cards into the Background Composer activates background images and/or related sounds in mixed media combinations. Background images can be created by the children (e.g., drawings, collages, composition of elements picked from the real world) or they can be selected from a database. The Background Composer provides a continuous output, so even if there is no card in it, a live video image is shown as background. In a sense, the live image allows children to “perform” a story in the real world on a virtual background.

Fig. 3

Silver mat

The Colour Wheel is located in the setting and is used to set the background colour for the screen. It uses four joystick buttons for controlling the colour value, and the effect will be visible directly.

The Sound Twister allows activating sounds by inserting a sound card into the mat (Fig. 4). The Sound Twister Tool is used to playback sounds that are stored on cards. The tool consists of a number of pads. The sounds stored on a card are assigned to these pads and are played if a pad is pressed. Sounds generated in this way are mixed so that multiple sounds can be played at the same time.

Fig. 4

The Sound Twister

The Sound Mumbo is used for sound playback using effects such as pitch shifting and echo. An effect can be selected by tilting the tool in the X–Y plane. The sound source can be real-time using an embedded microphone for input, or can originate from sounds stored on asset cards.

The Voice Tool allows users to insert their voices into the story. A controller allows recording and modifying these voices. Children can speak in strange voices and can add echoes and noises.

The POGO VCR tool (Fig. 5) is used to capture everything that is played in the POGO system. It can record video and audio streams that are generated with the play PC and convert them into a movie file. The Recorder/Reader records and displays story scenes. When a card is inserted in one side of the Recorder/Reader, the scene is recorded in real time. When it is inserted in the opposite side, the recorded scene is displayed.

Fig. 5

POGO VCR

4 The Evaluation of POGO

Rather than performing formative and summative evaluation sessions, we decided to constantly assess the outputs of our design process with teachers, children and colleagues. Thus in a sense, we renounced formal evaluations in favour of a longer term qualitative assessment of the project outcomes. In particular, testing of the final prototype attempted to understand if and how the designed technology had fulfilled the pedagogical goals, how the transition from current pedagogical praxis had been embodied into the praxis of the POGO design solution, and how the POGO environment could mediate the narrative cycle in various activity settings.

In the following, we report the evaluation activity performed at school on the final prototype with children aged from 6 to 8 years. All activities were designed, set up and co-ordinated by the teachers according to their pedagogical objectives. Different kinds of activities were proposed by the teachers ranging from free activities that were selected and coordinated by the children who were responsible for creating the narrative as they wished, to more structured activities proposed and coordinated by the teacher who decided timing, rules, content and dynamics.

All narrative sessions were videotaped and the dialogues transcribed. The resulting narrative productions were analysed and used as basis for debriefing sessions with teachers to analyse and interpret results. The results are described below.

4.1 Impact on the Narrative Activity Phases

Exploration

POGO seems to integrate smoothly with the current practices of collecting story elements: children can bring personal objects, intimate memories, photographs etc. into the POGO environment and evolve them into elements of the narration. The POGO environment supports the transition from everyday life experiences to the fantastic world of narration by affording the collection of different media such physical objects, sound and noises and transforming them into virtual objects thus creating a rich repository of elements useful for the story. In particular the Beamer, which enables the user to import a virtual version of any sort of object, stimulates children to store an experience represented by the object itself.

During the testing, the teachers encouraged the children to explore potential story elements by using the Beamer, projecting images of seashells gathered on the beach and mushrooms picked in the woods and so on. In traditional activities, they could only reproduce them in the form of a drawing, a kind of activity that is sometimes so time consuming as to prohibit a further elaboration of the drawings into story elements. With POGO, it is possible to immediately import an object, to draw and transform it in a virtual element of the story, so that the exploration phase is not conceptually separated from other phases of narrative activity. We consider this a clear added value over traditional ways of supporting the exploration phase.

Inspiration

As happens with traditional activities, during the inspiration phase POGO can be used to encourage children to rethink an experience, to analyse its constituent parts and to express it orally or by drawing. The change with POGO is in the resources available to stimulate thought and decision-making. In comparison with traditional practices, the POGO tools seem to offer greater support to the children. The possibility of combining and recombining elements on the Beamer table, and of displaying the result on the screen in real time, facilitates experimentation and comparison of different solutions. In addition, screen displays have an amplifying effect that facilitates perception and information sharing. The tools support personal reflection, collective comparison and meaning negotiation. This was particularly evident in one activity “Mushroom Development”, where the children spent nearly all the class in the inspiration phase. They placed themselves around the Beamer and the teacher encouraged them to use the material available to reconstruct the developmental phases of the mushroom. The Beamer acted as a support for handling the material, and for producing drawings. The teacher encouraged each child, in turn, to suggest ideas by modifying the material on the Beamer table. The screen enabled them to monitor their own production, as well as the productions of other children, providing a basis for further discussion. Intermediate products were stored in different cards recording individual as well as collective contributions.

Production

Narrative activity in the school is very rich, rewarding and successful. In designing POGO, we learnt from the most successful existing practices and tried to amplify them in the learning process through the use of the POGO technology. The testing confirmed that we achieved this objective, and the production phase was one of the most surprising in terms of creative constructions made by the children. They made new connections among contents just by manipulating the tools. They explored the flexibility of the tools in representing and structuring the narration. Furthermore, the POGO tools allowed teacher and children to take clear roles, from guidance to content direction, technical direction, and performance. In general we assisted an interesting process of role diversification. We observed a division of labour during the creation of scenes between ‘producers of content’, responsible for arranging into acts and scenes, and ‘technicians’, responsible for producing elements of the story like backgrounds, characters, sounds, etc. (Fig. 6). The first group created the story, and the second was focused on realising them. The new role of ‘technician’ enabled shy children who generally do not participate enough in conventional activities, to be more involved, doing something that allowed them to join the negotiation process through their actions and not their words. The distribution of the tools and their location in space helped the diversification of roles: the Beamer was the area to create contents (for technicians), other tools served to memorize (cards) or reproduce (screen) the results (the work of content producers).

Fig. 6

Production with POGO: distribution of roles

During the test, we witnessed a process of scaffolding and fading that was perfectly realised through the tools. When the teacher provided less guidance (fading), the children produced their story with greater independence—as occurred in two activities—“Castle invaded by witches” and “Story of sound”—where the puppet show metaphor was employed. During these activities, the children spontaneously sat behind the Beamer, facing the settings. As soon as the activity began, the children started moving the silhouettes on the glass plate of the Beamer and kept track of the results projected on the settings. Their characters were animated against background scenery, like a Chinese shadow puppet show. The teacher then suggested improvised dialogues, giving voice to the characters, rather than describing the action, as they did in other activities.

The example reveals how role taking with POGO can be extremely varied and imaginative. Children can be actors during the performance, or spectators when looking at the screen.

Sharing

The POGO tools can be used to amplify and enhance collective sharing of the children’s production. Children can share both the creative process and the product of the narrative activity. This meta level of sharing stimulates meta-cognition and meaning construction and negotiation.

This effect is demonstrated by the fact that children insisted on ‘redoing’ the story many times, and presenting it to other children who had not participated in the production. This need for sharing can be explained by the fact that during the creative process the children concentrated mainly on creating a scene, but at the same time they were exposed to the global view of the narration. With POGO they learnt to change point of view and acquire a different perspective on the story, from local events to coherent plots. This effect is difficult to obtain with conventional tools like paper and pencil where all elements of the story are located on one or two pages of a notebook. With POGO the contents are stored in different tools distributed in the space and represented by different media at the same time (Fig. 7).

Fig. 7

Sharing with POGO: distributed activity in the space over different media tools

5 Conclusions

POGO’s challenge was to design innovative technologies for children that should be equally attractive, fun, long lasting and yet offering sound pedagogical learning opportunities to be seamlessly integrated in the current context of European schools. The results so far have been very encouraging. The POGO world does not replace any of the current tools that the teachers successfully use in their teaching practice, instead it empowers these tools and integrates them with new opportunities.

But the most important achievement of our research was the development of an educational tool that supports the entire cycle of creative imagination, letting it evolve as a never-ending creative process.

Indeed the POGO tools allow a rich sensorial interaction where physical and virtual elements of children’s reality can be explored, analysed, decomposed, and recombined in new ways. The existing objects or the new one produced working with the different POGO tools can be captured by children and edited in real time. What a child builds or brings as a part of the personal experience can be combined with the products of other children in a continuous negotiation process where the evolution of transformations of the objects is recorded and the movement along this process of meaning construction can be used as a way to understand the other’s points of view.

Moreover the physical objects that are produced in this iterative and combinatory activity remain live features of the process and can be used as the physical address for the articulated production of future creative activity.

The POGO project advocated a design that was focused on children and teachers’ activities and grounded in thorough research into those activities. Its purpose was to provide a sustainable solution that can help children create and enjoy intellectually interesting activities.

POGO presents a new ‘type’ of system, an open system. It is a kind of ‘personality’, capable of intelligent responses. Depending on how it is used POGO it is a camera, a video recorder, a microphone, a display screen. POGO reacts to the user and adjusts its behaviour accordingly. It is open to change. And, while at the moment it supports children in building stories together, POGO offers many avenues of exploration. It proposes new ways of looking at interaction design, of handling knowledge management systems, of enhancing electronic learning for children of all ages. It points into new directions for collaborative working in the office and collaborative, creative activities in the home.