The Promise of Multimedia Enhancement in Children’s Digital Storybooks

Abstract

Apart from being vibrant, many children’s digital books offer a rich source for learning as is confirmed by meta-analytic findings. The present chapter pinpoints ways in which children’s books have been affected by digitization and which multimedia enhancements explain the boost these books provide particularly in groups of easily distractible children. There is increasing evidence for a positive impact of multimedia enhancements in digital books on children’s story comprehension, especially of building in camera movements and motion into static pictures to guide children’s visual attention. In contrast, common playful, interactive features reveal negative results although they might enhance children’s engagement. This article discusses new ways in which designers can use interactive features in a meaningful manner.

Illustrations play a major role in children’s story comprehension (Martinez and Harmon 2012). In fact, picture storybooks for children are multimedia presentations: that is, they include not only words but illustrations and text work together in diverse ways to support story comprehension (Mayer 2009). Most digital storybooks are enhanced multimedia presentations: that is, in addition to illustrations, digital storybooks may include a variety of digital storytelling techniques that, just as illustrations in paper books, aim at supporting comprehension of the narrative. Storytelling techniques typical for digital books may include camera movements, details in motion, background music, environmental sounds, and playful interactivity. Do these features enhance engagement in the story and story comprehension along with side effects such as vocabulary acquisition, compared to pictures alone (Revelle et al. this volume; Van Daal et al. this volume)? And do these enhanced digital storybooks change the crucial role of the adult during book reading?

These enhanced digital books became even more interesting when research revealed that they are particularly beneficial for specific groups of children (Plak et al. 2015). An intervention study in Dutch kindergarten classrooms revealed that a large subsample, children with a genetic disposition for distractibility – 30–40% of all kindergarten children – benefited more from digital books than from regular book reading at home and in kindergarten. This large-scale experiment showed that independent digital book reading narrowed the gap in language and literacy skills for distractible children. A replication of this large-scale experiment with similar results (Plak et al. 2016) reinforced our interest in the relevance of digital books and their benefits. As a rational justification we speculated that in a regular classroom environment, distractible children are easily flooded with irrelevant visual and auditory stimuli and often distracted during regular group reading activities. They may become less susceptible to irrelevant environmental stimuli, due to sources of sensory information in digital books. It is even conceivable that the high attentional load of these books may result in a state of hyperfocus: the books put a load on these children’s visual and auditory perception, which might help them focus resulting in better performance as compared to their peers who are not at elevated risk for attention problems.

1 How Multimedia Support Learning

Around 1900, there started a discussion among educators in the Netherlands about illustrated, multimedia books for young children. Van Kol (1903), the initiator of the discussion, doubted that illustrations are useful additions. She expressed a strong preference for books with pictures or books with a narration but not both combined in the same book. When books include illustrations, she speculated, there is no incentive to create visualizations and this may have a negative impact on the development of young children’s imagination. In addition, when a book is generously illustrated the narration may be marginalized and attract less attention than the illustrations. Half a century later these naïve notions were experimentally tested. The Canadian psychologist, Alan Paivio, developed a theory about the role of visualizations in understanding language (1986). His experiments revealed that the human brain is able to process visual and verbal information simultaneously and that information from both sources can be synchronized in working memory and support each other. Visualizations can thus help understand a narration especially when the language is complex due to unknown words and grammar.

With the help of children’s eye fixations on illustrations while they listen to a narration, we tested whether children indeed synchronize pictures with the narration as Paivio’s theory of dual-coding predicts, thus promoting that visual and verbal information are integrated. In a recent experiment (Takacs and Bus 2018) we therefore observed whether children fixate the parts of the pictures that are highlighted in the oral narration and whether they do so in the order in which those elements appear in the narration. All participants were exposed to the same pictures but under different conditions: with a narration that fitted the picture (congruent condition), without any narration (picture only) or with a narration that did not fit the picture and, accordingly, elements of the illustration were not highlighted in the narration (incongruent condition). In line with previous results (Evans and Saint-Aubin 2005; Verhallen and Bus 2011), children paid most visual attention to details in pictures when those were simultaneously highlighted in the story text. Target elements were fixated much longer in the congruent condition than in the incongruent or picture only conditions. See duration of fixations on details that are highlighted in the narration in Fig. 1.

Fig. 1

Longer eye fixations at details in the pictures if those are highlighted in the oral narration. (Based on Takacs and Bus 2018)

These findings thus support that children use visual information sources and if the corresponding pictures concretize the narration, and both, narration and picture, are simultaneously available in working memory, the two can be integrated as a result of dual-channel information processing. Apparently, this is how children learn: they try to dispose of pictorial information in addition to the oral narration and create a situation that enables the synthesis of both sources of information. The order in which children fixate the details in the illustration highlighted in the narration was similar to the order in which they were mentioned in the narration, thus confirming that the integration of verbal and visual information happens on a moment-to-moment basis (Tanenhaus 2007).

2 How Digital Storytelling May Help Synchronize Picture and Narration

Digital storytelling techniques may help young children to synchronize picture and narration and support dual coding thus stimulating children’s story comprehension. In Fig. 2, it is demonstrated how camera movements including zooming and panning – digital storytelling techniques – can guide children’s visual attention through the illustration and thus help to synchronize narration and illustration in a digital storybook, Lightning by Coenraads and de Wijs (2017). The first screen shot presents an overview of the scene; we see a dad running after his son who is very eager to go to the school visible in the background. When the narration says, “Today I will learn how to make lightning”, the camera zooms in on the boy as is shown in the next screen shot in Fig. 2. In the last screen shot when the text mentions that he is going to “the weather school”, the camera pans to a depiction of the school.

Fig. 2

Three screen shots demonstrating how pan and zoom work as a digital storytelling technique. (Permission granted by Het Woeste Woud, the Netherlands)

Likewise, putting details of illustrations into motion is an effective way to attract a longer and steadier focus and thus facilitate dual coding of verbal and non-verbal information (Takacs and Bus 2016). In many digital books, motion is added with a purely ‘decorative’ or incidental goal. Motion might make children’s television programs and cartoons more realistic but it is not added with the intention to guide children’s attention to details that are highlighted in the narration as in digital storybooks and may therefore not benefit narrative comprehension. For instance, in the picture in Fig. 2 many details could be set in motion, which would contribute to a film-like effect: the safety glasses, the screwdriver, the pencil and the tapeline. However, attracting children’s attention to those elements would not support the integration of narration and pictures. In fact drawing attention to details that are not relevant for the story is likely to distract attention from the main message.

The guidance that young children may receive by adding camera movements and motion to digital books mindfully may be comparable with guidance offered by adults while they read books to their young children (Bus et al. 2015). It is a myth that adults typically initiate dialogic reading including story-related utterances and distancing prompts. For instance, parents for the most part are not capitalizing on shared book reading as a context in which they explicitly coach vocabulary development; see a study by Evans et al. (2011). On the whole, adults and children mainly point and comment at details in pictures while they listen to the narration (Sorsby and Martlew 1991). Their utterances most often involve mapping language onto immediate perception, similar to what camera movements and motion do, thus attracting visual attention to details in the pictures and stimulating a synthesis of narration and pictures.

3 Efficacy of Enhanced Multimedia

About 10 years ago we started experimenting with digital books that included an enhanced multimedia presentation to test the efficacy of these new additions (Verhallen et al. 2006). Winnie the Witch by Thomas and Gorky (1996), a story with memorable characters and an impressive storyline, includes all components of a well-formed story: a description of characters, setting, time and activity; events that advance the storyline, including the problem; a resolution; and an ending. In addition to a version with static illustrations, we used an enhanced digital version. This version was supplemented with camera movements and motion that may help focus attention on significant visual details. The additions drew attention selectively to congruent content in the illustrations, thereby helping the child to select helpful visualisations while processing the narration. For the rest the two versions were identical: the narration is told in the same voice and both are presented on a computer screen.

Five-year-old children from immigrant families with low language proficiency, due to the fact that they learn Dutch as a second language, profited from repeated encounters with the storybook with static pictures but much more from repeated encounters with the animated version of the story. Multimedia additions in the digital book were most pronounced for understanding the goals, intentions, motivations, and feelings of story characters. For instance, children in the static condition understood that Winnie the Witch kept stumbling over the cat (action) and that she changed the cat into a green cat (action). Retellings of the story after being exposed to the digital book enhanced with animation and sound/music, on the contrary, did not just contain actions but implied elements as well. Children in the digital condition mentioned the fact that Winnie the Witch got angry when she had fallen once again and decided to do something about it (implied event) and they emphasized states of minds of main characters (“sees,” “is furious,” or “decides”). After listening four times to an enhanced digital presentation children’s scores regarding implied events were significantly higher than in all other conditions including the version with static pictures after four readings; see Fig. 3.

Fig. 3

Percentage of implied events mentioned in the story retelling after one or four readings of digital books with static pictures or enhanced with animation and sound/music. (Based on Verhallen et al. 2006)

In addition to animated pictures, digital books often include music and environmental sounds. Winnie the Witch, for instance, presents immersive background music non-stop throughout the book. There are also environmental sounds that match the events of the story: we hear the noise of Winnie stumbling down the stairs; we hear the cat snoring when she climbs down the tree. Is such a mix of environmental sounds and music a beneficial component of the enhanced multimedia presentation in digital storybooks? Whether music and sound are beneficial for or hindering learning may depend on qualities of the auditory stimuli. For instance, music during homework may block off the rest of the world and may therefore be helpful especially when children are easily distractible (Anderson et al. 2000). Likewise, elevator music while listening to a story might work as a barrier against distractions (Ben-Shabat 2017). On the other hand, negative findings may be expected especially for low-profile babble noise like a playground or a bear mumbling continuously. The listener may zoom in on particular components, which may make those distracting and cause a negative effect on story comprehension.

4 Effects of Music and Sound

There are several experiments running testing whether music and background sounds further enrich animated pictures or interfere with learning from digital books. The first results come from an experiment carried out in Turkey targeting typically developing children (41 boys and 58 girls) aged 4–6 years of age (M = 61.32 months, SD = 9.67) from middle socio-economic status families visiting two public kindergartens (Sarı et al. 2019). The design of the study was a randomized control trial with a control group (children only participated in general language assessment and post-test sessions) and four experimental conditions in order to test the effects of both animated illustrations and sound tracks separately.

The animated versions of the target books – including well-chosen camera movements and motion – were especially effective for gaining knowledge of implied elements of the stories that referred to goals or motives of main characters; note the higher scores of animated digital books in Fig. 4. Animated pictures had a moderately strong effect size of half a standard deviation (d = 0.49). Music and sounds, on the other hand, did not have an effect on knowledge of implied elements. The effect of music and sounds (d = −0.14) was negative but statistically insignificant. In line with a prior study targeting children with serious language impairments (Smeets et al. 2014), music and sound effects had a moderately strong negative effect on children’s receptive vocabulary (d = 0.56). Even though music and sounds were present only in the background and provided to supplement images, they may have attracted attention (Barr et al. 2010) and might have caused cognitive overload for these children because they had to process verbal and other auditory information simultaneously. Whatever the exact nature of the interference of music and sounds may be, an important message of these findings is that adding sounds and music to stories might diminish rather than enhance the learning potential of enhanced digital storybooks especially for children who have problems with verbal processing. In the research so far, target books included a mix of music and sounds. Further experiments are necessary to specify which kind of auditory stimuli in particular may cause negative effects.

Fig. 4

Mean number of implied elements (and 95% confidence intervals) mentioned in the story retelling in the five different conditions. (Based on Sarı et al. 2019)

5 Efficacy of Playful Enhancements

Since the first generation, digital storybooks include playful elements that are incentives for interaction with the illustration (Korat and Shamir 2004). Exemplary are the digital storybooks based on the stories by German author-illustrator Janosch (1998a, b, 1999). These were among the first digital books that came out in the Netherlands in the late ‘90s and are exemplary for many other digital books. A scene from one of the stories shows Dr. Cornelis Frog examining Tiger because he did not feel well. After the events were dramatized, it was possible to play: children could click within the frozen screen on about five details in each illustration such as the light bulb, the little duck on the floor, or Tiger, whereupon visual and/or sound effects were activated, often not related to the story content. Similar interactivity is still used in commercially available digital books. This aligns Apple’s policy: Apple’s App Review Board commonly rejects book apps that have limited interactivity and a lack of media features: if your App doesn’t provide some sort of lasting entertainment value, it may not be accepted (Apple 2017).

The question, however, is whether there is support for the designers’ underlying conception that those techniques increase children’s motivation to listen to stories and interactivity thereby promotes story comprehension (Tønnessen and Hoel this volume). Quite a few studies show positive effects of these features on child behavior during book reading: Children are more attentive to and engaged in digital books as compared to print books (e.g. Richter and Courage 2017). From the Richter and Courage study, however, it does not appear that there is any difference in story comprehension despite that children are more attentive in the condition with digital books as compared to print books. Unfortunately, the children in this study were read to from the target books only once which is known to be insufficient to demonstrate (differential) effects of book reading with young children (Verhallen et al. 2006).

In an early experiment, de Jong and Bus (2004) tested effects of playful elements by contrasting a book with such additions embedded in illustrations with a story without playful additions and a control condition (no readings between pre- and post-testing). In a publication about this study, the authors presented a non-significant effect on a variable based on how many words and phrases in children’s retellings of the story were derived from the original text. Conclusions differ when we focus on story comprehension – the number of pages where the story that the child told was similar to the original story (story structure). Targeting this comprehension measure, children’s score reduced with more than 25% when the story was enhanced with playful elements; compare in Fig. 5 the results in the condition with playful elements (blue pile) with the results without playful elements (red pile). There were many interruptions of the story in the digital book condition; on average children activated 35 playful elements (SD = 26) per reading.

Fig. 5

The average percentage of pages where the story told during the retelling was similar to the original story, for three conditions: digital book enhanced with playful elements (blue pile), print book without playful elements (red pile), and control condition (green pile). (Based on data by de Jong and Bus 2004)

In the same vein, Parish-Morris et al. (2013) and others showed that important aspects of dialogic reading like story-related utterances and distancing prompts were diminished when parents and children read digital books with playful elements as in the Janosch stories, while less productive behavior-related interaction (“Stop pressing the buttons and listen to the story”) increased. An implication of this is that adult guidance cannot compensate for distractors in interactive digital books but might strengthen the negative effects.

Although these playful enhancements are designed to be interactive, motivating, and self-paced (e.g., Ricci and Beal 2002), adding ‘bells-and-whistles’ to a multimedia presentation seems to distract children from the story content and diminish comprehension (Mayer 2001). We wonder whether additions that include story-related content (e.g. a dictionary, questions about the story content) may have the same negative effect as playful enhancements. The human information processing system has a limited capacity; sharing resources among various tasks (e.g., memorizing and integrating story events in between playing) may come at a cost for performance (Kahneman 1973). Preschool children’s learning may suffer from task switching between game-like and other interactive features and story understanding. According to Kahneman’s (1973) capacity theory a person’s ability to process several information sources simultaneously depends on how much “capacity” separate sources require. When demands exceed capabilities, part of the material will not be attended to and may result in distortions of the narrative content or less detailed retellings.

6 Towards Literary Interactivity

There are currently experiments running with digital books that stimulate new ways of interactivity. Children can carry out small actions that are in line with the narration, thus providing users opportunities to engage more deeply with the thematic content while maintaining “the integrity of the story” (Yokota and Teale 2014, p. 581). Instead of features that cause young readers to lose sight of the main storyline these apps successfully integrate playful interactivity with the narrative line. A well-known example is the There’s a Monster at the End of this Book story (Callaway Digital Arts 2010). In this book, playful interactivity and the narration are intertwined in a very natural way. The interactive features tie directly into Grover’s attempts to keep the child reader from turning the page because of the monster at the end of the book. Grover ties knots, nails up boards, builds a brick wall; the child is able to break through each of these by touching hot spots and thus moving the story forward.

Sargeant (2015) developed an app, How Far is Up?, that allows users to move a toy rocket around a scene by tilting the hardware device or by dragging a finger across the screen. This simple play activity may provide users with the experience of moving an object around the space as happens in the story. “This activity”, argues Sargeant (2015), “was designed to provide thematically linked, experiential play, allowing users to maintain a connection with narrative content whilst they engage in this simple activity” (Sargeant 2015, p. 461).

The Dutch app developer Christiaan Coenraads experiments with forms of interactivity that promotes children’s reflection on story events and deepen story comprehension. He makes users carry out the same actions as the story characters that aim at solving core problems. For instance, when the main character is looking for the correct button on the dashboard of a machine we first see him sitting, puzzled, in front of the dashboard. Then the camera zooms in on the dashboard and one of the buttons lights up while the boy says: “Mmmm, which is the correct button? Maybe this one?” The story only continues after the user has clicked on or touched this button, even though it is evident that it is the wrong button.

The user thus becomes complicit in the actions of the main character, which may stimulate reflection on the action and the possible consequences that it may have. In order to test the efficacy of this approach, the researcher (Bus) observed four- and five-year old children while exploring this new app. She noticed that many children are hesitant to touch the button and make spontaneous comments (“this is the wrong button”; “what will happen?”) suggesting that the action promotes reflection on the story content in children. She also observed that children are very eager to find out what will happen in the next scene after they pushed the (wrong) button. Other junctures in the story reveal similar observations. When the boy “borrows” his father’s weather machine he has to crack the lock attached to the machine. The user has to click on or touch a hot spot, otherwise the story does not continue, thus creating a moment of reflection. Many children make spontaneous comments demonstrating that this action makes them think about the events. They start hypothesizing about what will happen (“the dad may wake up”) and whether the action is allowed or not (“his dad will be angry”). A randomized control trial comparing an animated book without interaction with a book with such interactive moments is carried out currently to test whether this kind of built-in interactivity deepens children’s story understanding as may be expected.

In addition, new experiments are needed to test whether this interactivity may facilitate adult-child interaction while they share the story. The built-in stops may be incentives for story-related utterances for both adult and child, thus stimulating adult-child interactivity about the story content in a very natural way. The stops take place at crucial moments in the storyline and are therefore very suggestive towards themes for discussion. So far, the researcher noticed that children often start talking to the adult sitting next to them when the story discontinues at these junctures and look for confirmation of their responses and feelings about the story events (disbelief, uncertainty, involvement). In line with this, Kim and Anderson (2008) showed that giving the parents and the children control over the pace of the story resulted in more interaction compared with a closed format in which pages were turned automatically.

7 Conclusion

The ability to process a narration is central in becoming literate and it is therefore vital that children have a chance to interact with books. For this purpose, it is important that books are read to young children from an early age thereby creating sufficient opportunity for developing foundational literacy skills. We discussed different features built in digital books that may support but also ones that might hinder story comprehension. We made a small selection of well-designed studies to test how children benefit from mindfully enhanced multimedia books.

Well-designed digital picture storybooks integrate illustration and narration so that each complements the other and, together, they provide an enhanced multimedia text experience that may help story comprehension. There is convincing evidence for the benefits of well-designed visual enhancements like zoom and pan and motion added to illustrations in digital storybooks. Concerning background sounds/music as is often built in the books, the findings so far are not very promising. The effect of the mix of sound effects and background music tends to be negative with some exceptions. More experiments are needed to distinguish between interfering and supportive auditory information.

A new, promising approach is adopted by adding interactive features to engage children in the story and to facilitate adult-child interaction. We cannot deny that interactive books have high appeal to young children according to children’s attention and engagement while reading these books. Children may be distracted from the story content by the touch-activated interactive features that digital books typically provide. However, some interactivity may support learning. We sketched the contours of apps that include promising interactive literary digital storytelling techniques. We described examples of interactions with the story that seem to have the potential to engage children in the story, facilitate adult-child interaction and thereby deepen children’s story understanding. Many questions remain. It is for instance not yet clarified whether these techniques demand special stories or can be added to any digitized print book. Further research is needed to reveal which criteria interactive elements should meet to be effective.