1 Introduction

In the last two decades, storytelling practices and contexts have undergone huge changes, as pointed out by media theorists and practitioners [1, 12, 17, 23]: moving from its traditional realms of textual, linear delivery, to the digitally mediated, interactive communication that characterize today’s media, traditional storytelling has migrated to new forms of narration and new narratives types. Following the advent of “transmedia storytelling” [12] brought about by new media, social and mobile media have opened the way to “multi-party, co-constructed narration” [17], with new narrative forms where story and reality mix at the junction of gaming and social experience [1].

As advocated by M.L. Ryan [23] at the onset of this revolution, these changes may have not affected the nature of stories into depth, but have innovated the way stories are presented and the way the audience is engaged. In today’s media, the relationship with the audience has become less obvious, blurring the expectations about the audience’s response granted by traditional forms of storytelling. Given this background, it is necessary for today’s authors to refine their ability to handle the relation with the audience, taking into account the audience response since story editing. In this paper, we describe a software platform that establishes a tight relation between editing and interactive delivery of the story. The platform has been developed with the goal in mind to train the creators of narrative contents to write their stories while actively taking into account the response of the audience, and the emotional response in particular.

Traditionally considered an indirect achievement of the gifted storyteller [16], the evocation of an emotional response is less straightforward in digital platforms than in traditional media [9]. Our system, called DoppioGioco (“DoublePlay”), deconstructs the emotional response of the audience, factorizing it at the chunk level in the story editing phase. Offline, the author attaches emotional information to each story chunk. On stage, the system plays a live game with a simulated audience, reacting at each turn to the audience’s response.

The paper is structured as follows: after explaining the motivations for our work (Sect. 2), we describe the functioning of the system in Sect. 3 and its architecture (Sect. 4). In Sect. 5, we illustrate a real case study in using of the system for training story editors, describing the re-design issues and guidelines emerged from this case study. Discussion and conclusion end the paper.

2 Background and Motivations

The inspiration for DoppioGioco draws from two main sources. On the one side, the line of research in studying and designing the experience of the audience [2, 20], situated at the junction of HCI and media studies (see the survey provided by [7]). In particular, [26] reconsiders the notion of “subversive player”, developed in game studies to describe the user’s attempts at breaking the boundaries of the interactivity allowed for by games, with the goal of making this behavior become an intrinsic propellant of the game. On the other side, a relevant source of inspiration is given by the attempts at bringing the paradigm of improvisational theater into interactive storytelling [3, 18, 25]. Typically considered a training tool for dramatic writing, the paradigm of improvisational theater increases the sense of dramatic presence and the engagement of the users, as demonstrated by [25]. The dynamics of improvisational theater has been described by [6] in the perspective of interactive storytelling, using the Decision Cycle from Newell’s Unified Theory of Cognition (receive new inputs, elaborate new knowledge, propose actions to take, select one of those actions, execute the action) as a conceptual framework for analyzing the way each performer takes advantage of the scene advancing moves of the others.

With respect to the approaches described above, our goal is not only to improve the effectiveness of the storytelling system, but to test an innovation in the paradigm of interactive storytelling for training purposes. Combining the tradition of the improvisational theater [2] with the insights from player studies, the “Decision Cycle” in our system achieves an improvisational dimension through the emotional response of the audience. The narrative component is mostly handled offline, at the story design time. The relationship with the audience is handled by the real time component, which is mainly concerned with emotions, those attached to the story contents and those expressed by the audience. Emotions have received much attention in interactive storytelling, both as a component of artificial characters, and as a feature of the story plot. In artificial character applications, emotions have become part of the processes that generate the character’s behavior [5, 10]. As a property of the story, emotions have been accounted for by a few storytelling systems [11, 21, 27], since the seminal work of Mateas and Stern [15], Façade, where the emotional engagement of the user is controlled by a drama manager that keeps it aligned with the notion of dramatic climax.

Our approach is targeted at digital storytelling applications in new and social media, where story is not seen only as an appealing sequence of incidents, but a looser concatenation of events delivered in an interactive way. This is often the case in situated storytelling (e.g. in storytelling applications for cultural heritage and tourism), where the intrinsic coherence of the story gives way to the emotional engagement of the user. This design goal is achieved by moving the focus from the story level design of the audience response to a local, adjustable level of engagement in story delivery. DoppioGioco puts equal emphasis on editing and delivery, transforming the interactive delivery of story in a tight interplay of the audience’s emotional responses and counter-responses by the system.

3 DoppioGioco

Consider the following scenario. W.S. jr. is a young author who wants to master the art of interactive storytelling. He intends to captivate the engagement of the audience, and wants to do so through the use of emotions. After conceiving the story line, he has created a number of story chunks that represent the alternative turning points of the plot and has associated each of them with the emotions he figures out in response by the audience. He runs a simulation of the story in front of the imaginary audience: he plays a story chunk, then observes the reaction of the audience: looking at the imaginary audience, he stares at their faces, trying to spot unexpected responses. Sometimes, he decides to please the audience, making the story advance in a way that reverberates the feelings expressed by the audience. Sometimes, he decides to strike the audience with something opposed to the audience’s emotional reaction. Rehearsal after rehearsal, W.S. jr. navigates the story with different audiences and different attitudes, until he has attained a deep knowledge of the potential of his own creation. He can now edit the story again, and play it from scratch to find the desired balance of story direction and emotional twists and turns.

The software platform provided by DoppioGioco aims at realizing the scenario described above. Basically, the offline system consists of a navigable story chart annotated with emotions, and rules for managing the attitude towards the emotional reaction of the audience. Each story chunk, or unit, corresponds to a media asset that is played when the unit is selected. Online, the system plays the part of the storyteller, whose attitude affects the navigation through the story chart. At each step, the audience reacts to the last played story unit, then the initiative goes to the user: she/he decides whether to empathize with the audience or to antagonize it. The system selects the next unit among the suitable ones, depending on the interplay of the audience’s emotional reaction and the selected attitude.

Fig. 1.
figure 1

DoppioGioco. The interface of the Story Manager.

Full size image

The system has two main functions: the Story Manager is the tool for editing the story units offline and organizing them in a plot; the Stage Manager is the online platform for managing the story advancement in response to the audience’s reactions.

The Story Manager allows the author to create and organize the units offline. Each unit consists of an audiovisual clip and a set of metadata elements describing it, such as title and textual description. For each unit, the author has to provide the information needed to the story engine to create a consistent story at runtime: the precedence relations with the other units, needed to generate a causally motivated story, and the emotions attached to the unit, needed to account for the response of the audience. So, each unit must be tagged with a set of emotions to ensure that the story engine can drive the story advancement based on the combination of the emotional response of the audience and the attitude chosen by the user. Figure 1 shows the interface for the editing of the units. The interface is divided into four areas: the top left area allows managing the unit metadata; the top right area allows uploading and viewing the media asset that realizes the unit (typically, an audiovisual clip); the bottom left area contains the emotion tags: for example, the unit displayed in the figure is tagged with the emotions of “anxiety” and “panic fear”; finally, the bottom right area allows specifying the units that precede and follow the unit. The story chart editing tool (not shown in the figure) allows the author to manipulate the position of the units in the story chart.

Fig. 2.
figure 2

DoppioGioco. The interface of the Stage Manager.

Full size image

When the system is run in the online mode (“on stage”), the Stage Manager enters into play. By selecting an initial unit, the story begins to be told, unit by unit. After each unit, the audience reacts by displaying a set of emotions: like a real audience, the individuals who compose the public may react differently from each other, so the system computes the most frequent emotion. At this point, the author can decide to please or oppose the audience. After setting the reaction type (“against” or “pro”), the system will pick up the next unit among the continuations allowed by the story chart. If several continuation are compatible with the current story unit and the response of the audience, the system selects one of them randomly. Figure 2 shows the interface for managing the stage. The interface is horizontally divided into two areas which represent, respectively, the storyteller (top area) and the audience (bottom area). The top area contains, on the left, the last played unit; the right area is occupied by the console for managing the attitude of the reaction to the audience: the four buttons it contains refer to the global parameters of the emotional control of the continuation, namely intensity (low or high) and polarity (negative or positive, i.e. “pro” or “against”). The lower part of the interface contains a button for getting the emotional reaction of the audience (“Get reaction”, not visible in the figure): when pressed, this button reveals the reaction of the audience and the button becomes invisible until the next unit is selected. For attaining a more realistic impression, the facial expressions of the single members of the audience are displayed (in the figure, the emotion of “joy” has been computed by the system for the audience). The “Go” button posited below the console (in green, see Fig. 2) triggers the selection of the next unit: the new unit is loaded, the console is reset and the faces of the audience are replaced by the button “Get reaction”. The system contains also a tool for managing the audience (adding and removing members, attaching a facial expression to the each member’s emotions, etc.). Notice that the reactions of audience are currently simulated by a random mechanism (see Sect. 4.3).

4 Bringing the Audience into Play

Following a standard practice in interactive storytelling, in DoppioGioco the story is represented through a plot graph (the story chart), with rules for transitioning from a node to the next one in order to generate a complete story.

4.1 Narrative Model

Formally, the story is structured as a directed graph, where the nodes represent the story units. Edges represent the transition from a unit to the next unit. Nodes with no input edge are start nodes; nodes with no exiting edges are end nodes. Starting from a set of initial nodes, a story is a path traveling from a start node to an end node.

In the online mode, when the next unit is chosen, the clip associated to it is played. Units can be marked as initial or end units (not both). Each unit is labeled with a set of emotion tags, as shown in the previous section: they represent the emotions that the author expects the audience to feel when the unit clip is played. At each step, the story engine computes a set of admissible emotions for the next unit given the reaction type selected by the user. By doing so, emotions reduce the space of the possible stories to the stories that are consistent with the emotion model embedded in the system.

Fig. 3.
figure 3

The visualization of the plot. The box in the top left corner shows the details of some units: each unit has an id and a name; for brevity, emotions are replaced by tags: e.g., ir for “irritation”.

Full size image

Figure 3 shows the visualization of the story automatically generated by an apposite functionality of the Story Manager: each box represents a unit, the edges represent the possible transitions from a unit to another (as described in Sect. 5, this functionality was added during the re-design following the testing of the system). The design of the graph is inspired by the following principles, aimed at facilitating its interpretation by the user:

  • The graph is directed horizontally, distributing left to right the units that implement the shortest path from beginning to end. In particular, on the horizontal axis, units are displayed when they are first reached in moving from the initial units.

  • On the vertical dimension, from the middle to the extremes, the nodes maximize the entering edges.

  • Paths that are longer than the shortest paths (without repetitions) are represented with the contribution of edges that go back to a unit already introduced.

4.2 Annotating Emotions

As mentioned in Sect. 2, different computational models of emotions have been proposed in the last two decades, for purposes that range from the annotation of emotions in media to the generation of emotions in synthetic characters. The former systems typically rely on dimensional models, such as Russell’s circumplex model of affect [22] or Plutchik’s wheel of emotions [19], which lend themselves to the general description of the affective content of media, while the latter systems tend to draw inspiration from cognitive models of emotions [24], which are easily integrated in goal directed characters.

DoppioGioco relies on the model described in [4]: originally designed to support the creation of a corpus of clips displaying the perfomance of emotions by human actors (GEMEP), this model is based on an extensive survey of the existing theories and models of emotions, including cognitive and dimensional models. GEMEP, GEneva Multimodal Emotion Portrayals, is a collection of audio and video recordings featuring actors portraying affective states, with different verbal contents and different modes of expression. Thanks to its syncretic and methdologically robust design, geared on performance, this model is especially suitable to annotate the affective content of media. In GEMEP, emotions are grouped based on two axes: the polarity (positive/negative) and the intensity (high/low arousal). The combination of these two axes provides four emotions families, each including 3 emotion types: amusement, anxiety, cold anger (irritation), despair, hot anger (rage), fear (panic), interest, joy (elation), pleasure (sensory), pride, relief, and sadness. Within each family, the emotion categories are mainly characterized by different arousal levels (such as fear and despair).

  • Positive, high intensity: amusement, pride, joy;

  • Positive, low intensity: relief, interest, pleasure;

  • Negative, high intensity: hot anger, panic fear, despair;

  • Negative, low intensity: irritation, anxiety, sadness;

In DoppioGioco, the story units and the reaction of the audience are both annotated with the 12 emotion categories of the GEMEP model. Designed for describing emotions in performance, the GEMEP model accounts for the cognitive emotions, such as fear or anger, that drama studies and narratology consider suitable for the narrative content; more importantly, since the GEMEP model relies on a polarity–based account on emotions, it is suitable to deal with the polarity of the reaction to the audience’s response implied by DoppioGioco: the decision to play against (or pro) the audience can be directly mapped onto the negative/positive dimension of emotions in this model, with “against” corresponding to the “opposite polarity” and “pro” to “same polarity”. The following Reaction rules, applied in a cascading way, determine the continuation of the story:

  • R1: If the selected choice is pleasant, then select the emotion families with the same polarity; else (the storyteller decides to be opponent), select the families with opposite polarity.

  • R2: Tune the intensity level of the reaction to the selected intensity (low or high), given the polarity established by R1.

In order to emphasize the elements of arbitrariness that characterize a live, interactive performance, a random element was introduced in the selection of the emotions within the selected family: given the available units, the system randomly selects the next unit among the available ones, so that the user does not have complete control on the selection.

For example, consider Fig. 2: the current clip corresponds to the unit KUZ001, tagged as “amusement” and the user has chosen to keep a pleasant attitude toward the audience (the active button, “pleasant”, is outlined in red) and to respond with a low intensity emotion. Following the reaction rules described above, if the audience responds with “joy” (as shown in the figure), the system will choose, among the available units, those that belong to the “Positive, Low intensity” family, namely one of the available units tagged with Relief, Interest and Pleasure.

4.3 System Architecture

The architecture of the system encompasses four main modules (see Fig. 4): the Story Manager, the Stage Manager, the Emotion Manager and the Audience Manager. The knowledge about the story and the media assets, created and uploaded offline through the interface displayed in Fig. 1, are embedded, respectively, into the Story Manager and into the Stage Manager (where they are stored in the Media Repository). The system loop orchestrates the interaction of the four modules in the following way:

Fig. 4.
figure 4

The architecture of DoppioGioco. Red lines represent the control flow of the system.

Full size image
  1. 1.

    The Audience Manager generates the reaction of the audience. The Audience Manager contains the description of the audience members and manages generates their emotional reaction when requested by the Stage Manager. In the current implementation, this module randomly extracts an emotion for each audience member and computes the most frequent emotion to generate the audience reaction.

  2. 2.

    The Emotion Manager computes emotions compatible with the selected attitude. The Emotion Manager takes as input the audience reaction and the selected attitude, and applies the Reaction Rules to compute the emotions, which become the input to the Story Manager.

  3. 3.

    The Story Manager selects the next unit. The Story Manager, the properly called the story engine, consults the story graph to select the possible continuations, filtering out the units that are not emotionally consistent with the attitude chosen for the response to the audience.

  4. 4.

    The Stage Manager takes as input the unit selected by the Story Manager and sends it to the player, activating the reaction console in the interface.

Each module of the system is implemented as a web service, so as to allow the portability of the system across different devices and media. The services are written in PHP and rely on a mySql database. The current interfaces have been developed as web pages and rely on the Ajax technology to support a fluid interaction with the user.

5 The System at Work

During the academic years 2015-16 and 2016-17, DoppioGioco provided the online platform for a writing lab targeted at graduate students in Arts and Media at the University of Turin. The goal of the lab was to teach the students to design and produce an interactive story by accounting for the emotions in the relation with the audience. We think that this lab provided a valid test-bed for assessing the potential of the platform for training goals.

The staff consisted of a writing coordinator, a media producer, and a developer. The writing coordinator was in charge of coordinating the conception of the story, the design of the characters and the editing of the story in textual form. The producer coordinated the production of the audiovisual clips that realize the story units. A developer was included in the staff to debug and redesign the software system timely during the conduction of the laboratory. The first edition included 18 students; the second one, 17 students. A contest was held among the participants to decide the story plot, then the class collaboratively designed the story graph. Finally, the students were divided into small groups and were assigned the editing, annotation and production of the story units. The final product was delivered online.

In the first edition of the lab, the story, entitled “Clark”, was a thriller set in a hospital. In this story, the main character of the story is a hospital worker, Clark, who comes across some apparently paranormal events. With the help of a quite and poised young girl, Magdalene, Clark investigates the origin of the events in a climax of horror. But the tale is nothing but a dream made by Clark before his first day in the new job: the alarm clock rings, and he wakes up. In the second edition of the lab, we tested a looser type of narration, with a romance set in an American smalltown in the Sixties. In this story, entitled “Hot bread”, the characters go through changes in their personal lives as a result of their professional and relational crossroads. The story is more inspired to serial formats (such as TV fictions) than to the classical U-shaped stories that characterizes traditional storytelling.

The experience with DoppioGioco was crucial to develop a writing methodology that takes full advantage of the potential of the platform, and to identify the most suitable narrative type for working with the platform. In general, the students appreciated the approach underlying the system and correctly grasped the relation between the annotation of the units with the emotion tags and the generation of the story in the interplay with the audience. However, during the first year the management of the story graph emerged as main difficulty in using the system, so the plot visualization tool for the analysis of the story was added to the system (see Fig. 3). This led, in the second year of the lab, to a more careful design of the story graph. Thanks to the visualization of the story graph, the participants were able to limit the branching factor of the story by pruning some edges. Moreover, during the story editing, a minimality strategy emerged in the transition from the first to the second edition of the lab: in the second edition, for each family of emotions, only a representative was kept in the annotation, so as to limit the number of continuations by avoiding, at the same time, the proliferation of units and of emotion tags. As a result of the changes in the design and genre of the story, the number of units dropped from 156 in the first edition to 105 in the second edition. Moreover, due to the more thorough design of the story, in the second edition the duration of the story was a function of the intensity of the emotions: more intense audience responses (remember from Sect. 4.2 that intensity was one of the two dimensions of emotions) led to shorter stories, i.e., stories composed of lower number of units.

6 Discussion and Conclusion

The tension between the creation of well crafted, consistent story line and the emotional response of the audience was both a challenge and an opportunity for the trainees. In particular, the factorization of the emotions in story editing, and the dualism between the roles of story editor and producer, were suitable to promote a change of paradigm in their design and editing practices.

The writing labs conducted with the system provided some useful insights for the redesign of the system. Putting forth emotions in the design of the story actually went against the standard practices previously learned by the students, mostly based on causal connections: as a result, designing the story in terms of emotions tended to put at risk the logical consistency of the incidents, requiring a careful control on the story graph. To alleviate this problem, a tool for checking their consistency, similarly to [14] may be added to the system.

The solution adopted in the current version of the system to generate the emotions of the audience was effective for training the students to cope with audience responses, but their randomization was sometimes frustrating. In order to reduce the gap between the author’s expectations and the reactions of the audience, the set of emotions may be reduced to a more basic set. Moreover, the management of the audience may be dealt with by a dedicated audience modeling module, as proposed by [8], employing a linked data approach like the one exemplified by [13] to create a more realistic audience simulation from real data.

As future work, we plan to extend the use of the system to other storytelling forms, such as teaching and presentation, and in mixed forms, such as edutainment and infotainment.