Keywords

1 Introduction

The fruition of interactive cinema provides new opportunities for telling stories within large entertainment industries. Interactive cinema offers both new venues for artists to explore and new types of experiences for the audience. Because the opportunities appear so vast, understanding how to design the experiences that maintain cinema’s appeal is essential.

The term lean-back serves as a metaphor that describes traditional media experiences such as cinema [16, 26] – We lean back, and behold the spectacle unfolding unlike lean-forward mediums such as games [11]. For the narrative absorption, it has been argued that active choice becomes a disruption [5, 27] - including even just the mediation of choice through an interface [25].

Passive Brain-Computer Interfaces [33] (BCIs) have therefore become of interest in recent projects of brain-controlled film [23, 27] as a mean to tap into the mind of the viewer, and to apply passive control for changes in a film for a more personalized experience. Previous studies have sought to artistically augment and personalize the film experience, such as using BCI measurements of attention and eyelid blinks to control sound, shots, and timelines [22]. It appears however that conscious control, though passive, can still disrupt the narrative immersion [22, 26]. Proposed lean-back interactions seek to increase narrative immersion without posing demands to the user, or them necessarily being conscious of the control [26].

This project seeks to leverage lean-back interactions in a practical manner for the viewer. Cinema as a product for the masses arguably suffers, as smaller tangents of a plot or embellishments of its story world are discarded to provide a lean product with mass appeal—sometimes seen with director’s cut opposed to the studio’s cinematic version. Here, interactive cinema can earn a functional quality as a way to indulge both the creator’s and viewers’ interests at a more individual and subjective level.

Therefore, this project investigates how individual interests can be catered to with the use of a framework for skipping past segments of a film found boring by the viewer, while maintaining a coherent narrative with authorial control. Our own short film, which was an adaptation of scenes from the movie Ex Machina (2014)Footnote 1, included 4 zones with content that could be skipped based on the viewer’s attention. With a between-subject research design, we aimed to find a difference in narrative engagement. Though no significant difference was found between the two versions of the experiences in self-reported narrative engagement, there is still knowledge to inform future endeavors with brain-controlled film.

This paper will first review related work, then present the background for brain-controlled cinema and areas researched for the presented framework. Then we will describe the experiment and the evaluation methods, followed by a discussion of the main findings of the study, and a conclusion.

2 Related Work

Interactive cinema has been experimented with for many decades [27], though the first to apply passive control by measuring physiological responses was Tikka’s installation ’The Obsession’ [31]. Since then, the term of ’neurocinematics’ has been coined by Hasson et al. [13] who investigated films’ effect on our brains, and now, in turn, how this can affect the film itself, with the field of brain-controlled films. This section will look into some examples of these.

In a study by Pike et al. [22] with the film called The Disadvantages of Time Travel [2], the occurrence of viewers’ eyelid blinks, as well as attention and meditation levels, are used to control the view of the narrative as well as blending between video layers. In continuation of this research, Ramchurn et al. [26] conducted a study of the brain-controlled film The MOMENT [3]. The film blends narrative threads and soundtracks using an attention-based algorithm with data from the previous scene to determine the combination. The algorithm was designed to maintain continuity while also allowing for variation and subconscious control. A drop in attention triggers a cut between threads, the thread continues if attention is maintained or increased. Pike et al. reflected on their work with brain-controlled films [24], and proposed themes for research questions yet to be answered, among them, how individuals’ subjectivity can be catered to and whether viewers should be conscious of control.

Kierkels and Pun [17] investigated if the viewer’s interest in movie scenes could be discerned using various physiological measures, not including EEG. They found no correlations between interest and movie genre, arousal and valence, but that changes in interest level across a scene and between viewers were detectable. Kirke et al. [19] used physiological measurements, among them EEG, evaluated at specific points in order to determine which ending of the film a viewer would find most exciting. They found that their system was possibly able to succeed in this task, but that it was inconclusive to what degree the system did so by appropriate selection of measurements.

This study attempts to explore how consumer-grade BCIs can be leveraged further in designing experiences with practical benefits for the audience and does not require the creators to completely rethink their art in the creation of interactive films.

3 Background

As this study aims at exploring the research area of catering to individuals’ interests, this section will firstly look into some of the arguments and assumptions of these previous projects before presenting the theoretical basis for this project’s suggested solution.

With the term ’interactive cinema’, it is naturally implied that some interaction has to take place. Over the years, there have been many different takes on interactive cinema, ranging from more traditional active decision making, as for example in Netflix’s title: Black Mirror: Bandersnatch (2018) [1], and for more passive or sometimes even non-conscious interactions bending conventional understanding of interaction.

The ambiguous space of control has been considered in previous projects [22, 27], and has been further reflected upon with Benford et al.’s conceptual framework of contested control [6], where control is considered along the dimensions of surrender of control (voluntary/involuntary), self-awareness of control, and looseness of control (how predictable the relationship between measurements and system output is).

The previous studies on brain-controlled films featured control that shifted across these dimensions as users consciously tried exerting control, or forgot they had it. While notions as subconscious control might appear contradictory, it relates to the state of Csíkszentmihályi’s flow [10], the mental state of full immersion in ones activity, as has been previously argued [6, 22]. Flow relies on a balance between boredom and anxiety; in context of interaction, this requires the controls to take background and allow the activity itself, or narrative, to take foreground for it to stay engaging [25]. This has been acknowledged to be the challenging predisposition of lean-back interactions [26]. Attention offers an interesting control as it too shifts, and is an inherent part of the viewer’s engagement with the narrative. If interactive systems are able to rightfully adjust to viewers’ attention, they could offer less friction in the interaction.

Attention is a profusely complex phenomena to study, as Siler puts it “studying attention is like peering into an ocean of data that covers and connects the entire human sensorium, and interpreting the flow of data, currents and all the other changing details” [29] (p. 191). The ‘attention’ measurement of consumer-grade BCIs, such as products by NeuroskyFootnote 2, produces this metric using proprietary non-open source algorithms, therefore introducing some validity concerns. This means that it is not clear how ‘attention’ is derived, though studies have found that it does correlate with the cognitive process of attention [7, 18, 20]. Films however offer a very controlled environment for studying attention. Smith investigated how attention is guided across cuts with his Attentional Theory of Cinematic Continuity [30]. In this study the bottom-up and top-down processes that drive our attention are described. Bottom-up processes are involuntary, as our eyes follow movement, look at the bright parts of the image, or gravitate towards faces. Top-down driven processes are driven by our cognition as we formulate and ascribe meaning and connections to our observations. This type of processing is more cognitively effortful, as they require the viewer to actively connect the dots.

Vorderer et al.’s model of the media experience [32] presents the viewer’s interests as one prerequisite for enjoyment. This intangible subjective interest of the user as a prerequisite to engage just for the sake of doing so. For this project, it is the top-down processes between viewers we would like to differentiate, to investigate if viewers’ interests and engagement with content can be derived from the measurement of ‘attention’. The following sections investigate what motivates our attention to seek more, and how this could be considered structurally to both engage and reengage the audience.

3.1 Curiosity and Factors that Drive It

Gottlieb et al. [12] investigated the factors that drive curiosity from a neural and computational perspective, not only in humans, but as underlying mechanism of the brain that motivates exploratory behavior. Curiosity is the desire to obtain more information, an intrinsic motivation, as the goal is to know more, simply for the sake of knowing it. Unlike the rewards of extrinsic motivation, e.g., money or food, the rewards of intrinsic motivation are more challenging to quantify. They investigate the mechanism of this complex and subjective experience and provide the factors of surprise, novelty, reward, and uncertainty that describe its structure.

Surprise is the subversion of expectations, and is therefore context-specific on the situation and beliefs of the observer. Novelty on the other hand, is dependent on the amount of the observer’s prior exposure to the observation. Gottlieb et al. cite a model of novelty from a computational point of view as the dissimilarity between a stimulus and the representation of similar stimuli in the observer’s brain [4]. Novelty and surprise elicit attention from the observer without necessarily being part of a goal-directed task, i.e., as part of obtaining knowledge to satiate curiosity. The factors of reward and uncertainty are more top-down driven, as the viewer selectively directs their attention to items of perceived importance; items that either close information gaps, or items that simply arouse a feeling of pleasure due to prior experiences.

3.2 Structural-Affect Theory

Brewer and Lichtenstein’s Structural-Affect Theory describes how entertainment stories organize and present events [8]. Suspense event structures present critical information which outcome is first revealed later on. Due to the uncertain consequences of the critical event, this structure builds suspense to the point when the outcome is revealed. A surprise event structure also contains critical event information early in the discourse, yet this information is not shown, and the viewer is unaware that the information is being omitted. Surprise will be evoked when the omitted information is revealed. Similar to the surprise event structure, in a curiosity event structure an initiating critical event is omitted from the viewer, but here the viewer knows that the information is missing, making the viewer become curious about the withheld information. The curiosity is later resolved by gradually providing enough of the missing information for the viewer to reconstruct the omitted event.

The inclusion and awareness of these structural elements of surprise, suspense, and curiosity will be useful for this project to build a narrative that keeps the audience engaged in the story.

4 Design of the Short Film

This section will first provide overview of the proposed framework, then how the short film was produced, and how the framework was implemented for the short film.

We propose using drops in the viewer’s attention level, as indication that the current segment of the film is of no particular interest for the viewer and should therefore be ‘skippable’. Skippable is to be understood in the sense that the viewer unknowingly skips ahead in the story in a seamless manner, past the segment of little interest – while maintaining story continuity. If a specific segment of a film particularly arouses the interest of a viewer, we assume that more attention will be allocated to it, as they engage with the content on a deeper, and hopefully, measurable level. To provide structure to the segments, the notion of an attention zone is proposed as illustrated in Fig. 1.

Fig. 1.
figure 1

The conceptual framework for Attention Zones and its segments.

Full size image

A zone contains at least two segments, as first, the viewer needs to be introduced to the content before it can be determined whether there is an interest in the topic. The first segment serves as a hook, to assess if the succeeding segments are of interest. Therefore, only succeeding segments are skippable. Attention is compared to a pre-zone baseline, with no major plot points that could spike attention. If there is more than two segments, then baseline for the third segment is the second segment. If a particular segment is not of interest to the viewer and the segment is thereby skipped, we also need to consider what comes after the segment. After all, if the viewer was disinterested in a particular aspect or part of the story, continuing with something boring would risk losing the interest altogether. So, we want to re-engage them with a new beat of the story - here, elements of suspense, surprise and curiosity should be considered as a way to amend lowered interest.

To apply this framework, an interactive film needed to be produced. Due to resources and scope of the project, it was decided to base our short film production on an existing movie. The movie Ex Machina (See footnote 1) is about the programmer Caleb who performs a Turing test on tech mogul Nathan’s newest creation, the humanoid AI Ava. It was chosen as it covers different topics that could be elaborated upon or condensed and featured several plot points that could be used in a short film of its own right.

The short film created that applies this framework contains four zones with content that can be skipped if the viewer’s attention drops. The content of the zones was considered for how they elaborate on topics of the film, topics that might be of different interest to viewers. Zone 1, as illustrated in Fig. 2, features entirely new content in the film, while the remaining zones and film are adapted from several scenes of the original Ex Machina script.

Fig. 2.
figure 2

Overview of zone 1 of the film featuring three segments.

Full size image

The themes of the remaining 3 zones concern respectively; Caleb’s work and his personal life, the philosophical Knowledge Argument, and Ava testing Caleb. The exciting plot developments for each of these zones are respectively; Ava asking if Caleb likes Nathan, Caleb asking Ava if she knows he is brought there to test her, and Ava asking what will happen if she fails his test.

The short film used in the experimental setup can be seen hereFootnote 3, and lasts 13 min, 3 min of which are skippable. The link features annotations for when the different zones and segments starts. Our short film production was made using virtual production techniques, more specifically with the use of live LED wall in-camera virtual production. This is a state of the art technique where output from the real-time game engine Unity to a live LED video wall. to a live LED wall is used together with real-time DMX lighting and camera tracking to create final-pixel imagery in camera, and has been employed in the recent and popular series The MandalorianFootnote 4 [14, 15].

5 Methods

To evaluate if the presented framework can be used to increase narrative engagement, a lean-back interactive short film was produced as described in the previous section.

With a between-subject experimental design, two independent groups were tested; The experimental group viewed an interactive version, and the control group viewed a non-interactive version. Sampling of participants was done by convenience sampling. To ensure equal distribution of skips in both groups, participants in the non-interactive group watched a previous participant’s version from the experimental group. The setup featured a projector to give a cinema-like experience. Both groups wore a NeuroSky MindWave Mobile 2 EEG headsetFootnote 5, and were not told about the experiment’s purpose and controls. The measurements of ‘attention’ was used to determine whether a skip should occur. If attention dropped more than 10% compared to a baseline, a skip occurred. This was evaluated few frames before a segment ended. For the first segment, the baseline was the viewers average attention leading up to the beginning of the zone. These pre-zone baselines were of different lengths, as to ensure that no peak in attention could be caused by any larger plot points. For remaining segments, the baseline was the average value from the preceding segment.

While a participant was watching the film, an observer took note of their behavior and reactions. Following the film, feedback from participants’ self-reported narrative engagement was gathered. First, according to Busselle and Bilandzic’s Likert scale for measurement of narrative engagement [9], which has sub-scales that could highlight different aspects of the solution. Then, according to Schoenau-Fog et al.’s Continuation Desire assessment method [28] participants responded to the single statement: “When the film ended, I wanted the film to continue.” Along with basic demographic data, participants were also asked whether they had seen the original movie or not. After answering the questionnaire, a short semi-structured interview was conducted with questions related to the film’s tempo, and whether they felt bored or ‘zoned out’. The participants were also asked for each of the zones how interesting or exciting a zone’s theme was to them, and were asked to give each zone a rating on a 10-point scale.

The ratings were used to assess the performance of the skipping algorithm which was evaluated. A confusion matrix was developed to assess the correlation between the participants’ non-conscious control of the film and their interests in the topics of the four zones. The confusion matrix was developed by classifying a zone with low interest (below 5), and thereby an expected skip, as positive, and the opposite as negative. Likewise, a determined skip by the skipping algorithm was classified as positive, where no skip was classified as negative. These classifications were done for all participants. For the control group, however, calculations were made for each participant’s attention data, to find out where skips would have occurred if they had seen the interactive version of the film. Various derivations were then made from the confusion matrix to assess the performance of the algorithm.

The algorithm was assessed according to gender differences, and whether participants had seen the original movie. Having seen the original movie could prove a large confounding variable, as viewers could compare the versions, removing elements of surprise and novelty. However, due to low availability of participants, it was chosen not to be a disqualifying factor. Gender differences was investigated as differences in attention between male and female has been found in a previous study [21].

6 Results

The experiment was conducted on 24 students, 9 females and 15 males in age range of 20–27 years old. Two participants had only seen some of the original movie, 11 had seen it, and 11 had not. In the control group six participants had not seen the original movie, two had seen some if it. In the experimental group, five had not seen it. The following sections will first present findings between the experimental and control conditions, then qualitative data gathered from interviews and observations, and finally results of assessment of the skipping algorithm.

6.1 Experiment

A normal distribution was observed in the data with Shapiro-Wilk test (p > .05) homogeneity of variance with Levene’s test (p > .05). A two-tailed independent t-test was used to detect significance in the result data. No significant difference was found in the total self-reported narrative engagement scores between the group viewing the interactive version (M = 41.08, SE = 6.39) and the group viewing the non-interactive version (M = 41.92, SE = 6.53), t(22) = −.32, p = .76, r = .07.

Upon deeper inspection of the data, the narrative engagement sub-scales showed no indication of difference, nor did the continuation desire: Both conditions had almost the same scores, with the control condition having a slightly higher but non-significant score in both scales. However, the attention levels between the groups (see Fig. 3 (A)) differed, as the group viewing the interactive version (M = 46.77, SE = 5.14) had higher attention levels than the group viewing the non-interactive version (M = 43.11, SE = 4.44), t(22) = 1.87, p = .04, r = .39.

6.2 Interviews and Observations

In order to compare the algorithm’s identification of attention with viewers’ own interests, we compared these as well. The self-reported interest scores from participants in relation to each zone can be seen in Fig. 3 (C).

Regarding pacing or tempo, four participants explicitly noted that the film was too slow. All of them were in the control group. Looking at the attention data of those four, two would have skipped 2 segments, one would have skipped 3 segments and one would have skipped 5 segments, and half of these skips corresponded with their self-reported interests.

Concerning the topic of ‘zoning out’, 5 participants stated that they did not zone out. Other participants noted that they reflected on the story and the characters, but nothing unrelated to the film. For comments relating to boredom or zoning out, 9 were possible to pinpoint to specific zones, and 10 only to a lesser degree. Of the zone-specific, six of them were in respect to zone 1, two to zone 2, and one to zone 4.

Fig. 3.
figure 3

A) Box-plots depicting the measured attention between experimental and control group. B) Depicts the attention levels across the film with smoothed by a rolling average. C) Depicts the self-reported interest scores for each of the zones.

Full size image

One participant in the control group vocalized a large amount of mind-wandering. Thoughts revolved around intricate implications of the story, the film’s production, and the experiment itself. Had this participant taken part in the experimental group, the film would have skipped three times, though it should not have in respect to his self-reported interests.

One participant noted a great deal of confusion in zone 3 in relation to its content, and its meaning to the story. No skips were identified, though it should have based on her self-reported interest score.

Multiple of the participants who had watched the original movie brought it up in some form during the interviews. One participant who had recently watched the original movie, mentioned doing comparisons between the two versions in relation to actors, mise-en-scène, and changes to the story. Other participants mentioned being able to remember or recognize close to nothing. Some suddenly recalled aspects of the film they had forgotten.

6.3 Assessment of the Skipping Algorithm

The confusion matrix, which was developed to assess the performance of the skipping algorithm compared to expected skips based on participants’ interest ratings, showed the following results: true positive (correctly identified skip) = 12, true negative (correctly identified no skip) = 49, false positive (incorrectly identified skip) = 25, false negative (incorrectly identified no skip) = 10. Derivations from the confusion matrix are presented in Table 1.

Table 1. Derivations from confusion matrix.
Full size table

7 Discussion

The experiment described above showed no difference in narrative engagement between the experimental and control group, but it did indicate that the experimental group had significantly higher attention. Together with the fact that all participants who thought the film was too slow were in the control group, this indicates that the interactive skippable segments could have some sort of effect.

The short film was well-received according to the measurements of narrative engagement, despite containing parts some viewers did not find particularly interesting. This is likely a result of the skippable content being too short to make a measurable impact, and the content zones were too generally enjoyable, as only the zone 1 appeared controversial in relation to the viewers self-reported interest scores. Another factor relating to the lack of difference in narrative engagements relates to the design of the film. While some viewers rated some zones rather low, it did not affect their overall experience. This could be due to the design amending transient lowered interest by re-engaging them with an exciting plot point. Future studies could feature longer additional and more controversial content to more clearly assess the effect of skipping certain content.

The assessment of the skipping algorithm showed that it did not correlate with participants’ interest scores in any high degree, as it only performed slightly better than random. Inspecting Cohen’s Kappa for the sub-groups indicate that the algorithm performed poorer than chance for males and viewers of the original movie, while it did appear to more correctly evaluate skips for females and those who had not seen the original movie. A previous study on gender differences in ‘attention’ showed that males’ are higher, but declines, while females’ are more stable [21]. In this experiment, a similar tendency is indicated, with the addition that females’ attention increased across the short film. Although further experiments with more participants are needed to make any generalizations, if such gender differences are existent they could be needed taken into account. In relation to having seen the original movie, it is somewhat obvious that this likely affected the experiment. With novelty and surprise reduced, it is likely that this would have a diminishing effect on attention levels. How it could have affected the design or evaluation is however not entirely accounted for with this experiment - The participants in this sub-group was however somewhat equally distributed between the experimental and control conditions. Future experiments with adaptions of existing content could be more inquisitive in this aspect, as it indicates it could have an effect on the viewers perceptions and attention.

In this experiment, participants were not informed of the control they had. Future experiments could investigate how knowledge of control in a design like this would work. Ramchurn et al. [26] found that while their lean-back mechanism was ambiguous and covert it its working, some users still attempted to exert control. In our design, could viewers be able exert control over their attention to elicit skips by disengaging with the content? How this would work at a conscious or subconscious level, and how this relates to the viewing experience, and if it is even desirable, require further research to be answered.

8 Conclusion

In recent years, the usage of passive control in interactive cinema has opened unexplored territory with the field of brain-controlled cinema. Previous projects had used a ‘loose’ mapping in assigning functionality to the metric of ’attention’ from consumer-grade Brain-controlled Interfaces.

With this project it has been explored how individual film viewers’ subjective interests can be catered to, by attempting to use the attention measurement as an indication of interest in specific parts of a film. A short prototype film adaptation of scenes from the movie Ex Machina was made, that contained 4 zones with content that could be skipped according to the viewer’s level of attention.

The results from the between-subject test showed no significant difference in narrative engagement between the experimental and control group. There was however an indication of a significant difference found in their measured attention-levels, with the experimental group having a higher average attention than the control group. Furthermore, there appeared to be indications that gender differences in attention-levels affected the performance of the skipping algorithm, as well as whether participants had seen the original film. Additionally, it was found that skips often did not seem to correlate with participants’ self-reported interests in the presented topics to any reliable degree. Uncertainties about how the attention data can be interpreted and how affected it was by external factors make it difficult to form any definitive conclusions. However, due to the average attention being significantly higher for the experimental group, it suggests that the interactive film with skippable segments did have some impact that remains to be fully accounted for.