Key words

1 Introduction

No one doubts that media has the potential to attract the audience . Nevertheless, little is known about how the audience processes media stimuli. Many researchers agree that an essential function of fictional stories is the generation of emotions (e.g., [1]). Tan describes movies as emotion machines [2]; Konijn and ten Holt name emotion as “key construct in processing media messages” ([3], p. 37). Media induces emotions through the presentation of a story by using a huge variety of storytelling techniques and formal features that aim to involve the viewer. In fictional stories, ideas and ideals concerning our social world are presented in a way that is easier to grasp and to comprehend than our complex social environment [4]. Thus, if we aim at understanding film art, it is essential to understand the techniques of emotion elicitation as well as the emotional processes that de facto occur during the reception of a movie.

In order to understand the nature of such emotional processes, we analyzed patterns of physiological and facial reactions to a movie by using the T-pattern-detection of THEME. The study, we took the data from, was conducted to investigate emotional reactions to the animated short film Father and Daughter [5, 6]. The narrative structure of the movie and the most important formal features were determined using different theoretical approaches in film studies. Those structures and features are now—in the context of this contribution—conceptualized as events, i.e. as time intervals within the movie that are hypothetically of relevance for the viewers. We also use THEME to determine reoccurring combinations of events in the movie. In the reception study, heart rate and skin conductance of 30 participants were measured. The faces of the consenting 16 participants were videotaped during the reception and were analyzed using the Facial Action Coding System [7]. We will focus on the analysis of reoccurring combinations of physiological and facial reactions associated with particular events in the movie for those 16 participants. The results are interpreted in the light of emotion theories.

2 Media-Induced Emotions

Emotions are key features of our mental architecture. They are activated by means of specific cues in our environment and focus our attention on urgent and relevant information. Furthermore they allow a fast evaluation of environmental stimuli or events for the organism’s needs, plans, or preferences. In addition to this they initiate psychological and physiological processes to support rapid reactions for dealing with a given situation [8]. Further, the nonverbal aspects of emotions include “the communication of reactions, states, and intentions by the organism to the social surround” ([9], p. 557). Most theorists support the view that emotions have several components or subsystems (e.g., [10, 11]). Conceptualizing emotion as a process, Scherer defines emotion “as an episode of interrelated, synchronized changes in the states of all or most of the five organismic subsystems” ([12], p. 93). The Component-Process Model by Scherer distinguishes five subsystems of an emotion: (I) the cognitive appraisal, (II) the physiological arousal, (III) the motor system and most notably the facial expression as part of this system, (IV) the subjective feeling, and (V) the motivational system [12, 13] An emotional episode is conceptualized as a component patterning process driven by cognitive appraisals. This means that an emotion is not due to the eliciting event itself, but to the evaluation of the event by an individual.

Thus, emotions are a process of sequential appraisals, which result in a certain pattern of outcomes in the emotion response systems. The process of appraisal itself is a cognitive process, thus it is not accessible to direct observation and also impossible or at least difficult to access via verbalization. But each appraisal evokes an adequate reaction in the subsystems and thus triggers a corresponding reaction, for instance, in facial expression . For several outcomes of stimulus evaluation checks, Scherer and colleagues [14–19] predict related changes in facial expressions. For example, appraising an event as novel is related to raising the eyebrows or frowning; appraising an event as pleasant is related to pulling the lip corners upwards and raising the cheeks (like in smiles). If a certain appraisal does result in a change of facial expression—as proposed—facial expressions are observable indicators of unobservable emotional processes.

Thus, we use knowledge about specific changes in the physiological and the motor system (facial expression) for the interpretation of emot ional processes that are operationalized via T-patterns of events in a movie and the reactions of the recipients. Emotions that arise as a result of the reception of a movie may differ from those that are experienced in daily life in many respects due to the specific stimuli and due to the specific (principally safe) reception situation (for more details see [20]). However, we assume that media-induced emotions are processed in the same way as naturally occurring emotions ([20, 21], see also [3]).

3 The Movie and Its Structures and Features

Under discussion is the animated short film Father and Daughter by the Dutch film maker Michael Dudok de Wit [5]. The 8-min-long movie is centered on a human conflict:

One day a father leaves his little daughter behind by rowing away in a boat for reasons we are not told. The daughter returns again and again with her bicycle to the place on the shore from where her father’s boat left. She is obviously longing for his return, but this does not happen, not even by the time the girl has become older. She is shown in different ages and circumstances. Finally, as an old woman, she goes down the hill and into the water; she seems to commit suicide. But then the movie switches to a dream world: The water changes to grass and she finds her father’s empty boat in it. She lies down in the boat, and then she stands up and sees her father again; while running to him she becomes a child once more [6, 21].

Movies are compositions of narrative structures ; they are a temporal arrangement of events , which serves as a reception-guideline [22]. This arrangement is basically meant to involve the spectator. But, there are also features that allow the spectator to create distance by bringing to mind that the movie is an artificial, esthetic product and not reality. Thus, in a first attempt, we distinguish between two opposing modes of reception: involvement and distance. The relation between involvement and distance is not meant to be a dichotomy—involvement or distance. Instead a cognitive and emotional “going into“ the movie is possible, because of a distance now and then [23].

Different theoretical approaches in film studies are combined to analyze the movie. The premise hereby is that the effects of isolated cinematic elements, like for instance a certain protagonist or a particular acoustic element, cannot be empirically concretized in an appropriate way without considering the narrative context. The cinematic composition was approximated by determining the basic narrative structures [22]. This analysis was complemented by considering acoustic features and specific features of the movie that allow for distance. The analysis finally results in a list of 11 event types for the short film Father and Daughter (see Table 1). In the paragraphs that follow, all 11 types of events are described step by step (for more details see [6], p. 44–48).

Table 1 Types of events in Father and Daughter
Full size table

The film scientist Wuss [22] describes both, the sequence of central events on screen and the viewers’ reception as processes of problem-solving. In movies often problematic or conflicting situations have to be solved. Two different kinds of conflict are connected in Father and Daughter. The central conflict that results from the father’s leaving and the inner conflict of the daughter, which is intensified as it becomes increasingly clear that she cannot reconcile herself to the loss of her father. The movie contains three scenes which clearly represent the central conflict: (I) The father leaves his daughter at the beginning of the movie; (II) the final resignation of the daughter when she seems to commit suicide, and (III) when she lies down in the boat to just sleep or even to die and rest in peace (see Fig. 1).

Fig. 1
figure 1

Three conflicting situations

Full size image

The movie offers three clearly markable solutions to the conflict. At one point in the movie it seems that the daughter may reconcile herself to the loss of her father, when she comes to the shore with a friend, sitting on the back of his bicycle—probably the man she loves. At another point in the movie the director switches from reality to a dream world, in which everything is possible: The daughter lives—a scene shown after the suicide-scene. This also indicates a solution. Finally, she lies down in the boat. The viewer may conclude that now she is dead. But then she stands up and meets her father once more. This is the third solution.

According to Wuss [22], the so-called Topic Lines generate sense by the repetition of “small“ events, which are difficult to detect for the recipient (as well as for the researcher). The narration of Father and Daughter is structured by a main Topic Line, which intensifies the inner conflict:

The daughter returns again and again on her bicycle to the place, where her father left. There she looks out over the empty water and then rides away again. Although these attempts seem quite different in their visual expression, they prove to be very similar in terms of content: Each episode invariably leads to the same disappointing result. By the repetition of the invariant situation it becomes clear that the father’s leaving represents an essential loss in the life of his daughter, and that her attempts to deal with this loss are futile [6, 21].

Two event types are differentiated: The daughter’s looking out over the water that signifies her longing for her father, and her riding away that signifies futility (see Fig. 2).

Fig. 2
figure 2

Example for one of 12 events signifying longing (one the left side of the figure) and an example for one of seven events signifying futility (one the right side)

Full size image

A third Topic Line can be identified: On her way to the place of farewell, the daughter encounters people who are also on their bicycles. The fact that the protagonists, who meet each other, are of different ages supports the illustration of the respective life phase, the daughter is in (see Fig. 3).

Fig. 3
figure 3

Two examples of nine events showing encounters

Full size image

At the beginning of the movie the director already establishes another Topic Line , which is repeated four times during the movie: The protagonists ride their bicycles uphill or upwind. We see the strain the protagonists are experiencing and we feel part of it ourselves. The scenes illustrate the enormous effort it takes the daughter to deal with the loss of her father [6, 21].

In Father and Daughter each change-over from one segment to another is reinforced by remarkable changes of the music. These turning-points provide a clear segmentation of the movie and thus help the audience to structure the plot [6, 21].

Humor in Father and Daughter allows for distance. The term humor stands for funny or spontaneous events, which are characterized by overstatement. Two events in the movie are characterized by humorous elements: In the first scene, the daughter on her bicycle is blown forward by the wind and passes an old woman. In the second scene, a young girl meets the daughter, who is an old woman now. After the encounter, the young girl hops with her bicycle over a stone and the daughter, who is now an old woman, rings the bell of her bicycle [6, 21].

The next event type is called hole. Such holes provide the possibility to reflect, because the pictures on screen contain only minor information. For instance, pictures of landscape just indicate that time passes by (“epic distance”) [24, 25]. Another form of distance is possible by using symbols. In Father and Daughter different symbols interrupt the flow of narration and point to the movie as an esthetic construction. Such pictures are often enigmas, which have to be decoded (“lyric distance”) [24, 25]. Finally, distance may be induced by the music. The music swings up and then tilts into a more bright or joyous direction in contrast to a somehow painful event on screen. Or reversely, the music is sad in contrast to the cheerful action displayed, for instance, when children are playing on the shore. The antagonism between picture and music, so-called counterpoints , allows for distance [6, 21, 26].

4 Operationalization of Reception Processes

In order to operationalize the reception pro cesses, we refer to data of 30 participants gathered in a previous study [6]. In this study, the movie was preceded by a cinema-like program with commercials and movie trailers in order to allow the participants to adjust to the unfamiliar situation. At the outset participants answered a short questionnaire to obtain information about movie experience and socio-demographical data. During the movie reception heart rate (HR) and skin conductance (SC) were measured. For those participants, who consented to be videotaped, facial behavior was recorded via a camera. Each participant’s liking of the movie and self-reported emotional experience was assessed during a personal interview, which was conducted right after the measurement. In sum and in terms of the conceptualization of emotion processes by Scherer [12, 13], the physiological, the motor, and the subjective components of emotions were measured.

The movie was presented on a 19-in. diagonal computer monitor. The camera was positioned behind the monitor. Physiological data were collected with a transportable measurement system. For measurement of HR, three electrodes were attached to the upper part of the body. For the measurement of SC, two electrodes were attached to the palm of the nondominant hand. All dates were arranged for the evening between 5:30 p.m. and 8:30 p.m. in order to control for possible differences caused by the day time. Data recording was conducted individually for each participant. The conditions of the situation were standardized for all participants. Each session was supervised by the same (female) experimenter. During the session the experimenter sat 2 m (about 6 ft) away from the participant. She logged the observed body movements of the participants into the system using a keyboard that is part of the measurement device. With the help of these information artifacts caused by movements could be paralleled exactly with the physiological reactions and be removed for the analyses. One session lasted approximately 45 min (for a more detailed description see [6]). For our analyses we used the data of the 16 participants, ten women and six men, who consented to be videotaped. Table 2 provides an overview of their physiological reactions and body movements.

Table 2 Number of significant physiological reactions and of body movements
Full size table

The recordings of the facial expressions of the participants during reception were analyzed using the Facial Action Coding System (FACS) , which was developed by the psychologists Ekman and Friesen [7]. FACS is a reliable method of describing the contraction of each facial muscle. Certified FACS-coders deconstruct an observed facial expression into so-called Action Units (AU). Specific combinations of Action Units stand for basic emotions. The coding was done by one coder in a blind-trial, without her knowing, which scene of the movie caused the reactions. The most frequent facial expressions were AU 45 (the eye blink; approximately 60 times for all participants), AU 14 (the lip corner depressor or dimpler; approximately 50 times), and AU 14 in comb ination with other AUs (approx. 30 times).

5 T-Pattern-Analysis

First of all, we are interested in patterns of events within the film Father and Daughter. Such patterns allow for insights into the narrative complexity of movies [27, 28]. Secondly, we want to analyze the dynamic relations between aspects of the movie and the reactions of the spectators. The analysis of T-patterns perfectly meets the requirements of an analysis of the manifold and complex dynamic relations between events in the movie and reactions of the viewers: Using THEME enables us to analyze both, immediate and delayed reactions. Even an anticipatory reaction preceding the event is possible. Moreover, the fact that recipients remember past events with a longer time delay can be taken into account. In Fig. 4 the data input prepared for an analysis with THEME is visualized.

Fig. 4
figure 4

Time series behavior record of all subjects

Full size image

The behavior records of all subjects were joined in a single data set. On the timeline, 16 series of events (one for each of the participants) appear one after another. The event types include the coding of cinematic events (event types 1–22; for each event begin and end were coded), the FACS-coding (23–119), the coding of movements (122) and the coding of physiological data (heart rate deceleration 120, heart ra te acceleration 121, and skin conductance response 123).

5.1 T-Patterns of Events in the Movie

The 11 event types include 83 events overall. We looked only at highly significant patterns (p < 0.005), which appear at least three times during the movie. Figure 5 shows the longest pattern we found for the events defined in the movie Father and Daughter.

Fig. 5
figure 5

The longest pattern of events (i stands for involvement, d for distance)

Full size image

This pattern can be “read“ as follows: After showing the futility of the daughter’s attempts to deal with the loss of her father the music changes. Then, there is a hole, i.e. pictures with minimal information are shown on screen. Then, the daughter encounters someone. Then she looks out at the place on the shore—symbolizing her longing for her father. Again, the music changes. Again, there is a hole. And again, she encounters someone. This pattern occurs three times in the middle of the movie. A look at the most frequent pattern in the movie (see Fig. 6) reveals that a part of this longest pattern is consistent during eight repetitions: After a scene showing the daughter’s longing for her father, the music changes and then she encounters someone (see also [29]).

Fig. 6
figure 6

The most frequent pattern of events

Full size image

This most frequent pattern impressively points to the repetitive narrative structure of the short film, and at the same time demonstrates the importance of Topic Lines (longing, futility, encounter). Topic Lines generate sense by the inner-textual variation of important themes or motives. They differ in range, dispersion, and complexity. The term Topic Line summarizes quite different circumstances: Often, behavioral manners that characterize the protagonists, are merged into different contexts ([22], p. 68–81). Topic Lines are more inconspicuous and relatively unconsciously perceived cinematic structures. Nevertheless, the repetition can evoke latent expectations in the viewer’s mind. The evidence of Topic Lines is low at the beginning of a movie and increases with the frequency of repetition throughout a movie ([22], p. 29, 70; [6], p. 42f.). Combinations of different Topic Lines evoke measurable effects on the spectators [6]. THEME can help to describe and to visualize structural relations in movies, which cannot easily be recognized neither by watching the movie again and again, nor through a detailed dramaturgical analysis, but nevertheless are of relevance for reception processes (for analyses of other movies see [27, 28]).

5.2 T-Patterns of Topic Lines, Physiological Reactions, and Facial Expressions

To further elaborate on the notion of the relevance of Topic Lines for reception processes we conducted analyses concentrating on patterns of Topic Lines, physiological reactions, and facial expressions for the 16 participants, who consented to be videotaped. The main Topic Lines in the short film Father and Daughter intensify the inner conflict of the daughter by showing her frequent returns to the place, where her father left, signifying her longing and the futility of her attempts to deal with the loss of her father (see 3). We find peculiar T-patterns that enable us to draw conclusions about the impact of these narrative structures on the recipients: Seven (of 16) participants smile before a scene ends, in which the daughters’ longing for her father is shown. Altogether, 14 times a combination of AU 6 (the cheek raiser) and AU 12 (the lip corner puller) was observed (see Fig. 7).

Fig. 7
figure 7

T-pattern combining longing and a smile

Full size image

The combination of AU 6 and AU 12 indicates that these scenes in the movie are appraised as intrinsically pleasant and/or as conducive to reach goals [18]. The participants, who smile, may expect that the father will come back now. However, we also find significant patterns that indicate a defensive reaction immediately after the movie scene that shows the daughter’s longing for her father: We observe eight significant heart rate accelerations immediately after the longing-scenes (for seven participants). And even more obvious, we observe significant skin conductance reactions for nearly all participants for most of the longing-scenes throughout the movie. HR acceleration and skin conductance reactions indicate a negatively valenced reaction. Looking at patterns of the futility-scenes with facial expressions reveals a more detailed interpretation: At least four participants move their body (after longing- and) during futility-scenes and in their faces appears a combination of AU 10 (the upper lip raiser) and AU 14 (lip corner depressor or dimpler). The combination of AU 10 and AU 14 indicates, that the participants of our study appraise the scene as violating external and internal standards [18], when the futility of the daughter’s attempts to deal with the loss of her father is shown in the movie. The participants, who show these facial expressions, may think about the inappropriate behavior or the unfairness of a father leaving his daughter alone. The T-patterns illustrate the possibility of stepwise appraisals postulated by Scherer and colleagues [14, 15, 18, 19] and their transferability to movie reception.

AU 14 is one of the most frequent facial expressions observed in the study reported in this contribution as well as in a number of other studies with audiovisual media stimuli [30, 31]. AU 14 thus indicates a very typical emotional appraisal when watching TV or movies (in the laboratory). If this can be interpreted as a kind of “Leit-Affekt“ linked to mo ral evaluations is currently under dicussion [32, 33].

5.3 T-Patterns of Acoustic Events and Facial Expressions

As searching for T-patterns can help to find out more about reactions the participants are not aware of, we decided to also have a look at patterns consisting of the acoustic events in Father and Daughter and the facial expressions . The music supports the clear-cut segmentation of the short film. We find four occurrences of a T-pattern combining changes of the music with a combination of AU 1 (the inner brow raiser), AU 2 (the outer brow raiser), and AU 5 (the upper lid raiser). This combination of AUs signalizes that the situation is appraised as new and kind of sudden [18]. The combination of AUs also indicates the basic emotion surprise [7]. Moreover, we found patterns of changes of the music and AU 45 (eye blinks; four occurrences), which can also be interpreted as concomitants of newness and surprise. Even more interesting are other patterns, in which a combination of AU 4 (brow lowerer) and AU 7 (lid tightener) occurs (seven times) shortly before the music changes. This combination of AUs is commonly interpreted as an appraisal of an unexpected event [18, 34]. The participants seem to anticipate the change within the plot. The fact that T-patterns help to operationalize expectations makes this method especially interesting for movie impact research.

The results of the T-pattern analysis of the counterpoint s (antagonisms between picture and music) are also interesting. Counterpoints allow for distance [26]. In fact, we find patterns in which AU 14 (the dimpler) accompanies scenes during which the music contradicts the content at least for three persons. Basically, AU 14 signalizes that an event is appraised as “something is wrong”, i.e. as a kind of irritation. Possibly, this was the exact intention of the movie’s music composer. Very little is known about the effects of such acoustic features. The results call for further empirical research at any rate, because we observe that the acoustic events are very often accompanied by body movements. As body movements are not coded in detail in this study we do not report the results here (see also [35]).

5.4 T-Patterns Associated with Moments of Impact

A regression analysis of the physiological data of the 30 participants, conducted by Suckfüll [6], revealed a very strong effect for the humorous scene , in which the daughter is blown forward by the wind and a strong effect of the conflict situation, when the daughter seems to commit suicide (see Fig. 8).

Fig. 8
figure 8

Scenes causing strong physiological reactions

Full size image

The strong effects of the two scenes can be ascribed to their dramaturgical embedding: Both scenes are prepared by the use of Topic Lines . The motive of effort prepares the recipient by eliciting motor mimicry for the relief effect of the scene in which the daughter is blown forward by the wind. And the longing for her father and the futility of her attempts, repeated eight times during the short film, prepares the recipient for strong emotions, when the final resignation of the daughter is shown.

But, how exactly do the facial reactions of the recipients look like? The humorous scene clearly elicits positive emotions. A number of participants smile, which was indicated by a combination of AU 6 and AU 12 in reaction to the scene, in which the daughter is blown forward by the wind. The pattern occurs seven times in the behavior record of all subjects. The smile indicates a pleasantness-appraisal [18] and the basic emotion happiness [7]. Grodal speculates that in movies a sensation of relief may be accompanied by eased laughter ([36]: Chap. 8). Systematic research is required, in which other possible indicators of relief (e.g., body movements) are observed, coded, and analyzed.

The patterns associated with the conflict-scene, which causes strong physiological reactions (see Fig. 8, right screenshot), also involve smiles. Yet, the smiles (combinations of AU 6 and AU 12) occur before the daughter seems to commit suicide. All participants, who smile before the daughter seems to commit suicide, have a significant skin conductance reaction afterwards (four participants, see Fig. 9).

Fig. 9
figure 9

T-patterns combining smile, conflict, and skin conductance reactions

Full size image

Another pattern, which occurs 14 times (eight participants) illustrates that the conflict-scenes are often followed by a heart rate deceleration. A decrease in HR can be seen as an indicator of the initiation of an attentional state [6]. It is interesting that the first conflict-scene (the father leaves the daughter) is not part of this pattern. In the beginning of the movie, the participants obviously did not yet realize the central conflict the movie is based on.

6 Implications for Future Research

It is impossible to discuss all T-patterns that we found in this paper due to limited space. In summary, the analyses enlighten the dynamic nature of movie reception: Important scenes of a movie are prepared with virtuosity, creating “lines“ or repetitions of motives combined with each other. The film maker plays with the expectations and emotions of the viewers—expectations are raised and then disappointed. THEME is perfectly suitable for analyses of iterative and constantly changing processes. One main advantage of adapting THEME for movie reception processes is the possibility to systematically analyze reactions that occur in advance of an event in the movie.

We have already mentioned some suggestions for future research throughout the description of the patterns, but we want to add one more idea for future research: It is especially interesting to look for patterns associated with scenes in a movie, in which nothing happens. Thus, future analyses should focus on the so-called holes, the scenes in a movie that contain only minor information. It is conceivable that the viewers show intense reactions exactly during these scenes as the film maker offers a break or a pause for reflection about prior scenes or for imagination in the sense of integrating the cinematic events into the viewer’s own life. At the same time such scenes may have the function of raising suspense.

The possible emotion processes elicited by such stimuli cannot be analyzed without appropriate data collection methods and tools for analysis. To fully exploit the potential of THEME for an understanding of film art, it is essential to systematically prove hypotheses on the basis of a bigger sample or with manipulated movie sequences in an experiment. The results reported in this contribution support the formulation of concrete hypotheses. The study was already replicated under optimal conditions in a laboratory designed for reception studies with 150 participants watching the short film Father and Daughter. The analyses are not yet finalized.