Truthfulness and Affect via Digital Mediation in Audiovisual Storytelling

Abstract

This chapter investigates different ways in which the film techniques of digitally mediated images—such as found footage, diegetic camera, and computer screen—achieve story truthfulness and affective engagement in the viewer’s narrative interpretation process. The pursuit of truthful storytelling is to demonstrate objective facts, while mediated images in film are predominantly subjective. The chapter starts by reviewing the perennial paradox of two seemingly mutually exclusive narrative functions and then tackles the paradox by proposing a multi-leveled framework, synthesizing semiotic conceptualization and cognitive research findings. It also analyzes the various forms of digital mediated images in films over the last two decades and sheds light on how the functions of truthfulness and affective engagement can be closely intertwined rather than in conflict.

1 Introduction

This chapter investigates the intermedial relation of different media frames blended in narrativefilm and in what ways the intermedial blends, particularly the digital mediated images in films, impact on the two narrative functions—namely, the viewer’s interpretation of message truthfulness and affective engagements. The term ‘digital mediated images’, as it is applied in this chapter, should be read as a broader conception than that of just the new digital media used diegetically by fictional characters in the film (cf. Tseng 2018b): in this chapter, it describes various forms of added realism, among them news footage, intra-diegetic camera, and computer screen.

We live in a culture that is increasingly reshaped by transformations in audiovisual media. Within the last 20 years, our film experience and its presence in everyday life have changed rapidly—the film experience now encompasses a larger, ever-shifting human interaction with technologies of perception and expression. Against the backdrop of our increasingly technologically mediated society, this chapter addresses why digital mediation in film is a valuable resource for exploring the persuasive and rhetoric capacities of audiovisual storytelling itself and how it addresses questions of perception, affect, and truthfulness.

For decades, new media features have been frequently blended in the making of narrative films. In the 1960s, McLuhan (1964) already proposed that new inventions and technologies produce variations in sensory input that customarily require adjustments from our sense organs. More recently, Brown also argues, “digital technology has expanded cinema and the psychological sciences have expanded our understanding of perception to such a degree that new theories of cinema and our perception(s) of it are urgently required” (Brown 2013: 8). In summary, it is generally believed that the emergence of new media forms leads to the constant modifications to the way we experience audiovisual storytelling. Drawing on these technology-centered proposals, one focus of this chapter is precisely to interrogate to what extent evolving technologies blended in narrative film modify the viewers’ narrative interpretation process and viewing experiences.

Academic discussions of digital mediation in film have often been focused around particular genres, for instance, found footage in horror films (Heller-Nicholas 2014; Sayad 2016), computer screen and diegetic camera in thriller or war films (Stewart 2009; Pisters 2010). This chapter poses the question from a different perspective and instead broadly asks how the aesthetic choices of digitally mediated images generate the seemingly paradoxical narrative impacts and interpretative outcomes—subjective affective intensity and objective narrative truthfulness. In principle, mediated images in film often function to enhance the viewer’s interpretation of truthful storytelling. For example, historical docudrama or biopics often use news reports or historical archive footage for endorsing the authenticity of information source. Thrillers or horror films use the strategy of found footage, namely, images shown via the character’s handheld camera or computer screen, to add narrative realism through simulating the visual aesthetics we are familiar with in our day-to-day life. In many other cases, using mediated images in film are dominantly subjective and evoke a viewer’s affective attachment rather than detaching the viewer to reflect on objective fact. Section 5.2 will first review perennial paradox of two seemingly mutually exclusive narrative effects: emotional engagement and message truthfulness. Section 5.3 will then tackle the paradox by proposing a multi-leveled, semiotic framework synthesizing cognitive research findings and semiotic conceptualization (cf. Tseng 2018a; Tseng et al. 2018), distinguishing the two analytical levels: presemiotic level of media properties and semiotic level of narrative semantic structure (cf. Elleström 2020). This chapter will then employ the framework to analyze the various forms of digital-mediated images in film. The analyses will shed light on how the affective and interpretative functions are closely intertwined rather than paradoxical.

2 Perennial Paradox: Achieving Affective and Truthful Impacts

For decades, affective engagement and cognitive-logical interpretation in audiovisual narratives have been regarded as incompatible processes. Several film scholars, for instance Deleuze (1986), have argued that the transformative and immersive power of cinema specifically arises from an ability to produce non-cognitive affective immersion that can persist beyond the conscious consideration of narrative interpretation. While cognitivism sees cinema as naturally conducive to human systems of meaning-making, the affect-centered perspective argues that film narrative is able to destabilize the schema of human thought of logical order of event perception and action, and this non-cognitive, bodily experience is the basis for generating affective response in film Deleuze (1989: 169).

The paradox of emotional engagement and cognitive-based interpretation of message truthfulness has also been raised in the broader realm of communication research. Recent empirical evidence has shown that affective engagement in the character-based narratives can powerfully achieve narrative persuasive impact, and the narrative focusing on cognitive reasoning of truthful events decreases the strength of such impact (Green et al. 2002). One explanation for this is that the fundamental constituents of human memory and social communication rest on a person’s experiences and the experiences of others, rather than the cognitive activities of knowledge reasoning (Schank and Abelson 1995). In other words, research suggests that it is the default mode of human thought that a narrative way of thinking drawing on someone’s personal, emotion-laden experience endows individual cases and anecdotes with significant weight toward evaluating evidence (Bruner 1991).

Moreover, the paradoxical relation between affective engagement and logical reasoning of truthful events and actions is also the center of debates in the context of educational impact. Some scholars believe that emotional engagement in fictionalstory might distract the learning of socially and politically important issues, because by engaging a narrative in an affective way, the communication fails to reach the goal of engaging scientific, logical reasoning (Bogost 2007, 2017). Moreover, scholars also indicate that, since affect-centered storytelling can be highly persuasive regardless of the validity of the underlying truthful claims, the use of narrative messages becomes an oversimplification at best or manipulative at worst (Dahlstrom and Scheufele 2018). Even an otherwise desired outcome (such as correct knowledge about climate change) could be viewed negatively if the desired reasoning process of message truthfulness was not engaged. Hence, in terms of educational purposes, the paradox comes into focus: affective storytelling can engage people and make educational information relevant to personal experience, while simultaneously encouraging a narrative way of thinking that places scientific stories on a similar level to any other plausible story that may or may not support message truthfulness.

In sum, it is believed that the processes of affective engagement and the representation of message truthfulness are mutually exclusive. One decreases the impact of another, that is, the more affective engagement is triggered, the more uncertainty, less truthfulness of the narrative becomes. And vice versa, the more authentic the message one intends to communicate, the more affect-neutral narratives one needs to construct. The next section tackles the issue by showing the interconnectedness rather than mutually exclusiveness of the two processes. This tricky relationship can be best unraveled by analyzing the coexisting functions of affective and truthfulness in film using the techniques of mediated images.

3 Tackling the Paradox via Semiotic Approach to Narrative Functions

As various digital, dynamic forms of mediated images emerge in cinema, film researchers have argued that inserting multiple frames stirs up the viewer’s awareness of non-diegetic manipulation of filmmaking and hence destabilizes the coherent semantic contents of the narratives (cf. Stewart 2009; Ecke 2010; Poetzsch 2012). Nevertheless, our previous work has insisted that the multiple frames in the film actually do not disrupt the narrative construction at the semantic level (Tseng 2016, 2018b). In other words, despite the ever-present framing devices of digital mediated images, such as timecodes, hyperlinks, shaky images—the linearity and sufficient cohesion mechanisms of the narrative semantics still construct a straightforward meaning comprehension path for the viewers. Our analysis untangled the confusion of narrative incoherence and disruption of digital mediation, and this is based on the semiotic framework distinguishing the two fundamental analytical levels: media materialities and semantics of narratives.

For tackling the paradox of affective engagement and meaning interpretation in film, this chapter expands our previously proposed semiotic framework and puts forward a multi-leveled approach to narrative functions, integrating semiotic and cognitive findings, and addresses how the media affordances of digital mediation can actually synchronize different narrative functions at the narrative semantic levels.

In particular, we discuss these issues by exemplifying and analyzing the beginnings of several recent films substantially using digital frames, such as Redacted (2007), Cloverfield (2008), Searching (2018), and Profile (2018). We focus on the beginnings of the films because beginnings in all films function specifically to establish a hypothesis and emotional expectation, to provide first impressions that later developments of the narrative will be measured against (Hartmann 2009). In psychological terms, the function of the initial portions of a film has been described according to the primacy effect and priming (Luchins and Luchins 1962) or anchoring bias (Tversky and Kahneman 1974). A distinctive structuring function has also been theorized in studies of textlinguistics, for example, by Martin (1992), who develops the notion of “macro-themes” to describe a communicative function that serves the role of signposting the organization of the text following.¹

In addition, we select these films for analysis because they represent the application of evolving media technology in narrative films within the recent two decades. Some films are made in the 2000s when portable digital camera is widely used for documenting and preserving events, while others are made ten years later, when online communication and social media become a crucial part of the communication and information distribution. As the following sections will show, despite the evolving technologies, the main narrative functions of truthfulness and emotional engagement remain stable, while different media affordances between camera and computer screen indeed bring about some distinctive ways of realizing these narrative functions.

3.1 Multi-leveled, Semiotic Approach to Narrative Functions

Several film scholars have proposed that a general distinction needs to be made between the process of narrative interpretation and the perception of individual media techniques. This distinction is particularly significant when examining the evolution and functions of film style. More specifically, it has been frequently argued that a high degree of narrative and discourse stability is the basis of narrative inference and genre expectation for spectators, despite the gradually dynamic deployment of audio-visual devices in recent decades. For instance, Bordwell (2006) identifies major features of spatial-temporal styles that have been astonishingly robust throughout the evolution of filmmaking. Jones (2015) compares 3D and 2D formats and shows how they function similarly in terms of narrative coherence and effect. Bordwell and Jones both argue that despite the evolving visual techniques over time—such as shorter average shot lengths, the use of wider range lenses of 3D format and computer-generated images—the composition of “space, time, and narrative relations (such as causal connections and parallels)” in mainstream films remains straightforward to identify and leads the viewer to effortless comprehension and prediction of film narrative (Bordwell 2002).

The distinction between filmic semantic meaning and media properties is shown in the diagram in Fig. 5.1. The analytical levels are developed for film, building on the theoretical notion of semiotic stratification, in particular, on the distinction between a stratum of discourse and one of form (Martin 1992). The left part of the diagram is the multi-leveled framework generally divided into media techniques at the bottom level and narrative semantics, social-cultural domains at the higher levels. The distinction is reminiscent of stratification that occurs in semiotic approaches such as the distinctions of expression plane and content plane by Hjelmslev (1953).

Fig. 5.1

Strata of narrative functions in film analysis

The more important notion, as far as meaning making in film is concerned, is the interrelationship between these strata. These strata are interrelated by realization: that is, film semantic configurations are ‘realized in’ the contextualization of media properties, and at the highest level, social-cultural aspects such as genres or aesthetic styles are realized in the contextualization, configuration of semantic structures. On the basis of this framework, what needs to be particularly noted is that a single type of material or film devices does not lead directly to any specific kind of meaning interpretation. For instance, as mentioned in the previous section, the media property of ‘digital mediated frames’ does not directly lead to any dynamic or demanding meaning interpretation processes. It is the contextualization of these digital frames in the broader narrative structures, which achieve specific narrative functions.

Similar distinction has also been explicitly pointed out by Elleström (2020), when pointing out the crucial categories of presemiotic phenomenon of mediation and semiotic phenomena of representation. In his words:

Mediation is the display of sensory configurations by the technical medium (and hence also by the media product) that are perceived by human sense receptors in a communicative situation. It is a presemiotic phenomenon that should be understood as the physical realisation of entities with material, spatiotemporal and sensorial qualities—and semiotic potential. For instance, one may hear a sound. Representation is a semiotic phenomenon that should be understood as the core of signification, which I delimit to how humans create cognitive import in communication. When a perceiver’s mind forms sense of the mediated sensory configurations, sign functions are activated and representation is at work. For instance, the heard sound may be interpreted as a voice uttering meaningful words. (Elleström 2020: 39)

Applying the analytical strata to examine digital mediation in film, the rest of this chapter will address the hypothesis: The two seemingly paradoxical narrative functions, message truthfulness and affective impact, can actually complement each other, when the presemiotic property of digital media frames are contextualized/semiotized with the narrative strategies, such as restricted narration or media channels widely used in the societies of the audience.

To address the hypotheses, the stratified semiotic framework put forward above needs to consider the latest findings of the cognitive research, in order to explain how the functions of truthfulness and affective engagement are achieved by blending digital media frames in films. This is elucidated in the following subsections.

3.2 Media Frames, Human Memory, and Truthfulness

In general, blending conventional cinema with the media frames, which the viewers use in their day-to-day life, increases the viewers’ perception of message authenticity and enhances the persuasive and rhetoric function of narratives. This hypothesis draws on the recent findings of neuro-cognitive sciences (Zacks 2015: 101–107). The empirical evidence shows that a piece of information does not necessarily have a persuasive impact in the moment we perceive it, but its significance may grow in our memories over time. Moreover, our memories about the media channel which carries the information may be blurred over time. For instance, we have probably all experienced this before: you can remember the content of a certain piece of information but do not quite remember where you saw, read, or heard it. Hence, the plausibility and truthfulness of a message framed by mediated images in film may increase over time. This is precisely the empirical foundation of mediated image and the narrative function of enhancing truthfulness in film.

Using mediated images to increase the truthfulness has been applied in cinema for decades. For instance, Oliver Stone’s film JFK (1991) cuts between actual footage of the alleged assassin Lee Harvey Oswald and the staged images of actor Gary Oldman who plays Oswald. It mixes fact with fiction to propagate the idea that Kennedy was the victim of a conspiracy. Phyllida Lloyd’s Iron Lady (2011) uses a similar strategy, intercutting between close-ups of Meryl Streep and real news footage, which sometimes includes archival images of Margaret Thatcher, in an attempt to blur the boundary between factual frames and fictional frames. Apart from biopics, several recent war films also use online news report, YouTube clips, Skype chats, and diegetic camera to add the realism and truthfulness of the representation of soldiers’ experiences and trauma. Along the same lines, the mediated images enhance the persuasive function, message truthfulness, and ideological impact through blending fictional narrative with the media frames that we use in our day-to-day life (Tseng 2018b: 54–55).

3.3 Distinguishing Embodied and Contemplative Affects

The narrative functions of affective engagement and truthfulness can be seen as intertwining rather than paradoxical if we consider the multi-layered research framework of human emotion recently put forward by several affective psychologists (cf. Asma and Gabriel 2019).

In general, human emotions are filtered through the three inter-relating layers of mind: At the lowest primary level, affective intensity, such as fear, lust, thirst, prod human beings for the exploitation of resources. At the middle level, human brain creates a close link between these primary affective systems and the experiential learning and conditioning that we undergo in our daily life. At this secondary level, fear, for instance, becomes more specific due to the day-to-day encounter with people and other objects. For example, we tend to be afraid of the dark, we have fear of height, we feel uncomfortable in a restricted, cramped space or when hearing grating sounds. At a higher level, emotion is enmeshed with higher-level conceptual and narrative thinking. At this level, we arrive at social-cultural related emotions, such as ruminations and elaborate, contemplative feelings. The empirical evidence also found that, although higher-level emotions are still rooted in the lower level of human primary affective instinct, nevertheless, the higher level plays a crucial role in the cognitive executive functions of the mind, slowing and policing our more automatic primary responses.

We can generally map these basic layers to the affective functions of audiovisual storytelling reviewed above. The affective types, which are bodily grounded and regarded as non-cognitive, can be categorized as emotions at the two lower layers. For instance, embodied affective response triggered by restricted space in film, distorted images and creaking sounds. The digital media frames can trigger these embodied affective responses when they are contextualized in particular narrative strategies. This is exemplified in the war film Lebanon (2009). Digital media frames are contextualized with the strategy of confined space. This film is a compact war film focusing on a group of Israeli soldiers operating a tank in hostile territory during the 1982 conflict in Lebanon. Example screenshots of the film are displayed in Fig. 5.2. Many scenes of the film use the limited, rounded-off perspective of the frames (implying the soldiers’ perspective from the tank toward the battlefield outside). In other words, Lebanon blends the frames of the soldier’s telescope with restricted space to embody a cramped and suffocating affective response.

Fig. 5.2

Selected screenshots of Lebanon (dir. Samuel Maoz, 2009). All rights reserved

Finally, the higher-level emotions, namely, the more elaborated, empathy-related emotions are triggered by cognitive reasoning within particular social-cultural contexts. Mapping this to affective functions in film, this emotional level is the ‘product’ of cognitive, semiotic process, generated by the interpretation of film narrative events. This is precisely the level where the function of message truthfulness complements with emotional engagement—the viewers’ contemplative emotion is supported by the metaphorical link between film events and the viewers’ truthful, factual life experiences and is in turn the crucial factor for perspective taking, reflecting on the significance of social-cultural, educational issues dealt within the film.

Several recent films composed substantially of computer screen communications make use of the affective engagements based on this higher level. The film Profile (2018) is such an example. Shown in Fig. 5.3, this film follows a British journalist who dives into the online propaganda machine of the so-called Islamic State. The entire film is composed of online communication between the journalist and an ISIS fighter. In the beginning of the film, journalist Amy Whittaker creates a new Facebook profile under the alias of Melody Nelson, in order to investigate the recruitment of young European women by ISIS. She creates a persona online of a woman who has recently converted to Islam. Soon Amy is contacted by Bilel, an ISIS fighter from Syria. They begin to talk to each other via Skype, before she dangerously develops real romantic feelings for him. Throughout this film, the use of computer screen and online chats establishes the metaphorical link to the truthful communication form that is widely used in our digital age. The link then enhances the empathetic affective engagement drawing on the audience’s familiarity with Skype and other social media. This then supports the contemplative emotion triggered by issues and stories depicted in the film.

Fig. 5.3

Selected screenshots from Profile (dir. Timur Bekmambetov, 2018). All rights reserved

3.4 Forms of Digital Mediation in Film and Affective Engagement

It is often argued that blending mediated frames such as diegetic camera or computer screen in film enhances the viewer’s emotional attachment to the character due to the dominant use of point-of-view shots. Despite the embodied affective response possibly triggered by point-of-view shots, as analyzed in the Lebanon example, nevertheless, empirical investigation has suggested that the film technique of point-of-view shot itself does not automatically trigger any higher-leveled empathetic emotion (Andringa et al. 2001). In other words, the use of the point-of-view shot does not directly function to evoke particular emotional engagement in film (Smith 1994: 39). The chapter argues that this aspect again needs to be unpacked with the distinction of presemiotic media properties and semiotic structures. The affordances of point-of-view shots to generate empathetic, affective engagements can only be highlighted when these media properties are contextualized in the particular semantic structures of narrative, for instance, when restricted narration (Bordwell and Thompson 2010: Chapter 3) is constructed.

Restricted narration refers to confining the viewer’s narrative knowledge via a character’s first-person documentation, and observation is the narrative strategy that has been used in filmmaking for decades before the emergence of digital mediation. Ever since the emergence of the portable camera, films such as Georg (1964), The War Game (1965), and 84 Charlie MoPic (1989) have made use of the technique of mediated images to present a compilation of first-person observations. The use of footage from a mediated first-person perspective confines the audience’s knowledge to what a specific character knows. One major affective function of such restricted narration is that it powerfully builds suspense, uncertainty (this is why handheld camera footage or compute screen footage is widely used in horror films) and forces the viewer to empathize with the character’s experiences in the story world. In other words, restricted narration through first-person observation is an effective channel for locating the audience to empathize the character’s developments contextualized within the narrative structures.

The horror film Cloverfield, shown in Fig. 5.4, is a particularly interesting example of restricted narration. The entire film is composed of camera footages filmed by characters. The film begins with a clear exposition, endorsing the truthfulness of the footage resource—it is framed as a government SD video card. The official-looking writing tells us we are about to see video recovered from a camera found in Central Park. When the tape starts, showing the main characters in happy times in the bedroom of her apartment overlooking Central Park, its readout date of April plays the role of an omniscient opening title. In the course of the film, the digital readouts explicitly tell the viewers when the story events take place in the earlier phase of their love affair and when we are seeing the horror attack by a monster in May. In other words, although the footage looks fragmented in order to represent the truthfulness of recovered footage by the government, they are cohesively bound together into a conventional horror genre structure.

Fig. 5.4

Selected screenshots from Cloverfield (dir. Matt Reeves, 2008). All rights reserved

Due to the restricted narration throughout the film, the viewers know no more than the characters being attacked by a monster. But the viewers are first given glimpses of the main characters’ relationships and emotional responses to their peril and are able to build empathy drawing on the character-centered narrative developments. The film also gives more wide-ranging information about what is happening outside the characters’ immediate situation, by showing newspaper reports, radio bulletins, and TV coverage of action occurring elsewhere.

Another form of digital mediation draws on simulating the interactive experiences that we encounter each day, for example, social media on computer screen, smartphone interaction, and YouTube videos. As mentioned in the analysis of Profile, this form of digital mediation has the potential for triggering our empathetic attachment drawing on the metaphorical projection from the communication that we encounter within our daily social circle. Nevertheless, along the same lines, the affective intensity of digital mediation is built upon combining our familiarity of the media forms and the contextualization of these digital media properties within the narrative semantics of story contents. Often the affective impact substantially relies on the dramatically emotional storythemes that the characters are dealing with. In other words, digital frames might add a layer of emotional attachment to the overall emotional impact due to our daily experiences with these media platforms; nevertheless, the overall arch of story forms and contents remains the main trigger of empathetic, affective intensity.

This proposal can be best exemplified in recent films such as Redacted (2007). This war film is based on a real event that took place in Samara in 2006, when a group of young American soldiers raped and murdered a 14-year-old girl and killed her family. In an interview at the 45th New York Film Festival, the film director Brian de Palma explained that he was adopting an experimental method to tell the story by using footage he found on the Internet. However, in bringing these fragmented media stories together, he had to fictionalize and restructure that existing material, and it is this process that ultimately gives the film a classical, coherent narrative structure.

The film begins with a video diary recorded by Private Angel Salazar’s camcorder, which provides the main media source for the events depicted in the film. This is followed by several other mediated formats: a French documentary with voiceover narration, reports from Arab news channels, camera recordings played on Al Qaeda sites, embedded journalist reports, clips from ‘Soldiers’ Wives’ and the ‘Get Out of Iraq’ campaign websites, military surveillance cameras, recordings of military hearings, and so on. Pisters argues that in this film, “all these different formats and screens are entangled in complex ways and present different points of view of the same events” (Pisters 2010: 238). The dominant strategy of the film, namely, representing points of view, is used right in the beginning when the film opens with footages of soldier’s handheld camera.

As discussed above, these media devices are contextualized in the soldier’s war narratives drawing on the discourse of restricted narration—right in the beginning of this film, we are confined to see through the soldier’s eyes and are put to interpret truthful experiences of these American soldiers. The main character and his interaction with other soldiers are presented by him filming himself and his surroundings, shown in Fig. 5.5. This is the technique of mediated first-person point-of-view within restricted narration defined by Bordwell and Thompson (2010: Chapter 3)—although the character captures story events for the viewers, the footage nevertheless fit the premise of video recording to the demands of coherent, conventional narrative structures.

Fig. 5.5

Selected screenshots from Redacted (dir. Brian de Palma, 2007). All rights reserved

A decade later, several films start to use computer screen to represent first-person point-of-view. While several film critics celebrate the new way of filmmaking, nevertheless, the semantic strategies of restricted narration, for representing truthful message sources and emotional response drawing on characters’ narrative developments, remain similar to other forms of digital media frames used decades ago.

Searching, shown in Fig. 5.6, for instance, deals with a father who breaks into his missing daughter’s laptop to find out everything he possibly can about her sudden disappearance. The entire film takes place from the perspective of computer screens, which not only truthfully demonstrates the crime-solving potential of the Google search engine and social media information but also shows just how much of ourselves exist in online spaces. The film begins with several video clips and pictures shown directly from the main character’s computer. These videos and pictures linearly depict story background: the family Kim’s life and finally their loss of the mother/wife. The film beginning, like Redacted and Cloverfield, first anchors the viewer’s emotional attachment to the main characters by showing their intimate life experience and character features, before the traumatic events unfold.

Fig. 5.6

Selected screenshots from Searching (dir. Aneesh Chaganty, 2018). All rights reserved

As our example analyses show, film substantially using digital mediation often starts with endorsing the truthful media sources and lead to fragmented first-person point-of-view; nevertheless, these films still construct conventional and coherent narrative structures, providing sufficient background knowledge for the viewers to interpret the linear, straightforward story events, which evoke empathetic, emotional attachment to the main characters.

4 Final Remarks

To date, several film scholars have mixed the perception of media and narrative functions. This chapter pointed out the problems caused by the mixture of media properties and meaning interpretation process. It proposed to distinguish levels of mediation and semiotic representation drawing on the findings of cognitive and semiotic research. This chapter particularly uses the film technique of digital mediation to show the process of contextualizing mediated frames in order to realize semiotics-based narrative functions, above all, representation of message truthfulness and affective engagements, which have long been regarded as paradoxical.

Mediated images have been used in cinema for decades for enhancing the affective engagement and to endorse the first-person truthful narration. On the basis of the multi-leveled framework delineated in Fig. 5.1, digitally mediated frames is a presemiotic property—it can be contextualized in horror films to achieve the semiotic functions of affective intensity; it can also be semiotized in drama or war films for enhancing first-person truthful narration. As the film examples presented in this chapter, digital mediation in film can often interconnect the affective and truthful narrative functions. Moreover, in the last two decades, digital mediation has moved the mediated first-person point-of-view from portable camera to computer screen. As we could see in the above example, analyses, despite the change of media frames, the narrative functions of restricted narrative, representation of truthful message resources and straightforward character engagement remain dominant.

Nevertheless, the distinctive media affordances of computer screens used in film bring about other narrative functions which camera footage does not construct. While camera footage reflects a medium that bears narrative actions, computer screens simply bear narrative actions. Computer screens add a touch of immediacy to the first-person point-of-view. For instance, the character’s typing and deleting the message on the screen or dialing Skype calls, clicking on webpages, and so on are all part of immediate point of view. Essentially, the viewers are just watching words and dialing, clicking, browsing actions on a screen, but the actions feel relatable and human. In our real lives, we do not just passively watch computer screens like we watch camera footage; in our lives, we constantly interact with them as a means of communication which, in turn, has grown new forms of human behavior in our age. Computer screen-based, immediate point of view in narrative film depicts this new interactive format of ours and new affordance of media which has been explored in cinema today.

Nevertheless, the comparison between computer screen and camera footage holds only when we base the analysis on the multi-leveled semiotic framework—the function of enhancing immediacy in computer screens can be achieved only when particular semiotic structures are realized: for instance, when the character’s actions of clicking, typing directly on the computer screen are shown to the viewers within the first-person film frames. Simply showing a computer screen without co-patterning these event actions would not achieve the same effects. For instance, in the war film Redacted, analyzed in the previous section, several websites, YouTube videos, and Skype chats are also shown via computer screens (see Fig. 5.7). However, the semiotic structures of these scenes in Redacted do not reflect the character’s interaction with the media features of computer screens. Here, the computer screen is a platform similar to the intra-diegetic camerascreen which carries the mediated actions. This might still endorse the message truthfulness due to the digital frames the viewers are engaged in daily, but the first-person perspective is not realized in this kind of event construction.

Fig. 5.7

Selected screenshots of computer screen scenes from Redacted (dir. Brian de Palma, 2007). All rights reserved

In summary, this chapter has analyzed films made in the last two decades, entirely composed of digital mediated images. The analyses compared the different media affordances and narrative functions of camera footage and computer screens. While the narrative functions of truthfulness and affect remain stable, the affordances of computer screens add the immediacy of actions to the first person point of view in film. Through the analyses, I hope to have shown that subtle comparisons of semiotic representation, media perception, different types of affective response, and emotional engagements can be more effectively unraveled with a multi-leveled framework, which encompasses sufficient research synthesis across empirical findings of cognitive studies and semiotic theories.