Immersive Experience and Virtual Reality

Abstract

Much of the excitement about virtual reality and its potential for things like entertainment, art, education, and activism is its ability to generate experiences that are powerfully immersive. However, discussions of VR tend to invoke the notion of immersive experience without subjecting it to closer scrutiny; and discussions often take it for granted that immersive experience is a single unified phenomenon. Against this, we argue that there are four distinct types or aspects of immersive experience that should be distinguished: representational immersion, which corresponds roughly to what is sometimes called “psychological presence”; participatory immersion, which is related to the interactive aspects of VR experience; affective immersion, which has to do with the subject’s emotional relation to the experience; and narrative immersion, which captures the phenomenon of being caught up in the flow of events experienced. We argue that this four-way distinction helps us understand the powerfully immersive character of VR, while also recognizing its continuity with the immersion we experience with media such as novels, films, and music. We also argue that the account of immersion we offer here lets us better understand the connection between VR and empathy, charting a middle course between extreme enthusiasts who view VR as the ultimate “empathy machine” and extreme skeptics who argue that this attitude is misguided.

1 Introduction

Much of the excitement about the potential value of virtual reality not just for entertainment, but also for art, education, activism, and other applications is due to its power to generate uniquely immersive experiences. As the philosopher David J. Chalmers says,

What’s distinctive about VR is that its virtual worlds are immersive. Instead of showing you a two-dimensional screen, VR immerses you in a three-dimensional world you can see and hear as if you existed within it.¹

The psychologist Jeremy Bailenson also emphasizes the experience of being plunged into another world that VR creates:

One second you are strapping on an HMD [head-mounted display] and the next you are somewhere else. That sensation of “being there,” wherever the program you are running takes you, is what researchers call psychological presence, and it is the fundamental characteristic of VR.²

Characterizations like these are clearly metaphorical; even the term ‘immersion’ itself wears its metaphorical origin on its sleeve. There is nothing wrong with such metaphorical characterizations, of course, but their utility is limited when it comes to thinking systematically about VR and our engagement with it. They are in danger of running together several distinct elements that we think it would be useful to distinguish, and at the same time they miss other elements that, in our view, are important for understanding the immersive character of VR experience.

Our main goal in the first part of this essay is to bring out these different elements and distinguish them clearly. In Sections 2, 3, 4, and 5, we identify four distinct aspects of immersive experience that can be distinguished in our engagement with VR, providing illustrative examples of each in VR as well as in more traditional media such as films, novels, theater, and music. One benefit of explicitly acknowledging and articulating the four different kinds of immersion is that it can provide guidance to designers, artists, educators, activists, and others who hope to make effective use of the immersive potential of VR. VR offers a powerful technology for creating experiences, one that becomes more and more powerful with continuing improvements to things like processing power, rendering speed, and controller hardware. But as we will see, using VR to create experiences that are genuinely and richly immersive is much more a matter of art than of engineering. A second, more fundamental benefit of the account we offer here is that it makes it clear that the distinctive immersive power of VR does not lie in an ability to generate experiences that are immersive in some unique and otherwise unavailable sense. Rather, it lies in an ability to provide an especially potent combination of different forms of immersion that can also be experienced with other media as well.

Before we begin, we need to say a bit at the outset about the methodology employed here. One could approach the development of a systematic account of immersive experience in a highly empirical way. For example, Lee (2020) proposes an account of immersive experience that has some elements in common with the one we offer here and supports it on the grounds that it is the best way of synthesizing a large body of empirical research on immersive experience in psychology, game studies, media studies, and other fields. We do not take issue with this approach, of course, and in Sections 2, 3, 4, and 5 below we draw occasional connections between our discussion of the four kinds of immersive experience we identify and some of the very large body of empirical work on immersion. But our approach here is not primarily empirical. This is not to say that our account is intended to be an a priori conceptual analysis of the concept of immersion; we doubt that such a thing is feasible for such a fluid concept. Rather, the four-fold account developed here is offered in a broadly Carnapian spirit, as an explication of the notion of immersion at play in discussions of VR in philosophy, art, and other areas. In our view, the four elements we identify capture the core of this notion of immersion while also yielding a refined characterization that is more useful for systematic reflection about VR and our engagement with it. The account is intended to earn its keep by its utility, by helping us understand more clearly the aspects of VR and our engagement with it that the notion of ‘immersion’ is often invoked to explain, and by helping to resolve disagreements and debates to which the term sometimes gives rise.³

In this spirit, we argue in the second part of the essay (Section 7) that the account helps us think more clearly about the heavily disputed connection between immersion in VR and empathy. Some have argued that the immersive character of VR experience makes it a uniquely powerful tool for helping users empathize with people in very different circumstances, such as refugees or people with disabilities. Artist and filmmaker Chris Milk, for example, famously described VR as the “ultimate empathy machine.” Others forcefully reject this idea: psychologist Paul Bloom, for example, argues that the idea of VR as an empathy machine is confused and misguided. The multi-dimensional characterization of immersion we offer here lets us to chart a middle path between these two extremes. We argue that Bloom’s critique fails, because it depends on a conception of immersion that is too narrow and that overlooks some of the important aspects of immersion we highlight here. At the same time, we argue that much of the power of VR to generate empathy is not unique to VR, but instead is due to aspects of immersive experience it has in common with other forms of media, such as films and books.

2 Representational Immersion

In this and the next three sections we discuss four different elements that we think need to be distinguished in order to adequately understand the immersive character of our experience with VR: representational immersion, participatory immersion, affective immersion, and narrative immersion. Here we discuss them as four different kinds or types of immersive experience. However, one could instead choose to understand the term ‘immersive’ as a multi-dimensional adjective like ‘healthy’ or ‘beautiful’. One could then think of the four elements distinguished here as different dimensions that combine to yield an overall evaluation of the degree to which a given experience is immersive. While these four types clearly interact with and influence each other, we discuss them here individually to help make their different roles in immersive experience clear.

Representational immersion is the kind of immersion most directly invoked in the passage from Chalmers quoted in Section 1, which describes VR as placing one “in a three-dimensional world you can see and hear.” Seeing and hearing are perceptual experiences, conscious mental states that represent the subject’s surroundings as being a certain way – for example, as including certain objects and events – and that represent the subject herself as bearing certain relations to those objects and events. Representational immersion is tied to the ability of VR to generate or prompt states like this: a VR experience is representationally immersive when and to the extent that it involves a rich and coherent network of such representational mental states.⁴

For example, in VR a subject may have a succession of visual experiences that provide detailed representations of many aspects of a city scene: skyscrapers with mirrored windows that reflect the sky overhead, streets with cars and buses moving along them, a city park with trees swaying in the breeze, a construction crane swinging over a building site. The subject’s visual experiences also provide her with a specific point of view of the scene. She might perceive the city scene as though she is flying through it, with the street below her and the buildings passing by as she navigates between them. Because the HMD of a typical VR system is an occluding headset with binocular displays, the visual experiences it produces involve three-dimensional representations that fill the subject’s entire visual field. With sufficiently sophisticated technology, these visual representations can be extraordinarily detailed and lifelike. And they are accompanied with auditory experiences that further fill out the scene and the subject’s place in it, such as the sound of the traffic coming from below, the roar of the rocket that propels her through the air, and the wind rushing past her ears.

Visual and auditory perceptual experiences provide the most salient examples of how VR generates representational immersion. But representational immersion often involves more than just visual and auditory perception. For example, most contemporary VR systems have haptics in the hand controllers that are used to generate tactile perceptual representations. The VR program Bogo, for example, in which the user interacts with a small pet dragon, uses haptics to give the user the tactile experience as of the dragon purring when she places her (virtual) hand on its head or pets it. We sometimes work with our students in an archery simulation where the user holds the virtual bow using one hand controller, and then uses the other controller to draw back the bow string and release the arrow. After becoming accustomed to the simulation, students often report that they start to feel the tension in the bow as they pull back the bow string; this is achieved through the program’s effective use of the hand controller haptics in coordination with the subject’s visual and auditory perception of the motion of the bow string. Similar sorts of interactions can also be used to generate proprioceptive representations, such as the feeling that one is wobbling on a narrow virtual plank high above the ground, or the feeling that one is dropping in an elevator down a virtual mineshaft. This is why VR is often chosen as a medium for proprioceptive art that aims to manipulate the spectator’s perception of the orientation and motion of his or her own body. Fedorova (2013) attributes the effectiveness of VR for proprioceptive art to its ability to create “immersive virtual environments.” What she identifies here is what we are calling representational immersion.⁵

So far, we have been operating under the assumption that visual, auditory, and other perceptual experiences are representational mental states, as indeed the label “representational immersion” suggests. This accords with our own views, and we will continue to speak in these terms in what follows. But it should be noted that this is not an essential element of the idea of representational immersion. Theorists who endorse post-cognitivist, direct realist or other non-representational accounts of perception can acknowledge that perceptual experiences play a central role in immersive experience in VR, even if they resist characterizing those experiences in terms of mental representations.⁶ And whether perception is best understood in representational terms or not, representational immersion in VR should not be conflated with the idea that VR generates experiences that are illusory or inaccurate. For example, Chalmers (2017, 2022) proposes a view according to which the perceptual experiences of an experienced user of VR represent digital objects that really exist and digital events that really occur as having certain sorts of response-dependent properties that they in fact have; on this account there is nothing illusory about perception in VR. And Rolla et al. (2022) develop a non-representationalist view according to which perceptual experiences in VR are what they call allusions, as-if experiences that have the same kind of connection to action as ordinary perceptual experiences but do not incorrectly represent the user’s environment. To say that VR is representationally immersive is not to say that VR is a source of illusions.

Moreover, representational immersion as we understand it involves much more than perceptual experience. Just as important for representational immersion are the numerous sorts of non-perceptual, cognitive states that are generated within VR. For example, as part of the archery simulation mentioned above, the user acquires various kinds of information about the virtual world she is in: that the location where she is standing is the wall of a castle, that there is an army of invaders hiding among the hills she sees from her vantage point on the wall, that the invaders will try to penetrate the castle’s defenses through specific points of access, that those access points are reached by following paths that are partly obscured from her point of view, that hitting certain targets with her arrows will cause barrels to explode or hot tar to pour down on the invaders, and much else besides. Schubert et al. (2001) propose that mental representations of the different possible courses of action available to the user within the virtual environment are an important element in immersive experience. And Ratan (2012) proposes that immersion includes mental self-representations at different levels of understanding, from representations of one’s virtual body to representations of one’s identity within the simulation.

One can take different views about the exact the nature of the mental states that incorporate all these different kinds of information. They are surely not ordinary beliefs to the effect that this is how things really are (at least, assuming the subject is not deeply deceived, like Neo in The Matrix). But perhaps they are beliefs with a more complex representational content to the effect that things are thus-and-so according to the simulation. Perhaps they are states of imagining or supposing that this is how things are, or states akin to acceptance in the sense of Stalnaker (1984, 2002).⁷ Perhaps they are more like aliefs in the sense of Gendler (2008).⁸ Or perhaps some aspects of representational immersion are best understood in terms of mental models rather than discursive representational states.⁹ However they are best understood, mental representations like these are extremely important for representational immersion: they add depth to the subject’s immersive experience by filling out the virtual world that lies beyond her immediate perception of it.

If the intuitive idea of immersion is the metaphorical idea of VR transporting one into another world, representational immersion makes good on this metaphor by putting the subject into a state of mind that is very much like the one she would be in if she really were in that world: a representationally immersive experience provides a rich array of detailed mental representations of the subject’s virtual environment and her current circumstances in a way that is analogous to how she experiences the non-virtual world outside of VR.

VR is of course hardly alone here. Audiovisual media such as films and theater performances are unquestionably very often highly representationally immersive. When one watches a film, one enjoys a rich array of visual and auditory mental representations of the world of the film and the events that take place within it. Often these are representations from the perspective of a bystander or outside observer, but sometimes – as in the case of a film like Enter the Void directed by Gaspar Noé – they capture the world of the film as it is perceived by one of the characters of the story. Given that part of what one does when one engages with VR is something very much like watching a 3D film, the observation that films can create representational immersion is hardly unexpected.

What is perhaps less obvious is that our engagement with literary fiction is, in its own way, also often extremely representationally immersive. Balcerak Jackson and Langkau (2023) observe that our engagement with works of literary fiction paradigmatically involves what they call experiential imagination, which is imagination of what it is like to experience a certain situation in a sensory way. For example, here is a passage from Rachel Kushner’s novel The Mars Room:

Two years earlier, when I was arrested, I cried uncontrollably. My life was over and I knew it was over. It was my first night in jail and I kept hoping the dreamlike state of my situation would break, that I would wake up from it. I kept on not waking up into anything different from a piss-smelling mattress and slamming doors, shouting lunatics and alarms. The girl in the cell with me, who was not a lunatic, shook me roughly to get my attention. I looked up. She turned around and lifted her jail shirt to show me her low back tattoo, her tramp stamp. It said

Shut the Fuck Up

It worked on me. I stopped crying.

Reading this passage, you might imagine the metal clanging of jail cell doors and shouts echoing in the hallway, the sharp smell of the mattress, the feeling of being shaken, or the sight of the cellmate’s tattoo in the dim nighttime light of the cell. These are not perceptual experiences in the ordinary sense, but they are representational mental states that are in certain ways like perceptual experiences, in terms of both their representational content and their phenomenal character. It is partly because of its power to generate mental states like these that many of us were already familiar with the experience of becoming immersed in a novel or short story long before VR technology was invented.

3 Participatory Immersion

Representational immersion provides one way in which VR puts the subject into an alternate virtual world. Another way to be put into a virtual world is to become part of it, in the sense of being an active participant, an agent whose actions help determine what that virtual world is like, how events unfold in it, and how she experiences it. This is what we call participatory immersion. VR programs differ greatly, both in the specific kinds of participation they make possible, and in the degree to which they successfully achieve participatory immersion. But all competently designed VR experiences offer at least some degree of participatory immersion.¹⁰

The idea of participatory immersion obviously has something in common with the idea of being interactive. Interactivity, along with representational immersion, is probably what most people think of first when they reflect on the immersive character of VR experience, and it is certainly true that VR experiences such as playing video games are often highly interactive. But participatory immersion, as we conceive it, is a broader notion than interactivity. For example, some VR experiences are essentially three-dimensional narrative films, in which the subject experiences an unfolding story from a point of view embedded within the events of the story. The subject might experience herself standing in a field, for example, and look up to see a flying dragon swoop down from the sky and begin to glide in loops around her. There is very little about an experience like this that is interactive, in the usual sense: what happens in the story does not depend on the responses of the user, and the narrative unfolds in a pre-determined way that is completely independent of the user’s actions. But the experience still offers a significant degree of participatory immersion in our sense: it is up to the user to decide where to position herself within the scene and how to orient herself in that position; and it is up to her where to look at each moment, whether to stay in place or move somewhere else, and so on. This has a significant effect on the way the subject experiences the virtual world. The experience of looking up and seeing the dragon plummeting down towards you is dramatically different than the experience of looking straight ahead as the dragon suddenly appears out of your peripheral vision and loops around you.¹¹ This sort of participation is easy to overlook, but it is extremely important for immersion: it helps create in the subject the sense that she is part of a space in which she can freely move around and which she can experience from different vantage points, which encourages the experience of being an agent in a fleshed-out world rather than a patient passively receiving a stream of experiences.¹²

As for other media, it must be conceded that films rarely offer significant opportunity for participatory immersion (although there are occasionally cases such as Netflix’s brief experiment with “choose your own adventure”-style shows). However, theater is a medium that offers great opportunity for participatory immersion. For example, the Juggerknot Theatre Company in Miami staged a series of performances, “Miami Motel Stories,” in which audience members moved freely throughout the rooms of old de-commissioned hotels and motels, talking, dancing, drinking, and playing dominos with characters from different eras of the city’s history.¹³ There is a striking parallel between participating in the simulated reality of a theater performance like this – which, not surprisingly, is often billed as “immersive theater” – and participating in the simulated realities of VR.

Moreover, our engagement with literary fiction has a participatory immersive element that we think is far too easily overlooked. This may be because most literary fiction is not interactive, and there is a tendency to conflate participation and interaction. After observing that literary fiction usually prompts experiential imagination, Balcerak Jackson and Langkau (2023) emphasize that it always falls far short of determining what exactly the reader imagines and how she imagines it. They illustrate the point with a passage from Lewis Carroll’s Alice’s Adventures in Wonderland that begins as follows:

The door led right into a large kitchen, which was full of smoke from one end to the other: the Duchess was sitting on a three-legged stool in the middle, nursing a baby: the cook was leaning over the fire, stirring a large cauldron which seemed to be full of soup. “There’s certainly too much pepper in that soup!” Alice said to herself, as well as she could for sneezing. There was certainly too much of it in the air. Even the Duchess sneezed occasionally; and as for the baby, it was sneezing and howling alternately without a moment’s pause. The only two creatures in the kitchen, that did not sneeze, were the cook, and a large cat, which was lying on the hearth and grinning from ear to ear.

Reading this passage, one might experientially imagine all sorts of things about the scene: the sting of pepper in the nose, the odd visual appearance of the grinning cat, the piercing sound of a baby crying, the cooking fire glowing through the haze of smoke in the room, and much more besides. But it is up to the reader to decide for herself how many and which parts of all this to imagine. One usually does this quickly and automatically, with very little deliberate effort. Nevertheless, in doing so, the reader herself actively participates in the process of shaping her experience of the fictional world of the text, just as when one moves about and interacts in VR. Moreover, no matter how descriptive the text is, it inevitably leaves a lot of room for the reader to decide for herself what the precise character of the imagining will be. Suppose one imagines the look of cauldron over the fire, for example. Is it round like a witch’s cauldron, or does it have straight sides like a cooking pot? Is it suspended on a hook over the fire, or does it sit on an iron rack? What color is it? Can one see the soup bubbling up to the rim of the cauldron, or just steam rising from within? None of this is specified in the text; instead, the reader fills this in for herself in whatever ways, and in as much or as little detail, as she chooses. This is one reason why readers of a beloved literary fiction are so often critical of film adaptations. It is not just that the film often depicts things differently than the reader imagined when she read the novel. It is that the film takes away the reader’s active participation: watching the film, the viewer passively absorbs the result of the filmmakers’ decisions about how to imagine the world of the novel, whereas the text invites the reader to use her own imagination to actively help create the world of the novel herself.¹⁴

One could argue that VR is much more immersive in the participatory sense than our engagement with literary fiction and films because its interactive nature lets the user play a direct causal role in determining what happens in the virtual world. Of course, not all VR experiences are interactive; three-dimensional films are a case in point. But many are, and as technology develops it may eventually be possible for VR to facilitate levels of participatory immersion that approach that of a live interactive theater performance. Even if VR has the potential to offer a greater degree of participatory immersion than literary fiction and film, however, what we want to emphasize here is simply that participatory immersion is not something that is exclusive to VR.

4 Affective Immersion

Our engagement with the world of real life is not limited to input in the form of sensory mental representations and output in the form of participatory action. We also engage with the world in emotional or, more broadly, in affective terms: we are excited or scared, happy or sad, amused or worried, impatient or content, and joyful or angry about the things that are happening around us and to us. This is a matter of what we call affective immersion, the experience of being emotionally or affectively engaged with what is happening in VR. Affective immersion is a powerful contributing factor to the sense one has of being drawn into another world in VR. The classic plank simulation, in which the subject balances on a virtual narrow wooden plank high above the ground, is so effective not just because it provides a realistic representation of what it looks like to be standing on such a plank, but because the subject often cannot help but feel at least some of the fear and anxiety that goes along with being in that situation. It has often been observed, and we have seen it ourselves in our work with our students, that these feelings are sometimes so powerful that the subject cannot bring himself to step out onto the plank, even though he is fully aware that it is just a stimulation.¹⁵

As with the other kinds of immersion, VR experiences differ greatly in the both the extent to which they are immersive in the affective sense, and in the specific kinds of affective immersion they achieve. A VR simulation for job training at a fast-food restaurant might not be very affectively immersive for most subjects, no matter how realistic it looks and sounds. Nor is a three-dimensional painting and drawing program like Tiltbrush, even though it is extremely participatory. Some VR experiences, such as the Face Your Fears series, focus on affective immersion along the fear-anxiety spectrum, using jump scares, iconic figures from childhood nightmares, and confrontations with phobia triggers such as spiders. Other VR experiences endeavor to have the subject experience joy or wonder. Many VR experiences offer a complicated mixture of affective responses that are not easily categorized. In the puzzle game A Fisherman’s Tale, for example, the subject plays the role of a tiny wooden fisherman puppet living on a little wooden boat. The subject experiences a sense of foreboding – there is a dark storm brewing on the sea outside the window – as well as curiosity and confusion about what to do to escape danger. As the game goes on, the subject discovers that the little wooden boat contains a smaller wooden model of itself and everything in it, and that she can change things in the boat around her by manipulating the corresponding elements in the model. The player eventually discovers that the original wooden boat in which she started is itself playing the role of the little wooden model inside a bigger boat, that this bigger boat is the little wooden model inside a still bigger boat, and so on; the player herself exists and acts simultaneously in each level of an endless series of wooden boats nested like Russian dolls. The resulting feeling is a kind of “existential vertigo” that is difficult to describe in simpler affective terms, and it contributes to a powerfully immersive experience that goes far beyond the representational immersion provided by the (slightly cartoonish) look and sound of the game.

If we want to fully understand the ability of VR to create the experience of being immersed in another world, then, we need to acknowledge that it is partly a matter of being emotionally and affectively drawn into that world and moved by the things that happen in it. Clearly, however, other media offer just as much potential for affective immersion as VR, both because our emotional responses to things we experience in VR are very much like our emotional responses to novels and movies, and because in both cases those emotional responses are often closely tied to our engagement with the narrative of the work. We cheer for the heroes, fear and despise the villains, and hope for the couple in the romantic comedy to get back together. Music can be especially affectively immersive because of the potent way it combines our emotional responses to the moods of the music and the story it tells with our affective responses to the energy and beauty of the music itself.¹⁶

There is well-known disagreement about how best to understand our affective responses to literary fictions, films, and other traditional media. We may describe ourselves as being afraid of the Demogorgons in the Stranger Things series or being deeply saddened by the fates of the cloned children in Kazuo Ishiguro’s novel Never Let Me Go. But are these descriptions literally true, or are they better understood in a metaphorical or “as-if” way? Do we experience genuine emotions or merely simulated or pretend versions of them?¹⁷ We don’t need to try to settle that debate here; what matters for our purposes is just that it is to be resolved in the same way for our engagement with VR as for other media. It is deeply implausible that we merely simulate fear when we encounter a Demogorgon watching Stranger Things but experience the real thing when we put on a headset and encounter a Demogorgon in a Stranger Things-inspired VR experience (or vice versa). Whatever turns out to be the best way of understanding the psychological responses involved in affective immersion, traditional media and VR are both clearly equally capable of producing them.

5 Narrative Immersion

The grandparents of one of the authors were travelers, and the grandfather was an enthusiastic hobby photographer who took many photos during their travels. After each trip, he would host a lengthy slideshow where he shared with the family the photos of all the people, places, and things he had documented. While these slideshows often provided a highly detailed and informative representation of their travels, they could hardly have been described as immersive. (Certainly not for the young grandchildren waiting for the end of the slideshow to have cake and ice cream.) This was partly because of the passive nature of the viewing experience, which offered very little opportunity for participatory immersion. But it was also because the slideshow took the story of the trip and reduced it to a set of snapshots. What the travelers had experienced as a coherent flow of events involving different characters in changing settings, the slideshow viewers experienced as a disjointed series of frozen images. This is a matter of what we call narrative immersion: a VR experience is narratively immersive when and to the extent that the subject’s experience in VR has a coherent narrative structure and flow.

It is difficult to give a precise characterization of what constitutes narrative structure and flow, just as it is difficult to define what a story is. Many VR games have a conventional narrative structure, with a plot made up of events that follow some kind of coherent temporal or causal order, a setting in which the plot unfolds, and characters (if only just the player herself and enemy hordes to battle). Some VR experiences, such as the Bogo pet dragon simulator described above, offer a minimal plot and place much more emphasis on characters. And a sandbox game like those in the Legend of Zelda series achieves much of its immersive effect by creating an expansive and intricately detailed setting in which the user can wander and explore freely, and in which many different plots with different characters can unfold. Ultimately, there are as many different ways of creating narratively immersive experiences in VR as there are different kinds of narratives. What is important to emphasize is that in very many cases of immersive VR experience, the experience of being carried away into another world is as much a matter of getting drawn into the story of the world as it is a matter of being given a convincing representation of it.¹⁸

Traditional creative media such as literary fiction and film are of course no less immersive than VR in the narrative sense. After all, literary fiction and films are primarily used to tell stories, and most of us have had the experience of being swept up in the story of a good novel or a compelling television series. Ryan (2001) even suggests that we can understand our engagement with literary fiction as itself being a form of narrative immersion in virtual reality. Music is often used to tell stories, too, whether directly through the lyrics of a rap song or more abstractly through the movements of a symphony. Nothing about the elements of narrative – character, setting, plot, conflict and resolution, and so on – ties them to any specific medium, and when VR succeeds in generating narrative immersion, it is doing the same thing that is done by more traditional ways of effectively telling a story.

This concludes our initial survey of the four types of immersion, representational, participatory, affective, and narrative. One benefit of explicitly acknowledging and articulating the four different kinds is that it can provide guidance, and material for reflection, to designers, artists, educators, activists, and others who hope to make full and effective use of the immersive potential of VR. Making a VR experience that is extremely realistic or lifelike can enhance its representationally immersive character, but this only takes one part of the way. An effective designer also needs to consider how many and what kinds of opportunities there are for the subject to participate in and help shape the VR experience for him- or herself. She needs to consider how the subject will affectively engage with the experience, and to decide what kinds of affective engagement she wants the experience to create. And she needs to consider how to fold the subject into the narrative structure of the experience. VR offers a powerful technology for creating experiences, one that becomes more and more powerful with continuing improvements to things like processing power, rendering speed, and controller hardware. But using this technology to create experiences that are genuinely and richly immersive is no less of an artistic endeavor than creating a film or writing a novel.

A second benefit of the account here is that it lets us do justice to the powerful sense many have that there is something uniquely immersive about VR experience, while at the same time showing the continuities between immersion in VR and in other media. We have emphasized that none of the different kinds of immersion identified here are unique to VR and that can all be found to some degree or other in our engagement with films, literary fictions, theatre performances, and other traditional media. This should not be understood as taking anything away from VR’s potential for immersion. A well-designed VR experience can be at least as representationally immersive as any film. It can be as narratively and affectively immersive as a well-written literary fiction, and it can – in principle, at least – offer participatory immersion that is comparable to that of a live interactive theater performance. VR does not generate experiences that are immersive in some unique and otherwise unavailable sense. Its distinctive immersive power lies rather in its ability to provide an especially potent combination of all four aspects or immersion together.

6 Other Accounts

There are some points of overlap between the account of immersion offered in Sections 2, 3, 4, and 5 and other well-known discussions in the literature, as well as points of disagreement. In this section we briefly compare the characterization of VR immersion offered here with Thierren (2014) and Ermi and Mäyrä (2011). Our primary aim is not to criticize these other accounts, but rather to use the comparisons to bring out the contours of our own account more clearly.¹⁹

While Thierren (2014) does not advance a systematic account of immersion, it does survey several different ideas that can be used to help make sense of immersive experience in VR. Thierren highlights three key ideas, in particular, that provide useful points of comparison to the kinds of immersion identified here.

The first key idea is that of immersion as perceptual illusion, which Thierren describes as a primarily perceptual phenomenon linked to the “vividness and credibility of the represented reality,” along with a tendency to be pulled in and experience a “diminishing critical distance” from what one is perceiving (p. 959). Thierren even considers the possibility (suggested in Lombard and Ditton (1997)) of objectively measuring of the immersiveness of a VR experience in terms of the number of senses to which input is given and the extent to which those inputs shut out the real world. The idea of immersion as perceptual illusion clearly resonates with the notion of representational immersion discussed in Section 2. However, representational immersion in our sense is not at all limited to perceptual experiences. It includes representational mental states of various kinds, such as states of imagining or supposing, states of representing what is the case according to the VR simulation being experienced, mental models of the user’s virtual surroundings, and perhaps even genuine beliefs about the properties of virtual objects and events.²⁰ Moreover, it is not built into our notion of representational immersion that the subject’s mental representations are illusory or inaccurate, as we emphasized in Section 2. In our view, VR can be highly representationally immersive even if it involves no perceptual illusions at all.

Thierren’s second key idea is that of immersion as psychological engagement, which is a matter of being engrossed in a particular state of mind, of concentrating one’s mental resources on a specific activity to the exclusion of others.²¹ Thierren cites Csikszentmihalyi (1975), who identifies a kind of “flow state” one can enter during an activity such as playing chess or rock climbing, in which one is hyper-focused on certain aspects of the situation or the environment. We certainly agree that one can become engrossed or engaged in this way during a VR experience, as one can when reading a good book or watching a well-crafted horror film. But this can also happen when one is working on a difficult crossword puzzle or writing a computer program, or (as Thierren notes) while playing chess or rock climbing. For this reason, we find the notion of psychological engagement to be too broad to be helpful for articulating the immersive character of VR.

Thierren’s third key idea is that of immersion as visiting another world. Talk of “visiting another world” is metaphorical, of course, and indeed it is quite close to the metaphorical characterizations of immersion with which we began in Section 1. However, Thierren unpacks this metaphor somewhat by describing immersion in this sense as a sense of transportation the subject experiences when imaginatively engaging with a work that effectively portrays a “narrative world” (p. 950). We agree that this idea captures something important about VR immersion. In our view, however, it blends together several of the elements that we have sought to distinguish above. Imaginative engagement plays a role in representational immersion insofar as it contributes to the subject’s mental representations of the world of the work – visualizing a scene described on the page, for example, or anticipating the next moves of a non-player character in a VR game. We have also argued that imagination plays a role in participatory immersion insofar as it is an expression of the subject’s deliberate agency that helps shape the world of the work and her experience of it. And we have argued that the experience of being caught up in the narrative flow of a work is a further aspect of immersive experience, one that can be distinguished from the representational and participatory aspects. From our perspective, Thierren’s experience of visiting another world is something that is produced by the interaction of all three of these kinds of immersion (as well as with affective immersion, in most cases).

We turn now to the account of Ermi and Mäyrä (2011).²² Like us, they distinguish different kinds or aspects of immersion. Their SCI Model identifies three: sensory immersion, challenge-based immersion, and imaginative immersion. Sensory immersion, like Thierren’s idea of immersion as perceptual illusion, is a matter of the vividness of the subject’s perceptual experiences of the work and the extent to which they overpower her perceptual engagement with the real world.²³ As we have already seen, sensory immersion in this sense is one important aspect of representational immersion, but there is much more to representational immersion than this.

Challenge-based immersion, according to Ermi and Mäyrä, is “fundamentally based on interaction,” in that it requires the subject to play an active causal role in determining the sequence of events she experiences (p. 101). Interactivity is an important element of what we have identified as participatory immersion. However, in Section 3 we emphasized that participatory immersion is a broader notion: it is not merely a matter of having causal influence on what happens in the virtual world, but of having agency concerning how one experiences and engages with it. Causal influence by itself may not be very immersive; imagine watching a film whose plot has been determined by your responses to a questionnaire provided in advance. At the same time, a virtual 3D film in which you are free to shift your point of view anywhere you like can be quite immersive, even if you have no causal impact on how events unfold. Another difference is that Ermi’s and Mäyrä’s notion, unlike our notion of participatory immersion, involves the idea of presenting the subject with a challenge, some set of goals for the subject to achieve. Effective challenge-based immersion requires a challenge that is well calibrated to the subject’s skill level: it cannot be too hard for the subject to bring about the effects she wants – finding her way out of the maze, destroying the level boss – but it also cannot be too easy. This makes good sense in the context of games, but it is clearly not intended to apply to VR more broadly, and we do not see it as an essential part of immersive experience in general.

We turn finally to the third element of Ermi’s and Mäyrä’s account, imaginative immersion. Ermi and Mäyrä understand this kind of immersion both as a matter of exercising one’s imagination to visualize scenes, fill in unseen details and background information, attribute thoughts and motives, and so on, and also as a matter of getting drawn into the “characters and story-like elements” of the experience. It thus very closely related to Thierren’s idea of immersion as visiting another world.²⁴ As we already indicated, we agree that this identifies something important about the character of immersive experience in VR. In our view, however, it is best understood as something that emerges through the interaction of the different kinds of immersion identified here, rather than as a distinct kind or aspect of immersion on its own.

7 Immersion and Empathy in VR

In 2011, artist and filmmaker Chris Milk gave a TED Talk with the title, “How Virtual Reality Can Create the Ultimate Empathy Machine,” in which he argued that VR has a unique power for creating empathy for others because it transports the user into the world of someone else, rather than merely showing them a glimpse of that world through the “rectangular window” of traditional film.²⁵ Since then, the term “empathy machine” as a label for VR has become ubiquitous, as has the idea that the immersive character of VR makes it uniquely well suited for creating or enhancing empathy.²⁶ Milk’s VR experience Clouds over Sidra, for example, won awards for its portrayal of a young Syrian refugee named Sidra who had escaped with her family to a camp in Jordan. Others have developed VR experiences to help create empathy for what it is like to experience anti-black racism as one grows from young child to adult, for what it is like to come to the US from Latin America as an undocumented migrant, for navigating day-to-day life in a wheelchair, for the experience of life as a school child with autism, for losing one’s home and living as a homeless person, for what it is like to suffer long-term imprisonment in solitary confinement, and much, much more. The French government has recently been experimenting with the use of VR to help reform men convicted of domestic violence by making them more empathetic with the perspectives and experiences of victims of domestic violence.²⁷ Experiments have also been done to see if standard measures of implicit bias can be reduced in white subjects by placing them in VR experiences designed to create empathy for what it is like to be Black.²⁸

At the same time, the enthusiasm about the potential of VR for creating empathy expressed by many has been met with deep skepticism by others. Among the most prominent skeptics is the psychologist Paul Bloom – an expert on the psychology of empathy – who argues that it is “dangerously misleading” to claim that VR can “help you appreciate what it is like to be a refugee, homeless, or disabled.”²⁹ What are we to make of this disagreement?

The first step is to clarify what is meant by “empathy.” A distinction is often drawn between two types of empathy, affective and cognitive. Very roughly, to experience affective empathy for a person is to experience certain emotions because one takes the person herself to be experiencing those emotions, or to be experiencing corresponding emotions.³⁰ Having cognitive empathy for a person, by contrast, is a matter of imaginatively adopting the person’s perspective in order to understand what the person is feeling and what might be making them feel that way. We take genuine empathy to have both affective and cognitive components: to have empathy for someone is to take on their feelings and emotions (or corresponding ones), at least to a certain extent, and also to understand those feelings and emotions and be in a position to reflect on their underlying rationale.³¹ There is of course much more to be said about the nature of empathy, but this brief characterization will suffice for the purposes of the present discussion.

The second step is to clearly distinguish questions about the connection between VR and empathy from several other questions that we will not attempt to address here. Some of these are questions about the role of empathy in addressing issues such as domestic violence and racism. If exposure to VR simulations of what it is like to be Black fail to lead to a reduction in anti-Black implicit bias measures, for example, this may be because VR is not a very effective way to create empathy. Or it may be because empathy does not have very much to do with the mechanisms that underlie implicit bias. Similarly, it may be that VR is a highly effective way to help convicted domestic abusers experience empathy for their victims, even if it turns out that this doesn’t help prevent future offenses because more empathy doesn’t lead to better impulse control or more effective rage management. At least some of the pessimism that is expressed about VR as an empathy machine is probably better understood as pessimism about the potential for using VR to help cure social and other ills, or to help make us kinder, more caring people.³² Questions about potential links between VR and empathy also need to be clearly distinguished from questions about the extent to which empathy is an effective form of moral motivation that ought to be encouraged.³³ And even if empathy ought to be encouraged, there are difficult questions about whether VR is an appropriate way to do so, or whether it amounts to a form of “misery tourism” that disrespects or exploits the subjects for whom it seeks to engender empathy. We don’t intend to speculate about those questions here. Our focus is on the connection between VR and empathy itself, rather than on any potential further uses to which such a connection – if it exists – might be or ought to be put.

With those clarifications in place, we think that the more nuanced understanding of immersion that we have been developing here can help chart a middle path between the extreme enthusiasts, who see VR as a uniquely powerful way of creating empathy, and the extreme skeptics, who regard the enthusiasts as misguided and the connection between VR and empathy as greatly overblown.

In his critique of VR as an empathy machine, Bloom (2017a) argues that trying to use VR to create empathy for the experiences of refugees or homeless people can be “dangerously misleading,” because, while VR can immerse one in the physical surroundings of a refugee or a homeless person, those physical surroundings fail to capture what is important about the refugee’s or homeless person’s experiences. Referring to Clouds over Sidra, Bloom writes:

The awfulness of the refugee experience isn’t about the sights and sounds of a refugee camp; it has more to do with the fear and anxiety of having to escape your country and relocate yourself in a strange land.

This is certainly true. But Broom’s critique needs to be weighed against two important points. The first is that, while an audiovisual representation of a refugee camp clearly will not automatically cause one to feel the way a refugee feels, it is just as clear that experiencing the sights and sounds of a refugee camp from a first-person perspective can help one understand some of what a person living in such a camp might feel and why. Moreover, it can help one form an understanding that is more nuanced and complex than one might achieve otherwise. It is easy to understand that a refugee is likely to feel sad about losing their home. Upon experiencing the corrugated tin walls and harsh fluorescent lighting of the camp dwellings, however, one gets a much more detailed representation of the refugee’s situation, and this can help enable a much richer understanding of the refugee’s emotional state. One might come to understand the refugee not simply as feeling sad about losing their home, but as experiencing pain and disorientation at losing any sense of home at all; of feeling like one is sitting in storage waiting to be taken out, rather than living a life. Such an understanding would certainly be possible without the VR experience, and the VR experience is not guaranteed to deliver it. But VR offers an efficient way to make the possibility of such an understanding available to a broad range of people.

The second and more important point is that Broom’s critique considers only the representationally immersive aspect of VR experience, on its ability to give the subject the experience of how it would look and sound to be in a refugee camp. Broom’s critique doesn’t take the other kinds or aspects of immersion into account at all. Representational immersion is certainly an important part of what makes VR immersive, as we have seen, and it can contribute to a VR experience’s power to create empathy. But focusing exclusively on representational immersion is a mistake. Clouds over Sidra uses a 360-degree film to provide a representationally immersive experience of what a day in Sidra’s life in the camp is like. But it interweaves this with other elements, including, most prominently, Sidra’s own words as she tells viewers about her experiences and shares her observations, thoughts, and feelings. These contribute both narrative and affective aspects to the immersive experience of the film and these aspects contribute greatly to the power of the film to induce empathy for Sidra and other people in situations like hers.

Other VR experiences that seek to evoke empathy make even more intensive use of the other kinds of immersion. For example, The Key, by Celine Tricart, endeavors to capture the experience of refugees by taking a very different approach than Sidra. Using animation rather than 360-degree film, it involves the user in a highly allegorical fictional story that is clearly intended to evoke the kinds of feelings of fear and anxiety that Bloom identifies as important for empathizing with the refugee’s experience. But it does so without making any effort to accurately represent the sights and sounds that refugees experience, placing much more emphasis on the affective and especially on the narrative aspects of immersion. The Key also makes effective use of the participatory immersion of VR to evoke feelings of helplessness and loss, as the user repeatedly tries and inevitably fails to hold on to things that are literally being pulled from one’s grip and taken away. Another VR experience, Traveling while Black, by Roger Ross Williams, lets viewers sit in a virtual booth in Ben’s Chili Bowl, a historic restaurant in Washington D.C., as different people sit across the table and share experiences of what it was like to live and travel as a Black person during the Jim Crow era, and talk about what havens like Ben’s Chili Bowl meant to them. Here there is no attempt to use VR to try to recreate in the viewer the experience of living as a Black person under segregation – something Bloom would surely be right to be skeptical about the power of VR to achieve. Rather, VR is used to immerse the viewer in the situation of sitting face-to-face with someone as they tell you stories about their life – a highly affectively immersive experience that might evoke one’s natural capacity for empathy in much the same way a similar situation would in real life. Examples like these illustrate that the other kinds of immersion are just as important as representational immersion for assessing the potential of VR to create empathy. It is a mistake to assume, as Bloom’s critique implicitly does, that the power of VR to create empathy is limited to its capacity for representational immersion.

This point can be underscored by returning to the continuity between immersive experience in VR and immersion in other media. There can be no reasonable doubt that literary fiction, film, and other traditional media can be extremely effective tools for creating empathy, and it would clearly be a mistake to focus solely on representational immersion in these cases. It would be absurd, for example, to argue that Alex Haley’s novel Roots is dangerously misleading because its descriptions of the slave traders’ ships and slaves’ living quarters do not adequately capture the experience of being treated as a slave in North American history. Obviously, there is much more to the novel than such descriptions. The ability of such media to create empathy exploits not just their potential to be representationally immersive, but also to be immersive in the different ways we identified in Sections 2, 3, 4, and 5. Films like Philadelphia (directed by Jonathan Demme) and Dallas Buyers Club (directed by Jean-Marc Vallée), for example, create empathy for people coping with HIV/AIDS by providing richly detailed, representationally immersive portrayals of what life is like in their shoes, but also by telling compelling, narratively immersive stories with the support of strong, affectively immersive performances by their casts. Given that VR can be immersive in exactly these ways as well, there is no reason to think that VR cannot be just as effective at creating empathy, at least when it is wielded with the same kind of thoughtfulness, creativity, and artistic talent. It can be no more ridiculous to use VR to empathize with refugees than it is to use a short story or a novel, or a film or television series, or a play or an opera. Indeed, since the medium of VR allows us to generate high degrees of all four kinds of immersion and let them work in concert, we have every reason to think that a thoughtfully designed VR experience can be an especially powerful way of creating empathy.

At the same time, the continuity between VR and other media makes it clear that Bloom and other critics are right to resist the idea that VR is a “magic bullet.” Artists such as Milk who are frustrated by the limits of traditional film understandably look to VR as a new and potentially more powerful way to generate empathy. But much of what makes VR effective for creating empathy is what it has in common with film and other traditional media. One cannot reasonably expect to create empathy for a person or group simply by creating a plausible VR simulation of their circumstances, not even a simulation that does a good job of capturing how those circumstances appear from their own point of view. Nor do improvements in VR technology lead automatically to VR that generates empathy more effectively. One needs to do much of the same work one would do in writing a novel or making a film or staging a play, such as constructing an engaging narrative and finding ways to create emotional resonance.³⁴ For creators who are interested in making effective use of the potential of VR, it is worth paying particular attention to how one might take advantage of participatory immersion, because one of the things that makes VR stand out most strongly from traditional media is the richer and more flexible range of opportunities it provides for the subject to actively participate in the experience.

8 Conclusion

We have argued that the immersive character of VR cannot be fully understood just in terms of its ability to create compelling representations of how it looks and sounds to be in another world, which is a matter of what we have called representational immersion. Immersion in VR is also participatory, as the subject herself is actively involved in shaping what happens in the world and how her experience of it unfolds. It is affective, as the subject engages emotionally with the people and events she encounters in VR. And it is narrative, as the subject gets drawn into the story of the experience. None of these kinds of immersion are unique to VR. They can all be found in traditional low-tech media, and the immersion one experiences in VR is continuous with the experience one has of becoming immersed in a good novel or gripped by a good suspense film. What is unique about VR is the technology it provides for flexibly combining high degrees of all four kinds of immersion.

Drawing on these observations, we have also argued for an analogous attitude towards the connection between VR and empathy. VR is not unique as a tool for creating empathy, and the ways it can be used to create empathy are continuous with the ways empathy is created by traditional low-tech media such as literary fiction and film. At the same time, the uniquely powerful combination of the four types of immersion that VR enables means that there is enormous potential for VR experiences to create empathy, at least when they are designed with the same kind of art and thoughtfulness as the literary fictions and films that already have the power to move us.