Embodiment in Virtual Reality Performance

Abstract

In this publication, we present artistic and technical developments in creating and presenting dance performances in media art, where embodiment is crucial in the artistic process. We study and compare three distinct performances with dancers and one choreographer in the same dance company between 2009 and 2020. The degree of immersion in performance is then compared between the three pieces, created at Balleteatro, in Porto, addressing the transition from the real to the virtual in the performance perspective, with practical cases and direct observations in the way the audience learns different states of body representation through technological means. We initially present the NUVE performance (2010), interpreted by Né Barros, Co:Lateral (2016–2019), a performance that crosses different realities, and, finally, the transition to the UNA work (2020) that takes place in total virtuality. This publication focuses on the developments, the public experience, and the results obtained in more than 20 exhibitions in different locations, either in theater or auditorium and in conference venues.

1 Introduction

Balleteatro¹, founded in 1983 by Isabel Barros, Jorge Levi, and Né Barros with the designation of Ballet Teatro Contemporâneo do Porto, is a performing arts center based in Porto, Portugal (Fig. 1). It is currently composed of an auditorium, a company, a professional school, a training center, and an audiovisual and editing center, currently located in the Coliseu do Porto. Since its beginning, Balleteatro has had the mission of being a center for the development of the performing arts. Having inhabited several spaces in Porto, the Balleteatro is building an artistic community for the contemporary performing arts that until then had never existed. Having as a fundamental area the creation, it generated the first contemporary dance company in Porto, and presents today a vast repertoire. The continuity of this profound work, of contamination between creation and training, allowed that in 1989 the Balleteatro had seen approved its candidacy as a professional school of theatre and dance, the first in the country with these possibilities. From this school, several generations of artists representing the performing arts were formed.

Fig. 1.

The Balleteatro. Ballet Teatro Contemporâneo do Porto. Picture: José Caldeira.

In 2009 João Martinho Moura² joined Balleteatro as a media artist and researcher, working closely with choreographer Né Barros³ in the frontier between dance and digital art. Since then, both developed different works, presented in various venues, having the embodiment process in performance in common, exploring further possibilities in the confrontation between the body and virtual environments. In the last ten years, they collaborated. The digital body was thought, and many tests were carried out using different capture and presentation technologies to the general public. This publication will make a straightforward approach to performance and Virtual Reality (VR). It will focus on the developments, public presentation, and embodied comparisons between the three works carried out with an approximate interval of 5 years.

1.1 Performance and Media Art

Among the many intersections, digital art is an expanding domain in dance. Digital Dance refers to dance-making processes and performances that include digital technologies as an integral feature. These projects often involved close collaboration between dance artists and technology specialists, usually artists themselves [1]. Though the appropriation of new technologies can be found in many relevant examples in dance, it is a field with the potential to explore the aesthetic and artistic level [2]. Several contemporary dance companies and artists experiment with new ways to present choreographies and movement using the digital [3]. In a digital performance, where technologies play a relevant role, these can be seen not as tools but as filters for our meetings with other people or us [4]. The integration of VR and interactive technology in dance performances leads to new insights and experiments with choreographic methods that may ultimately take dance in a new direction [5]. Brooks, a pioneer artist in using different performance interfaces, says that many of these artists used the body as a central element of the canvas, often abusing or decorating the flesh and provoking audiences – bringing them into the action [6]. Brooks emphasizes the future interface’s concept: the free air space around one’s own body, tailored and controllable, maybe from the mind, or by the skin, as in his Virtual Interactive Space (VIS) concept focused on rehabilitation [7]. For Saltz, performance, such as dance and theater, is a visual and auditory event, but, above all, corporeal [8]. For Dixon and Kozel, performance translates into an emerging state, a deep interconnection [4, 9]. For Fred Forest, art maintains close relationships with reality and seeks to use its influence to modify its properties [10]. Contemporary art sets the scenery for a body exploration based on movements, actions, and behaviors.

1.2 Virtual Reality and Embodiment

In the 1960s, Morton Heilig created the Sensorama device, a machine that is one of the earliest known examples of technology [11]. Ivan Sutherland, in paper entitled ‘The ultimate display’, written in 1965, reinforces the idea of the display connected to a digital computer that gives us the chance to become familiar with concepts not feasible in the physical world, describing a room within which the computer can control the existence of matter [12, 13]. In the 1980s, Scott Fisher, founder and director of NASA’s Virtual Environment Workstation Project (VIEW), worked on prototypes to help users, such as pilots, make better estimates of spatial relationships on 2D monitors and developed specific capacitor monitors (HMDs) at the Ames Research Center [14].

Artist and researcher Jaron Lanier claim to have used the term Virtual Reality (VR) for the first time, in the 1980s, during a period of intense creative activity, which narrates how to enter a new world [15]. He described it as a computer-simulated environment with, and within, people interact, classifying it as one of the greatest scientific, philosophical, and technological frontiers of our era [16]. Of the definitions found in the several decades, VR, generally, can consist of three types of systems: virtual environments presented on the screen, environments based on multi-projection rooms (CAVE systems⁴) and visualization devices placed on the heads of people designated as Head Mounted Displays (HMDs) [17,18,19,20,21,22].

In the last twenty years, and with the evolution of technology, immersive vision devices with sufficient resolution have emerged so that we can visualize generated worlds, and we will probably witness a revolution in human interaction with virtual reality technology and the environment in the next decade [23]. However, the foundations of this technology are more than 50 years old. There are references in 1938 to the term “La realité virtuelle” by Antonin Artaud in theater contexts [24]. This emerging medium’s impact is now beginning to be felt more in society in general, since in recent years, computational processing power to render binocular images with resolution and acceptable refresh rate to human beings, without causing nausea, become more possible.

The notion of presence is relevant as a mediating variable between experience and induced emotions [25, 26]. Precisely, what distinguishes VR from other mediums and gives it this status as such, is a sense of presence: the feeling of “being there”, within the virtual experience produced by the artifact [17]. The feeling of presence, associated with the high level of emotional involvement, allowed by virtual experiences, makes VR technology a powerful tool for exploring what is possible to imagine, supporting personal and clinical changes [27]. In this sense, this “new” medium can be considered quite distinct from other mediums, such as video, and in this distinction comes the concept of teleportation, a transition to a remote location [28]. Thus, these devices allow us to consider the hypothesis that mediated perception (with a tool) and non-mediated perception (with a sensory organ) follow similar mechanisms of appropriation [29]. In the following section, we present three different dance pieces involving digital technologies, been developed over the past ten years.

2 Three Performances Exploring the Concept of Embodiment in Virtuality

The three pieces we will explore have something relevant in common for our analysis: they were created by a group of people who worked together, that is, by the same team. However, the three pieces were made at different times and used different technologies that have evolved and are related to varying degrees of immersion inside the performance. We also highlight a growing approach to the immaterial body in a constant virtuality progression in these three pieces. In this way, it makes sense to group the three works in this essay.

The relationship between materiality and immateriality mediated by technology, from the interface, is very well explained by Milgram’s research, which proposes the concept of “bilateral continuity” between several layers of abstraction between reality and virtuality (Fig. 2). The different degrees of immersion in the three works will be analyzed, taking as a starting point the concept of Virtual Continuum, postulated by Milgram et al. [30] in his publication “Augmented Reality: A class of displays on the reality-virtuality continuum”, written in 1994.

Fig. 2.

The concept of Reality-Virtuality (RV) Continuum proposed by Milgram et al.

In this essay, Milgram contextualizes the Reality-Virtuality Continuum, which ranges from authentic environments to completely virtual environments, presenting specific taxonomies for each immersion level from the unmodeled world (real) to the completely modeled world generated by computers (Fig. 3).

Fig. 3.

Taxonomies in the Reality-Virtual Continuum by Milgram et al. Extent of World Knowledge.

The works NUVE⁵ [31], introduced in 2010; Co:Lateral⁶ [32], presented in 2016; and UNA⁷, presented in 2020, will be subject to our analysis.

2.1 Nuve (2010)

NUVE is an artistic project that explores the artistic possibilities offered by the digital dance performances in the interaction between the individual and his virtual double. In NUVE (Fig. 4) we conceptualized, developed, and implemented a digital artifact, resulting in a digital fluid performance based on the analog body’s theme versus the digital virtual body [2].

Fig. 4.

Rehearsal by Né Barros and João Martinho Moura at the black box of Balleteatro, Porto. 2009.

In NUVE, the body is the principal motor of activity. However, the choreographic body advances into a connection with its own “virtual twofold” in an approach to make a discourse, testing the choreographic limits and adding new measurements to the motion. We were much inspired by pioneering works in the 1980s by Myron W. Krueger [33], Jeffrey Shaw [34], and Maurice Benayoun [35]. In NUVE (Figs. 5, 6), the “virtual body” separates itself from the performer at some phase in the performance, and de-embodiment processes happen. NUVE was created with openframeworks⁸, an open-source C++ programming language toolbox. At that time, before the Kinect interface, no depth extraction was easily possible. The work depended on an infrared firewire camera to catch the stage’s picture at 25 frames per second. Properly infrared lights were used to get the ideal image of the dancer. The audience could not see the infrared light in the dark. A blend of standard picture handling procedures was then used to catch the outline and the performer’s movement [31].

Fig. 5.

First presentation of NUVE in 2010 at Quintas de Leitura, Teatro Municipal do Porto - Campo Alegre. Balleteatro.

At certain moments during the movement, the dancer’s gestures entered in a feedback loop with itself, provoking reactions in the visual environment and sound. This consistent cooperation between the full body and the virtual imagery is conceivable by figuring the distinctive speed of the various parts of the body and space. Initially rehearsed in 2009, and firstly presented in 2010, this solo, interpreted by Né Barros, was the motif for several publications [2, 31, 36].

Fig. 6.

NUVE. Presentation at Balleteatro auditorium, in Porto, 2010.

2.2 Co:Lateral (2016)

Co:Lateral was developed from the NUVE artwork, a project for performance and digital art. Here, the body was projected and extended itself in a relationship of intimacy with interactive virtual reality. The performative discourse resulting from this connection calls for a poetic moment made of a mixture of realities, made of body, double, and images [37]. Co:Lateral evokes moments of the death of the swan immersed in an immaterial space of light and projection: a phantom of the archive of dance now returns to test itself in a reality of illusory imprisonment. The dialogue between digital art and performance allowed us to generate a communicative space that challenges choreography limits, recapturing gesture and movement, and creating a space for experimentation with new performative possibilities [37].

In Co:Lateral, the body capture is performed through two depth cameras, and tests were performed with different models [38,39,40] between 2015 and 2019. Specific software was developed for the different scenes, using C++ and openframeworks platform [41], Processing [42], OpenCV [43], applying different computer vision techniques, for example, background subtraction [44] and the optical flow, to capture the different directions where each body parts move, individually. Hence, it is a generative drawing that follows one body in motion, thought space, and time. Using the Lucas-Kanade algorithm [45], accelerations are perceived, and different movement causes appropriate forces in the environment. This visual abstraction causes a noteworthy convergence of generative draws, which makes the idea of the human figure to be seen drawn during the performance [46].

In the first moments, only two-dimensional representations of the moving body (Fig. 7) are projected.

Fig. 7.

Co:Lateral. Two-dimensional representations of the moving body. Né Barros and João Martinho Moura. Balleteatro, Porto. 2016.

The piece, initially performed by Sara Marú in 2016 and by Sónia Cunha in the next years, evoked the movement’s distorted memory. Let’s imagine that someone is drawing a moving figure and that the drawing reflects the current position, but, at the same time, the immediately preceding performer’s postures. The software presented a body with minimum connected possible lines and confronted the dancer with herself moments earlier. The image was projected in the middle of the stage, between audience and performer, on a transparent screen called tulle. This transparency creates a translucid illusion in the audience that the image is closer. At the same time, the dancer is behind, illuminated by a small light beamer (Figs. 7, 8). When we do not project, the audience sees only the dancer. When the algorithms begin to project images, those appear in her front, physically separated by 2 or 3 m, a poetic visual moment that enhances the dialogue between movement and the imaginary that it represents. This presentation scenario has enabled several possibilities that could not exist in the previous work. In Co:Lateral, the dancer played herself moments ago. The audience watched several moving personas recreated in a continuous loop in the visual environment (Fig. 8).

Fig. 8.

Co:Lateral. Presentation at Coliseu do Porto, in 2017.

Later, at certain moments, we turn on the Z depth coordinate, which allows us to obtain more information, such as the body parts closest to the audience. Jaron Lanier was one of the first artists to use gestures in electronic art with a device called Z-Glove, and used the hand in virtual environments to manipulate virtual objects that appeared in an image, suggesting, at the time, a broad spectrum of possibilities of representing objects in virtual environments of interaction.

In Fig. 9, we see a moment in which the performer draws, literally in the air, a gesture that materializes the letter N (of the word “no”), projected between the performer and the audience.

Fig. 9.

Co:Lateral. Performer Sónia Cunha drawing letters on screen. Presentation at Teatro Virgínia, Torres Novas, in 2018.

Throughout the performance, gestures are continuously analyzed, and specific scenarios consider the piece’s interactive narrative. We opted to process the original camera signal and not processed body points for the output visualization, as is quite traditional in body-tracking techniques for avatars’ recreation. We believed that the depth camera’s signal is entirely relevant to represent the performer’s body, as it is. In computing history, we have accustomed computers to immediately recognize human beings, whether through body detection [47] or facial analysis [48]. These mappings are sometimes not enough to describe the richness of detail in the subtle and artistic movement. There’s such rich information between shoulders and arms, harms and hands, head, neck, and chest, that point marking systems cannot catch. Figure 10 shows a relevant moment, where the performer gently ‘touches’ many white vertical bars, initially well marked, rigid, symbolizing a prison. After the interaction, lines react smoothly to virtual touch, immaterial, in fact, but real in the imaginary of the audience. With timed and more vigorous movements, the bars end up forming a volumetric image of the moving body.

Fig. 10.

Co:Lateral. The white bars end up, forming a volumetric image of the moving body. Presentation at Art and Tech Days, in Kosice, Slovakia, in 2019.

In the preparation of the Co:Lateral work, we had the opportunity to explore the three-dimensional body on stage, using multiple depth cameras. With the necessary spatial calibration, the body was recreated and presented volumetrically. Different aggregations of points demonstrate the fragile, even in low detail resolution, of the female body in motion (Figs. 11,12 and 13).

Fig. 11.

Volumetric fragile body in Co:Lateral. 2016.

Fig. 12.

Presentation at Arquipelago, in Açores, and Temp d’Images, in Lisbon.

This piece has been active for four years, and so has undergone enough developments in its course of presentations. It was presented in several theaters and stages.

In the various communications we made with the public, we felt that this approach to the stage equipped corporality’s performance, difficult to achieve without media art technologies. This volumetric body had more and more enhancement in presentations between 2017 and 2019. Several trials were conducted to show it in its fullness. In the following images, we can see frames of motion sequences obtained from rehearsals and performances. Videos can also be observed online⁹.

Fig. 13.

Sequences of motion in Co:Lateral. Presentation at Temp d’Images, in Lisbon, Portugal, 2019.

During Balleteatro’s 37 years of existence, Co:Lateral was one of the most presented performance, having been selected for EAI ArtsIT 2018¹⁰ artistic venue, in Braga, celebrating the UNESCO Braga Media Arts initiative, for ISEA 2019¹¹, in Gwangju, South Korea, as research, for Temp d’Images Festival in Lisbon¹², Portugal, and Art and Tech Days¹³ in Košice, Slovakia.

2.3 Una (2020)

In the rehearsals of Co:Lateral, we asked ourselves what if we could see that volumetric body in total immersion in VR. This question was a leitmotiv for creating the next work, UNA, planed at Balleteatro in 2018, as a trial in total immersion, and firstly presented in 2020 (Fig. 14). In NUVE, performed by Né Barros in 2010, the relationship between the choreographic body and its artificial double was explored, in the space-time, projected, and extended in an intimate relationship with the virtual. In Co:Lateral (2016), the immaterial space expanded, the image became closer to the public, and between the moving body and the audience, transparent, mixed, embodied realities were presented. UNA is a possible continuation of this immateriality, this time, in total immersion, where the audience (one spectator at a time) witnesses the body in movement, again, expanding. Ten years after NUVE, one goes back to testing, to the laboratory, transforming and questioning the performative and embodied space, where the spectator is one, is not in the audience, but in the center, in a space that does not exist, and where different understandings of the performative body are reflected [49].

Fig. 14.

UNA (2020). Né Barros performing in virtual reality. Balleteatro, Porto. 2019.

When using virtual reality, we noticed that many experimenters needed some initial time to get used to the technology. The performance was announced as an experimental trial. The audience was explicitly told that it would take about 5 to 8 min. Even though this information was made aware by the public, only at the venue, they felt that they would be teleported to another space, one by one. In this way, the spectator enters the Coliseu, stays in a waiting room, an antechamber, waiting for his/her turn. When called, the participant moves into the middle of a selected room, being received by an assistant, who explains that the performance will take place in total immersion and provides assistance in placing the equipment. In this room, Né Barros is positioned laterally, serene, and calm. Silence is total. There’s no big audience, only one spectator and one performer, and a room assistant. Headphones with noise cancellation are also correctly placed in the participant’s head. These headphones have two microphones that capture all the outside sound, invert the signal, and make the viewer hear nothing except the performance’s sounds. When the viewer opens his eyes, an abstract representation of the room appears. This representation is minimal and only serves to place the viewer in space. The participant is still there. In the first experiments, this teleportation happened quickly, and many viewers did not adapt well to the change. So, in a second experiment, we decided to digitize the entire rehearsal room (Sala Dois) of the Coliseu do Porto. Thus, when the participant puts on the helmet, the space he sees is an abstract representation of the area where he is, in reality. Looking sideways, they realize themselves inside the Coliseu. This smoother transition between real and digital gives the participant the time needed to adapt to the headset’s images. Not everyone feels at ease when experiencing virtual reality, as that equipment cause disorientation or nausea sometimes. So, the assistant is always present. After a minute, the participant is gently elevated to a virtual height of about 1km, thus leaving Porto’s city and begins to see a different scenario, the one where performance begins (Fig. 15).

Fig. 15.

UNA (2020). City of Porto and Coliseu do Porto. Balleteatro, Porto. 2020.

This elevation is accompanied by real wind. We put a fan on stage pointed to the participant, triggered at the elevation moment. This physical sensation caused chills in the participant’s skin when they felt taken to immaterial space. Né Barros thus approaches the capture area and places herself in the fetal position (Fig. 16). Its body shape begins to appear, slowly. All movements happen steadily, as the participant has the freedom to look everywhere and may not be in the performer’s frontal position (Fig. 17). Indications in spatial sound help the participant to better orient himself.

Né Barros gently approaches the participant and begins to gesture. The participant realizes there is a body nearby. In this scenario, there is no traditional physical barrier between the stage and the audience, as in previous works. The notion of performative space is challenged.

Fig. 16.

UNA (2020). Né Barros performing in virtual reality. Balleteatro, Porto. 2020.

The participant can also move in an area of 3 square meters. Furthermore, can approach a few centimeters from the performer or even incorporate her, depending on its position in space. The artist hugs the participant, touching him, virtually, positioning herself in front of the depth sensor (Fig. 18).

Fig. 17.

UNA (2020). Né Barros performing in virtual reality. Frontal view in virtual reality. Balleteatro, Porto. 2020.

Fig. 18.

UNA (2020). View from participants’ perspectives in different positions. Balleteatro, Coliseu do Porto. 2020.

At the final moment, the participants return to the place where they started. Né Barros positions herself laterally to the participant and held out her hand (Fig. 19). The headset’s virtual camera moves smoothly to a side position to capture both the participant and the performer’s bodies. She approaches very slowly. And then something unexpected happens, hands touch. The feeling of physical belonging happens. It was a remarkable moment for the Balleteatro, a long journey in this sequence of works related to the digital embodiment. Informal conversations with about 20 participants immediately after the performance presentation show that they felt embrace and bodily involved.

This work was presented two times in 2020¹⁴, the last one happening days before the COVID pandemic lockdown. After March, all the following exhibitions were postponed as embodied virtual reality between participants is very intimate. After the pandemic restrictions, we plan to return to new exhibitions (Table 1).

Fig. 19.

UNA (2020). Rehearsals and experiences with participants in virtual reality. Tests with touch between performer and participant. On the right: João Martinho Moura and Né Barros at Coliseu do Porto. Balleteatro. 2020.

Table 1. Comparison between NUVE (2010), Co:Lateral (2016), and UNA (2020).

3 Conclusions

The three pieces presented at the different moments in the last decade have as their central element one body, immaterial, always associated with the physical body that materializes it. Moreover, this body, educated for performance, is crucial for its expressiveness to be transposed to the digital narrative. Looking chronologically at the pieces, we quickly realized the technology itself influenced their creation, and, indeed, also limited or reinforced specific components that we will now analyze. Throughout this process, the body has been dematerializing. Although increasingly digital, it remains well present, and its physicality has never been called into question. The image was approaching the audience more and more to the limit of the senses. If in NUVE the image was behind the dancer, in Co:Lateral was in-between the performer and the audience, and, in UNA, the image became mixed with the audience. The presentation models also changed significantly between NUVE, Co:Lateral, and UNA, the space for the audience decreased, with more space for imagery. We have reached a point where one performance is presented to one participant at a time. The performer’s difficulty increased significantly as we entered total virtuality, which was notable in fatigue after presentations. As for the audience, it became increasingly immersed along with the different performance presentations. We noticed that this greater immersion requires a doubled public attention, especially when we advance on virtuality degrees. As the three pieces were created in the same dance company and rehearsed with the same professionals over the last ten years, we managed to have a generalized view of the audience’s receptivity. Because Balleteatro houses many students and the local artistic community in Porto, in many cases, some participants attended the three pieces in different years. We obtained continued feedback from these individuals, which reinforces what we mention in this conclusion. In the next table, we group and compare the three works by different parameters:

Returning to the idea presented at the beginning of this publication, the three works converge to Milgram’s concept of Reality-Virtuality Continuum presented in the 90’s, which we associate in this essay with embodied performance in virtual reality. These works were relevant in Balleteatro history, an institution that accompanied many developments and changes in artistic performance over the last four decades.

NUVE, Co:Lateral and UNA allow mapping an evolution of a relationship between performer and visualization. On this map, it is possible to locate the performing body, not in absolute physical presence but performativity’s most profound dimension. In other words, the cartography of the choreographic gesture is drawn through its dynamic qualities captured and transformed by the technological device, and recreate new and diverse entities. It is no longer so much about thinking about technoculture through the mutation in the perception of performing arts but about perceiving these entities’ impact on the ontology of performance. If in NUVE, the gesture expands and, in doing so, creates a new scenario for the performer, in Co:Lateral, the entire performative space is taken by the visualization of the expanded gesture. There is no scenario, but, rather, a dynamic and interactive place. UNA breaks with the subject’s barrier based on a principle of double immersion of the performer and the receiver and the performer with himself. In this virtual dimension, the performer’s experience is itself of embodiment: the performer moves through the perception of the movement he generates. There is an experience of duplication, of unfolding between physical body and gesture. If in previous works one could also speak of incorporation, be it through the experience of expanding the scenario to a body (NUVE), or through the recreation of an interactive place where the presence of the physical body gives way to the virtual presence (Co:Lateral), in NUVE the embodiment is revealed through a hypersensitive experience of displacement of the gesture. In UNA, the possibility of mapping the ballast of the gesture is achieved. It is no longer the gesture gathered by the memory of an image, but by a kinesthetic experience of the movement reinforced and offered by a type of visualization that the technological device allows performing.

In addition to these aspects that affect and challenge choreographic thinking, there is another of great importance: memory rewrites time. In any of the projects, the gesture’s visualization is the product of adequate control of its execution. Survival, which defines the condition of the gesture’s ephemerality, is revealed in this partnership (performer and digital media) as a very concrete condition in the production of a performative discourse. The control of forms that can become chaotic or constructing a choreographic nexus is hostage to a micro dialogue between executed gesture and visualization. There is a strong relationship between micro and macro and the possibility of working with the gesture’s memory through decalage effects. In Co:Lateral, the work on the gesture’s memory, where present gesture dialogues with the past gesture, reveals the vital principle of constructing the choreographic work. In the end, a kind of death, given by a three-dimensional image, symbolizes a journey of memory, of inner dialogue or, in another perspective, of a monad. All the incursions that these projects promote put performance and performativity necessarily in tension with the notion of presence or interaction, preferring an expanded domain as a territory for exploration and contact.