1 Introduction

In this paper, we describe a realization of virtual theatre, which we define as shared, live performance, experienced in a virtual space, with participants contributing from different physical locales. Using this definition, we see virtual theatre as a specialization of the more general term of telematic art [1], particularly suited to live theatre and making use of modern hardware and software that enable interaction, viewing, and immersion in shared virtual reality spaces.

Our goal is to produce a shared experience in the virtual world that mirrors that of theatre in a physical space. In contrast to theatrical productions that use make use of virtual elements to enhance physical theatre (e.g. virtual scenography [23]), we are concerned with performance that is realized, viewed, and experienced completely a the virtual world.

Since the advent of virtual reality, the use of virtual worlds for theatrical applications has been an active area of exploration. Stages in virtual worlds have housed productions of Shakespeares work [7, 17, 26], assisted in lighting and staging [13], been used to recreate historical theatre from the past [31], and served as rehearsal spaces for actors performing on a physical stage [25, 27].

Theatre is, by its very nature, a collaborative art; an active and live interplay between actors and audience. In addition to the basic technical issues related to building a system that enables this kind of performance, a major challenge is to create a shared experience where each participant feels as if they play an active part in the collaborative process even though they may be in different physical spaces.

We look to answer the question: What it is that makes live theatre such a unique experience? and explore if one can recreate that feeling in a distributed scenario even though it is experienced from different physical locales.

In this paper, we use the virtual theatre production of Farewell to Dawn Footnote 1 which was presented in Rochester, NY in December 2016 as a testbed to explore our ideas.

The remainder of the paper is organized as follows. In Sect. 2, we discuss attributes that distinguish theatre from other forms of entertainment. This is followed by a description of Farewell to Dawn. The next section provides a discussion on how our performance relates to the attributes mentioned in Sect. 2. Finally, the paper finishes with conclusions and future work.

2 The Theatrical Experience in the Physical World

In the physical world, theatre presents a unique experience as compared to other forms of media based entertainment such as television or film. In this paper, we focus on three attributes that define this experience and use these as guiding principles to frame our discussion about theatre for the virtual stage.

Liveness. Liveness can be described as connection of experiencing an event, as it is happening [4]. Each live performance is unique and different from every other performance, even if well rehearsed. Mistakes can be made and small variations in performance are not only possible but most likely to occur.

Theatre is unique in this sense in comparison to recorded media like film and TV, as the mere process of recording or saving for later viewing removes the liveness of the experience. Even events that are streamed live (e.g. live simulcast of theatrical performances or live TV broadcasts) loose their sense of “liveness” as, though they are viewed as the event is happening, the performers have no sense of connection and engagement with the audience [28].

Perspective. Perspective refers to the realization that the experience of each participant will depend upon their position in the theatre from where they view and interact with the performance. The actor’s perspective is surely different that of the audience, but even each audience member has his or her own seat in the auditorium making the point of view of each audience member unique from every other member. In addition, with theatre, an audience member is active observer. Unlike film where the perspective is carefully and creatively determined by the cinematographer, during a theatrical performance, the participant is in control of their point of view: where they wish to look, when they choose to look, or even whether they choose to look.

With theatre, the performance itself is the art that provides the entertainment [29]. Certainly, one can film a live performance, but in doing so, the audience perspective and point of view is fixed, determined by the positioning of the camera and controlled via the intentions of the camera operator. The determination of where the attention is placed at any given time during a performance is an integral component of film or video capture, thus taking this ability away from an audience member who may be viewing the same live performance in the theatre.

Social Presence. Theatre is inherently a dynamic social activity; it involves a group of people, assembling in a common space to tell or take in a performed story. Social presence refers to the extent to which people coexist and react to others in a given space [12]. Clearly, any interplay, whether explicit or subtile, between the actors and audience is unique to theatre and live performance. Social presence between audience members: the feeling of a shared experience of action simultaneously viewed in the same space; communication, both verbal and non-verbal between viewers; even the sense of proximity being seated next to a fellow theatre-goer, all add to the excitement of a live theatrical experience.

Fig. 1.
figure 1

View of the virtual stage for Farewell to Dawn, a virtual theatre performance. Actors controlling the avatars are shown in the insets.

Full size image

3 Farewell to Dawn

Farewell to Dawn [3] is a live, mixed-reality dance performance that combines virtual and augmented reality with motion capture and fully realizes our concept of virtual theatre. The piece explores the voyage of two dancers from a physical space to a virtual stage and back, as the day passes before them. Originally developed and performed as a mixed-reality experience in April 2016, a complete virtual theatre version was presented at the MAGIC center at the Rochester Institute of Technology in December 2016. A still from the performance, with inset of the actors controlling the avatars on the stage, is shown in Fig. 1 Footnote 2.

3.1 Physical Setup

The physical set up for Farewell to Dawn is illustrated in Fig. 2. The action takes place on a stage in a virtual space with the performers represented by stylized avatars controlled by human dancers in separate, physical, motion capture spaces. These dancers are equipped with augmented reality headgear, through which they can see the view of the virtual stage from their avatars perspective. Audience members view the performance from the perspective of a seat in the virtual theatre.

Technically, we adapt a distributed 3D gaming engine [6] to create the shared experience, allowing us the ability to create an application for experiencing the performance on a variety of different output devices. Motion capture is performed using a set of networked Microsoft Kinect sensors, two Kinect devices for each dancer.

3.2 Participant Experiences

Actor Presence. A major challenge with acting in distributed virtual spaces lies with the interface, as the default means of interacting in these worlds using a mouse and/or keyboard is unnatural and non-intuitive. The communication of emotion through body language and facial expression is paramount for the trained actor, dancer, or performance artist [16, 25, 34]. To address this, we utilize motion capture to provide a natural interface for actors in controlling their avatars. With regards to avatar motion in virtual worlds, theatre and the performing arts can not only be used as a guide for animating believable and expressive motion in virtual worlds, but can also be used as a gold standard in evaluation of such motion [20]. In particular, the use of nonverbal communication such as body language, and facial expressions are key elements in theatrical communication [22].

In theatrical expression, proxemics (the relationship between actors in the space they inhabit) [8] is as importance as the kinesics (posture, gesture, body motion) [2]. In our past work on virtual theatre [5, 6], projections on a large screen or wall were used to give the actor a very rudimentary sense of where their avatar stood on the virtual stage. Very often, motion of the avatar on the stage was achieved with assistance from a stage manager who would be watching and interacting from a separate workstation. The lack of the ability of the actor to accurately assess where he/she stood in the virtual space relative to stage setting, props, and most importantly, other actors, greatly restricted their expressive capabilities. Thus, it is essential that the actor in the physical space be able to experience the virtual stage from the point of view of their avatar.

Fig. 2.
figure 2

Physical setup for Farewell to Dawn

Full size image
Fig. 3.
figure 3

Actor performing in a physical motion capture space.

Full size image

We use augmented reality (specifically via a Microsoft Hololens) for this purpose. Though it would be tempting to fully immerse the actor in the virtual space, we chose an augmented reality experience for the actor as it is essential, particularly in dance, that the dancer be aware of the physical space that they inhabit. Figure 3 shows one of the dancers in the motion capture space.

The view of the stage as projected through the Hololens of one of the dancers is shown in Fig. 4. Though all participants share the same virtual space, the viewing experience can be customized for individual participants. For example, the dancers involved in the project found the particle representation of the avatar to be distracting, Thus, for the actor viewing app created for the Hololens, the avatar is represented more simply as a set of spheres; one sphere at each of the joints of the avatar.

Fig. 4.
figure 4

View of the virtual stage for Farewell to Dawn from an actor’s perspective as projected through a Hololens

Full size image
Fig. 5.
figure 5

External audience viewing performance projected in a physical space.

Full size image

Audience Perspectives. Audience members view the performance from the point of view of a seat in the virtual auditorium. Conceptually, individual audience members will do so from their own physical space using an immersive head mounted display. For the December performance, however, considering the limited number of head mounted devices available, we offered both an external experience, where audience members view the performance projected in a physical space (Fig. 5); and an immersive experience where the audience experiences the performance using an Oculus Rift.

Audience members are represented in the virtual space by futuristic heads whose orientation are adjusted based on the head motion of the person viewing the performance in the physical world as illustrated in Fig. 6.

Fig. 6.
figure 6

Audience avatars in the virtual space which move based upon the head motion of the audience member watching in the physical space

Full size image

4 Discussion

In this section, we discuss some the successes of Farewell to Dawn, particularly with respect to the attributes listed in Sect. 2. Though we did not conduct a formal user study, we present anecdotal observations as expressed by participants at the December performance.

4.1 Liveness

For audience members viewing the performance projected in a physical space (Fig. 5), a view of the dancers in the motion capture environment was projected on screens adjacent to the view of the action on the virtual stage. The correspondence between the motion of each dancer and their respective avatar during the performance was quite clear. Those viewing through the Oculus Rift did not have this “behind the scenes” view available to them. However, the general feeling was that the immersion afforded by the head mounted display increased the presence felt by these viewers, confirming studies previously reporting such connection [15] and contributing to sense of watching it live.

For the actors, the naturalness and intuitive interface of the motion capture environment made the performance feel like a live experience. Though it was the avatars that were performing on the virtual stage, the use of motion capture increased the sense of social embodiment [9] of each dancer with regards to the three aspects: self-location (experience of being inside the avatar body), agency (having full body motor control), and body ownership (self-attribution of the avatar body).

Fig. 7.
figure 7

View of Farewell to Dawn from the perspective of one sitting on the stage

Full size image

4.2 Perspective

Defining participant perspective involves setting both the position as well as the point of view of a user in the virtual space.

Theatre realized completely in virtual spaces provides extensive freedom of viewer placement and positioning. Being that all viewing is done from the perspective of a point in the virtual world, modifying the perspective for a particular participant can easily be achieved by setting the camera position to the appropriate position within the viewing application. This opens the door to some interesting new possibilities with regards to the means by which theatre in virtual spaces can be viewed and experienced by an audience member. Theatrical pieces such as Draw Me Close [21] and To Be With Hamlet [7] take advantage of this freedom to allow for new and unique audience perspectives.

In Farewell to Dawn, selected audience members were placed on the virtual stage allowing them to view the dance while seated amongst the tables and chairs of the cafe in front of where the avatars are performing, as shown in Fig. 7. This juxtaposition made many feel present at the cafe watching as one would on an actual city street (some to the extent that they were seen reaching out for the cup that was on the table).

Fig. 8.
figure 8

Audience members placed on the virtual stage. Audience members can view not only the dancers’ avatars but also the motion of other audience members, giving a sense of shared co-presence.

Full size image

4.3 Social Presence

Presence in virtual worlds is a rich and complex topic that can be explored from a variety of different perspectives [14]. With regards to virtual theatre, we consider social presence to relate to “presence as transportation” [14] where the participants not only feel transported to the virtual space, but also feel the presence of those around them in that space.

In the external viewing experience, the sense of co-presence amongst audience members naturally comes from the proximity and seating in the physical space. However, there is no social connection between the actors and the audience as there is no portal from the physical motion capture environment in which the actors are performing to the physical space where the audience is sitting.

In the immersive experience, co-presence is achieved thru the avatars of the audience members as well as the actors in the virtual space (Fig. 8). Multiuser virtual spaces have been shown to be successful in creating a sense of social presence [18] leading to what can be termed distributed liveness i.e. a sense of connectedness with other participants even though they may be in different physical spaces [33].

Finally, the sense of immersion provided by the head mounted display itself, combined with the proximity to the dancers in the virtual space, and the real time motion of the audience avatars, were reported to provide a further sense of connectedness with both the performers as well as fellow audience members.

4.4 Playback Options

Documentation of a theatrical performance has always presented a challenge as the mere act of recording is in direct contradiction with the “liveness” aspect that distinguishes live theatre [24]. Capture using video forces a fixed perspective and removes any sense of shared co-presence. A recent exciting trend in recording of theatre in VR (e.g. Hereos: A Duet in Mixed Reality [11] and The Lion King VR Experience [19]) makes use of 360\(^\circ \) video presentation providing a unique perspective for the viewer. In these experiences, the viewer is given some control over their point of view, however, the perspective is still limited by the camera positioning as defined by the film maker.

With virtual theatre, since all of the action is realized within a 3D engine, “recording” of the performance can be achieved by saving time varying signals (avatar motion, lighting, staging) for playback later. In this case, the playback is a reconstruction of the performance, which can be experienced from any location in the virtual space. Though this approach does not address the absence of liveness, it does allow for the possibility of providing the correct perspective and creating social connectedness, at least amongst audience members when viewing a previously recorded performance.

5 Conclusions and Future Work

In this paper, we explored our vision of virtual theatre: enabling theatre on a virtual stage with participants in different physical locals, through the presentation of Farewell to Dawn, a virtual theatre dance performance. The presentation served two purposes: First, from a purely technical standpoint, it was a proof of concept of a hardware and software framework for enabling such a performance. Secondly, and more interestingly, it provided a means of exploration of the theatrical experience and determining how that same experience can be recreated using a virtual world.

Towards this goal, we have identified three characteristics of theatre: a sense of liveness, providing perspective, and facilitating social presence that must be achieved in the shared space while participants are separated physically.

Farewell to Dawn allowed for an initial exploration of these three characteristics. We are currently in the process of defining a followup performance piece that will consider these characteristics in more depth. Specifically, this new piece will include:

  • More realistic avatar models – The use of a stylistic, “particles of light”, representation for the avatars in Farewell to Dawn was both an artistic and functional choice as the particle system is quite forgiving of the sometimes noisy motion capture data provided by the Kinects. Moving forward, we would like to incorporate more realistic and more human characters to the virtual stage. New models will also include facial models capable of displaying facial expressions driven by facial motion capture systems; not only for actor avatars but for audience avatars as well.

  • Deliberate and intentional actor/audience interaction – Farewell to Dawn presented a passive audience experience as there was no explicit interaction between actor and audience which is not uncommon for many theatrical situations. However, the next piece will force explicit and deliberate actor/audience interaction to test the effectiveness of social co-presence between the performers and the audience in the virtual space.

  • Formal measurement of participant engagement – All reported observations of user experiences for Farewell to Dawn are purely anecdotal. For our next piece, we are planning to perform more formal data collection via questionnaires, and the use of affective sensors (e.g. Galvanic Skin Response and/or Pulse [10, 32]) to assess participant engagement and presence.

This work has focused on the human experience during theatre: motion of the actors, the interplay between actors, the engagement of the audience. Looking forward, we would also like to experiment with other theatrical elements such as lighting, staging, and actor control and interaction with props and sets. The dynamic interplay of these elements with live music (e.g. [30]) would be an interesting direction in which to explore.