Keywords

1 Company Description

In The Room Global Ltd is a media-tech company specialising in conversational media technologies. The company’s software integrates voice recognition, AI, and exclusive audio-visual content to recreate the experience of a unique face-to-face encounter: enabling audiences to ask questions to a notable person, on demand, 24/7.

The company’s work evolved from The Forever Project, an educational initiative at the National Holocaust Centre and Museum. The Forever Project was designed to preserve the experience of “meeting” a Holocaust survivor in order to preserve this experience for young people who might never have the opportunity to meet one of these witnesses in person. Through a life-size interactive video projection, users could ask questions to the image of the survivor using their voice, and receive an answer immediately: providing a new way to engage with the testimonies of multiple Holocaust survivors (Coward et al., 2017).

The company now works across platforms and industries, with its technology being used to create experiences delivered via web browsers for mobile access, life-size in-venue experiences, or immersive experiences in VR.

2 Project Summary

The Audience with a Hero project was a collaboration between In The Room Global Ltd., Bright White Ltd. (an award-winning design company which had worked on the original Forever Project), and Manchester Metropolitan University (with involvement from senior staff from the School of Digital Arts). Beginning in 2019, this 18-month project sought to develop the principles of In The Room’s work and apply these to a VR environment to create a unique and immersive encounter. The project was funded within the Audience of the Future programme by UK Research and Innovation through the Industrial Strategy Challenge Fund and Innovate UK.

Audience with a Hero centred on a single “hero” as its subject: musician and producer, Nile Rodgers. Its focus was on creating a powerful, engaging encounter with a hero in virtual space by developing and testing new and/or improved production processes which created a sense of “presence”.

With the concept of “presence” in mind, the production approach was designed specifically for use in an out-of-home virtual reality experience to heighten immersion.

3 Project Details

During the 18-month project, a transdisciplinary approach was required to create an effective, engaging experience, the principles of which could be replicated efficiently. In building a virtual reality experience based on a person’s real identity, the design challenges led the production team to investigate how human conversational dynamics could best be virtually represented. A direct objective for the project was also maximising the ability to facilitate the feeling of “presence” (central to the project vision’s notion of an encounter), which can be summed up as “the pre-reflexive sensation of ‘being’ in an environment, real or virtual, which results from the capacity to carry out intuitively one’s intentions within that environment” (Riva et al., 2014) [or more simply, “the perceptual illusion of non-mediation” (Lombard & Ditton, 1997)]. Specifically, the intention was to develop within the user a sense that the hero was present in the same space as the user him/herself.

In order to provide a more compelling and naturalistic interaction with Nile, project team collaborators conducted design research into topics such as proxemics, prosodics, and parasocial relationships. Proxemics research considered the position of Nile in relation to a participant in this particular encounter and the type of high-quality audio-visual capture that would be technically reliable in a high-pressure studio environment whilst providing the desired effect. Prosodics investigation considered how best to interview the subject in a way that would mimic natural conversational dynamics. Lastly, parasocial relationships were considered as a consequence of having a celebrity as the subject of the Audience with a Hero project, which necessitated research into interpersonal dynamics with talent. This resulted in exploring how the study of parasocial relationship formation might intersect with the experience and affect the public’s perception of Nile as a media figure. The study of parasocial relationships, which are the attachments formed between a member of the public and a media figure through one-way channels such as television (Horton & Richard Wohl, 1956), continues to gain sociological relevance through the deepening complexity of mediated relationships between celebrities and the public (Kim & Song, 2016; Yuksel & Labrecque, 2016).

The design and production process was highly creative with significant complexity, including advanced video production, natural language data and asset design, creative scripting, and technical development.

The VR project was experienced by audience members as follows: after putting on the VR headset, the user can see an empty chair in front of them. Out of sight, and to the side, they can hear guitar music playing. The musician and guitarist Nile Rodgers then emerges from offset, walking towards the chair and sitting down in front of the user. He introduces himself and initiates an opportunity for the user to ask him a question. There then follows an engagement in a conversational exchange allowing the user to ask questions using their voice. After each question is posed by the user, they see and hear Nile Rodgers answering in front of them.

To produce the experience Nile Rodgers was interviewed before a stereoscopic camera rig over 2 days to capture real answers to over 350 questions about his life and career. These media clips were indexed as part of a conversational agent system that allowed participants to ask these questions using their voice and see and hear Nile Rodgers responding from the pre-recorded assets.

Matching what was said by the participant to the appropriate clip of Nile Rodgers was managed through conversational agent technology and natural language processing (NLP). For more human-like interactions between Nile and audiences, conversational agent (CA) and design research sought to find better methods for rapid agent development that were more accurate and reliable in response to audience questions.

In addition to the creation of the VR experience itself, the project sought to find production methods and efficiencies that would not only minimise the cost of development in these experiences but also make VR experiences more compelling. In order to evaluate the effectiveness of production innovations, it was necessary to evaluate the quality of the experience for users in terms of a powerful, engaging encounter with a hero in virtual space. Evaluation questions were devised to ascertain how participants felt about their virtual meeting and about Nile himself “as a person” after the experience, in order to provide insight into the value of authenticity in a virtual encounter. Questions in the user evaluation also addressed NLP and presence by, amongst other questions, asking participants whether they were able to “follow their own line of questioning” during the experience.

COVID-19 restrictions resulted in a small number of people being able to test and evaluate the full VR version of the experience. The majority of the testing needed to be conducted on a large flatscreen using a fixed perspective video input where the image of Nile Rodgers was comparatively small in relation to the screen size. This difference provided useful qualitative evidence of the particular affordances of VR.

4 Feedback from End Users

Almost all (97%: 28 of 29 participants) reported a sense of having met with Nile and having had a conversation. This was consistent across flatscreen and VR tests. Factors that enabled this sense of a meeting were the facility to ask a variety of interesting, curated questions and for Nile to answer them accurately and engagingly, a sense that Nile was engaged with the Q&A process of answering the questions put to him, and the novelty of being greeted and said goodbye to in a convincing and personally touching way. Factors that detracted from the “magic” were repeated answers, fallback intents (e.g. Nile saying: “I didn’t record an answer to that question”), or technical problems.

However, whereas flatscreen participants commented on the framing, testing environment, and quality of the image, comments on these aspects were absent in the feedback from VR testers where a more immersive experience was achieved. All participants reported the sense of a palpable physical proximity to Nile when wearing a VR headset. Due to the contingent nature of the flatscreen test it was expected that it would only be sufficient to properly test aspects of the user journey and interactivity in the experience. However, whilst taking into account the relatively small sample size, it was clear that all of the research done on proxemics, 3D filming, and post-production had created a best in-class VR experience that made it feel as though the participant was sat in the room with Nile. This was evidenced in all interviews and observations of the tests. At one point, one participant laughed and cried out “Ah! Don’t sit on my knee! I’ve only just met you!”. Whilst this was an extreme reaction it highlighted the sense of Nile’s presence in the experience:

“I think the room, the way that the atmospherics of the room were shot, it felt more like a one-on-one. It felt like it was just him and me. There was nobody else.”

VR Participant 1

“It was more real than I expected. The fact that you could look down and see his shoes. When he stood up, I leaned back! It felt like he was present.”

VR Participant 3

In support of this observation in the feedback was the frequency with which some of the flatscreen participants expressed a wish to have experienced the work in VR:

That isolation of being in a headset [would] draw you in a bit.

If you had VR headsets though, if you felt he was physically there, that would be even better. Because he was only small on this screen, but if you were immersed in it, that would be brilliant.

In addition to testing the experience itself, a small follow-up study allowed the team to informally investigate the opportunity for further research in the area of synthetic representations of real people. This early testing confirmed the importance of authenticity in the representation of media figures. Participant feedback suggested that an audience member’s perception of these figures, and their emotional reaction to them, is affected by whether they believe footage to be real or not. Knowing that the footage was authentic, and therefore seen to represent the person accurately, was important. Conversely, issues of trust and ethics arose if the footage was considered to be synthetic. These issues are important considerations for further research.

5 Future Outlook

The VR development and subsequent challenges related to COVID-19 acted as a springboard for the planned web launch of the experience. The learnings from testing informed the design approach to the browser-based version of the Nile Rodgers experience, “In The Room with Nile Rodgers”, which was successfully launched in March 2021. It has also informed In The Room’s consideration of its scalable conversational media platform for the production of web-based experiences in commercial and cultural settings. Meanwhile, the team has developed a core efficacy and efficiency in terms of developing high-quality VR that makes the production of meaningful and convincing experiences feasible for the representation of real individuals within immersive environments. This has potential for informing other types of encounters within other immersive environments and displays, including augmented reality, holographic display, gaming, and metaverse environments.

The development of applications using synthetic media requires careful consideration given issues such as trust and authenticity, but might also present opportunity for further creative endeavour.

6 Conclusions

The alpha project testing provided significant and corroborated evidence that the Audience with a Hero project created a valuable, entertaining, and memorable experience. It reportedly left audience members with a sense of having met with a hero of popular music culture, often with a desire to return to “meet” with Nile and ask more questions. Whilst it would have been ideal to test with all users in VR, the project’s flatscreen version allowed the team to evaluate key aspects of the project, in particular the interaction design and the central premise of the experience, i.e. creating a meaningful and engaging encounter with a “hero”. The reactions of the test users demonstrated the significant potential in this area for future application across a range of industries and settings.

Finally, whilst the increasingly sophisticated affordances of synthetic media representation remain a further area of research, it is important to note that the project showed that the strength of the experience lies not only in its technical execution and design, but also the strength of Rodgers’ career and personality as authentically portrayed, and the consequent connection with his fanbase eager to hear what he has to say (and play). This is an important consideration in the creative development and application of In The Room’s approach, combining technical benefits with a method that is leveraging the power of human “presence”.