Immersive Storytelling in Augmented Reality: Witnessing the Kindertransport

Abstract

Although hardware and software for Augmented Reality (AR) advanced rapidly in recent years, there is a paucity and gap on the design of immersive storytelling in augmented and virtual realities, especially in AR. In order to fill this gap, we designed and developed an immersive experience based on HoloLens for the National Holocaust Centre and Museum in the UK to tell visitors the Kindertransport story. We propose an interactive narrative strategy, an input model for Immersive Augmented Reality Environment (IARE), a pipeline for asset development, the design of character behavior and interactive props module and provide guidelines for developing immersive storytelling in AR. In addition, evaluations have been conducted in the lab and in situ at the National Holocaust Centre and Museum and participants’ feedback were collected and analysed.

1 Introduction and Background

According to Reggio Emilia’s concept of the ‘third teacher’, the physical learning environment and space, such as a museum, play a particularly important role in modern learning, can act as primus inter pares (first among equals) and create a direct communication with the young learners. However, many museums traditionally have perpetuated the “primacy of the ocular”, where they have supported visitors looking at objects, but discouraged them from actually handling or experiencing objects through different sensory modalities [1]. Though it was found that some museums were using innovative techniques and multi-sensory approach to augment children’s learning experience, there is still much to do to create compelling education for the young generation.

Narratives are proved to be a powerful method for empathy such as perspective taking and emotional engagement [2]. Many researchers claim positive association between empathy and prosocial behaviors [3,4,5].

The National Holocaust Centre and Museum (NHCM) is one of the few museums employing the narrative technique to unveil the history and enable young generations to carefully examine, commemorate and learn from the tragedy of the Holocaust. The Journey, one of its permanent exhibitions, tells the story using environment storytelling technique through the eyes of a fictional Jewish boy Leo who survived the Holocaust and came to the UK via the Kindertransport¹. Six rooms are restored to show how Jewish’s life look like including Leo’s family living room, Leo’s classroom in his school, the street after Night of Broken Glass, the tailor’s shop of Leo’s family, a train carriage for Kindertransport and refuge in the UK. In each room, audience can watch a short video of Leo giving a monologue of what he saw, heard, experienced and felt at that time [6]. The visitors can experience the complete story gradually by going through each room, interacting with objects and watching videos.

NHCM devotes to make the exhibition more compelling and accessible. With the invention of HoloLens [7], a Head Mounted Display (HMD) for Augmented Reality (AR), a new form and media for narratives and exhibition has emerged. This new AR device differs from the previous hand-held devices (e.g. mobile phone) or VR because it overwhelms participants sense by filling the space with virtual holograms and spatial sound through the HMD, but still keeping them connected with the real world. HoloLens can be considered as a new media which creates the unique Immersive Augmented Reality Environment (IARE) where virtually holographic characters and objects can be mixed into real-world space seamlessly while participants observe, walk and interact with things in the real-world environment. IARE enables people to navigate and interact with a physical space freely while watching a performance of holographic characters. In a word, HoloLens offers a new possible solution for museums conveying stories and a fully-immersive experience to the audience.

2 A Review of Immersive Storytelling in VR and AR

There are two distinct properties of HMD AR as well as HMD VR compared with other media. Presence is one of them, which refers to a subjective user experience of being in the virtual or mixed-reality environment rather than the actual physical locale [8]. Different from the flat screen, HMD is a binocular device which can not only help user perceive the accurate size of an object, but also cover the large part of user’s vision to generate an overwhelming sensory feeling.

The other feature is agency, which refers to the ability to “do something” in the experience—to interact or react rather than simply perceive. Agency is the core of VR/AR experience because virtual environment (VE) within headset gives the audience the native agency of looking [9]. In other words, IARE has the built-in nature of interaction as audience would like to have more ability to interact with the environment rather than looking.

Due to the lack of theory and barrier of the technical issues, though there were a few VR narrative animations/films on the market, such as works from Oculus Story Studio like Lost (2016), Henry (2016) and Dear Angelica (2017), works from Google Spotlight like Rain & Shine (2016), Buggy Night(2016), Pearl (2017), Back to the Moon (2018), Scoring Age of Sail (2019), VR documentary like Nomads (2016) and VR feature film Manslaughter (2015). These visual carnivals have achieved great successes from the market and Pearl even won an Oscar nomination. Yet most of them are timid and unimaginative in respect of screen grammar, employing either a complete static shot or a continuous long take. The only two creative films namely Pear explores the teleportation across the space and Dear Angelica explores collision of intensive color and abstract shapes to elicit emotions. There was also very few immersive AR narrative work, fortunately, Fragments, a suspense & adventure narrative experience of HoloLens developed by Asobo Studio [10], is a pioneer exploration as an AR narrative experience. The success of Fragments further revealed potential for narratives in IARE.

IARE is probably effective for narratives of serious purpose rather than suspense or adventure types. IARE can involve hologram of virtual characters that enable the richness and more possibilities of a profound storytelling suitable for adults. Further, IARE can also adapt easily to a new real-world space and projects the virtual characters, furniture and object into the new one. Lastly, the experience of narrative in IARE aligns with the experience of video games and immersive theatre, which is affable and approachable for young adults.

3 Design

The aim of the project is to develop an AR application, creating an innovative experience for museum which focuses on fostering children’s moral development. The objectives of this AR application are as follows:

• 1. engaging young generation to learn more details about this history;

• 2. fostering sensibilities to the principles of equality and justice for historical-political learning.

The design process followed 4-phases procedurals proposed by Hartmut Koenitz [11], starting with paper phase (Stage 1) to create the general outline of events and use a flow diagram to visualize sequencing and choice point of a Jewish boy Leo’s Kindertransport experience. In the prototype phase (Stage 2), the interaction design was examined without assets, and redundant narrative elements were removed and feasibility of interactions with characters and props was checked. The production phase (Stage 3) included the development and integration of final assets, such as modeling, texturing, animation, programming, audio and music. In the testing phase (Stage 4), the AR app was published and evaluated in the lab, and the app was revised according to user feedback to be ready for final experiment in situ.

3.1 Interactive Narrative Design

The first step was to develop an interactive narrative strategy for this AR app. In order to ensure historical accuracy, the script and dialogue need to be rewritten based on an existing script from a parallel project of the virtual journey app [12] in collaboration with a historian at the NHCM. One influential narrative approach taken in Facade allows for users to influence the story through natural language conversations established with the main characters, thus influencing its outcome [13]. However, considering the dialogues and plots of this app were required to be designed based on survivors’ testimonies and facts of the past, the narrative approach used in Facade can’t work in our case. In other words, the theme of the story should be clear and chain of events were limited to history facts.

Koenitz and Chen [14] summarized a model of interactive narrative strategy (see Fig. 1). Based on this model, the narrative strategy used in our AR app is illustrated as solid lines in Fig. 1.

Fig. 1.

A model of interactive narrative strategy

• As IARE has the built-in nature that allows the audience to walk within a real-world space and touch, interact with real-world or virtual props and objects, an explorative introduction was chosen, audience need to find their own ways into the narrative world instead of having a clear guide.

• As the purpose of The AR Journey is Holocaust education and the story of Leo’s escape from Germany through Kindertransport should be conveyed clearly, interwoven world type was chosen, in which an overarching narrative instead of several sub-narratives exists.

• Conditional-path means consequences of choices are influence by audience’s decision made through the story rather than entirely pre-determined by the author (fixed-path), meanwhile, conditional-path also involves more asset development. Therefore, a technique named fake conditional-path was used, which utilized a fixed narrative path under the hood, but letting the audience have an illusion that they can influence the plot.

• In order to maximize the display capability of HoloLens, a mixed presentation was made, including virtual character animation, graphic, dialogue, music and text.

• Point of View (POV) is an interesting topic for narrative in IARE. Since the audience and the virtual characters are in the same space—the augmented reality real-world space, the audience aren’t able to view as an omniscient viewer but an invisible observer (third person view). Audience could also act as one of the characters in the story (first person view) or act as a visible observer (second person view). If audience is a visible observer, the virtual character could “see and interact” with the audience, making the audience easier to empathy with them. Thus, in this project, either first person view or second person view could be a good choice. In this case, second person view is selected as the POV of the narrative.

The level design and scriptwriting were based on the above narrative strategy. Figure 2 shows the main story of Level 1 in a branch structure, using the fake conditional-path technique. The main story focused on a debate between Leo’s parents taking place in the living room and questions are raised to the audience whose opinion they agree with.

Fig. 2.

Flow diagram of Level 1

Level 1 is designed based on the real-world living room scene of the Journey exhibition. When wearing HoloLens in the living room, audience are introduced by an off-screen of a narrator’s voice that they are equipped with supernatural power to explore invisible materials and they need to find a submerged story inside this room. The user can interact with several objects in the room, such as telephone, radio, a suite case, a newspaper and a gramophone (see Fig. 3). Through these items, audience can find fragments of information like Nuremberg Race Laws, Nazi government propaganda radio broadcasting, and Leo’s parents’ happy life in the past. When finding Leo’s diary (key item in the Fig. 2), vivid virtual holographic characters of Leo and his parents appears in the room (Fig. 3 right), talking and debating with each other. At the decision point of branches, audience can help Leo to make decisions including supporting mother or father, attitude towards equality, raising questions to parents. Whatever the audience’s answers given to the branch 1, 2 & 3, the plot outcomes always lead to the same point (Fig. 2). However, as an on-site experience at NHCM, majority of the audience are supposed to have a one-time experience rather than repeating the experience several times. Therefore, the fake conditional path design can be possibly effective. After the main story ends, audience can explore more items or real-world props in the room or move to the next room/level to continue the story.

Fig. 3.

Concept image for the AR Journey in living room at NHCM (right) and layout of some props (left)

Level 2 is designed based on real-world classroom scene of the exhibition. As the Fig. 4 showed, Level 2 also uses a branched structure for the main story, and allows audience to have more interaction to find more fragments of the story. Leo talks about his distaste of going to school at the end of main story of Level 1, players then enter into the real-world classroom scene where the reason for Leo’s soreness can be explained. Audience could trigger the reenactment by finding Leo’s diary and witness the uncomfortable conversation between Mr. Becker the teacher and Leo. Audience can help Leo answer Mr. Becker’s challenge (see Fig. 5). Besides, if audience help Leo to find his favorite missing drawing, the virtual wallpaper will be mapped into the real-world classroom, consisting of floating text and pictures on walls. Audience can read and interact with the virtual wallpaper as well as the human figure icon placed on the desk (see Fig. 5). By activating the icon one by one to hear Leo’s classmates’ voice talking about their attitude, understanding and behavior towards Leo and Jewish people, it is like a virtual interview. Finally, audience are asked to write down their comprehension for the initial question on a sticky note and put it on the blackboard.

Fig. 4.

Flow diagram of Level 2

Fig. 5.

Concept image for the reenactment of school scene

Level 3 is designed based on real-world tailor’s shop scene of the exhibition. Level 3 is climax of the story and use the same narrative structure as Level 1 but with more branches. It has two parts which are Night of Broken Glass and family’s discussion on sending Leo to the UK via the Kindertransport. Audience can witness the mess after Night of Broken Glass and the sad family conversation. They can help Leo comfort his mother, find the missing item during the riot, decide which item to pack into the suitcase, decide how to spend the final day in Berlin (Fig. 6).

Fig. 6.

Players need to decide which item Leo should package into the small suitcase to take along with him (left), concept image for ‘white rose’ scene (right)

Level 4 is designed based on the real-world exhibition room where audience is encouraged to brace up Leo by leaving a voice message that turns into a swaying white rose in the air (Fig. 6). Audience can pick up a white rose of others to hear their message. In this way, players share their interpretation and memories of this AR Journey.

3.2 Input Strategy for HoloLens

The interaction design includes three parts: the HoloLens’ input mechanism, the character interaction system and the props interaction system. Storytelling with HoloLens is triggered by user input and the user needs to choose a branch of the story using HoloLens’ interface. The current available input for IARE, which consists of input type and input model, is analysed. The input type refers to fundamental genres of input in IARE including gaze, gesture, clicker and voice command. Audience can perform the same type using different apparatus, e.g. the action of gaze can be performed via head (head gaze) or eyes (eye gaze), the action of pointing and manipulation can be executed via hand gesture or a controller. It is important to understand that different input types can be combined or used alone with their own conventions to form an input model, which are listed as followings [15]:

• Direct manipulation is an input model that involves touching holograms/real world objects directly with one’s hands or controllers. (Hololens 2 only)

• Point and commit is a ‘far’ input model that enables users to target, select and manipulate 2D content and 3D objects that are out of reach using hand gestures or controllers.

• Gaze and commit is a far input model using eye/head gaze and commit action via hand gesture, a controller or voice command.

• Gaze and dwell is a hand free input model. The user keeps gazing (with their head or eyes) at the target and lingers there for a moment to activate it.

• Voice input is a hand free input model by using one’s voice to command and control an interface, e.g. the user can read a button’s name out to activate it

As HoloLens 2 is still not available in the consumer market and the motion controllers are too large to eliminate the gap between virtual and real worlds, input models of direct manipulation and point & commit are out of the question. Eye gaze is excluded because it is only available for HoloLens 2. Voice input is infeasible as Chinese voice input is poorly supported and hand gestures have proved problematic in our preliminary study. In summary, there are two possible paths of interaction left for the AR Journey, which are head gaze & commit with a HoloLens clicker² and head gaze & dwell (Fig. 7). To be more specifically, audience are able to use their head gaze to active the choice and confirm the choice via the clicker or gazing for a certain amount of time; audience can pick the gramophone using head gaze and perform “play music” action via the clicker or dwelling the gazing for a whole. Both interaction methods aim at virtual targets and lead to output in virtual world.

Fig. 7.

Interaction design in IARE (Solid line is the path adopted by The AR Journey)

As Coleridge stated that Narrative pleasure requires ‘willing suspension of disbelief’ [16], which refers to the willingness of the audience to overlook the limitations of a medium, making an essential contribution to the positive narrative experience. Murray addressed that the ‘active creation of belief’ in video game can be a substitute for ‘suspension of disbelief’ [17]. Therefore, to make the environment more convincible subjectively, secondary interactions and interactions with straightforward feedback should be included. This straightforward feedback, such as switching light, opening the curtain, pulling the drawer, etc., can be treated as active action to confirm the authenticity of the mixed-reality world. On the other side, narrative understanding describes how viewers make sense of or understand the narrative [2]. Story fragments and pieces of events are able to construct the narrative understanding.

4 Development

4.1 Coding

The programing was developed in the Unity3D game engine. HoloToolKit³ is used as the HoloLens SDK for Unity3D. The main challenge for coding was to develop the behaviours of characters and interactive props. There are two existing animation systems in Unity3D, the timeline system and the animator system. The timeline system, which is easy to cut, edit, and match different animation clips together with audio, is suitable for non-interactive character animation. The animator system, a finite state machine that contains different animation clips (states) and switches between different clips if the predefined conditions are satisfied, is suitable for interactive character animation. There are a lot of complicated non-interactive characters animations, e.g. Leo hugging his parents, and interactive character animations e.g. Leo finding and walking towards the user, and making eye contact with him/her. Therefore, it is important to combine the timeline and animator system. As Fig. 8 shows, a character behavior module was designed, mixing the timeline and finite state machine. The main idea was to put timeline animation and the finite state machine animation into three channels, known as playables in Unity3D, and use an animation mixer to mix the three playables with adjustable weight. The three playables are two timeline playables, which can dynamically load timeline animation assets and blend from one to the other seamlessly by animating the weight value, and a playable of animation layer mixer, which is the output result of the finite state machine animation. For the finite state machine part, a separate control of upper body, lower body and expression was achieved with avatar mask in Unity3D. In this way, the system can extract the lower body part of a walk cycle animation, the upper body part of a dancing animation, the expression animation of a greeting animation, and put them together. Besides, a layer of Inverse Kinematics (IK) animation with a weight value is added before the final output of the finite state machine part, which allows using a real-time changing target to drive the animation of the character. There are six IK goals including eye, head, chest, hip, hand and foot, the weight values of which can be assigned separately. In order to achieve a natural eye contact gazing animation of a character, the IK weight value of eye, head, chest and hip can be set to 100%, 70%, 40% and 20%. Moreover, the ‘hub and spoke’ pattern with an empty hub node is used as the main pattern to connect different animation clips. This pattern design can seamlessly blend any two animation clips of all the clips.

Fig. 8.

The diagram of the character behavior module

Props interaction is a crucial part for the overall user experience and storytelling, and there are many different kinds of user interaction, such as opening a virtual newspaper or picking up the receiver of a virtual telephone. In order to develop and manage the interactive props efficiently, a general module for interactive props was developed (see Fig. 9). In general, there are four states for interactive props: inactive, standby, active, activated. Inactive state is the default state to avoid false triggering. The standby state can only be entered when the distance between the prop and the user is less than a threshold and the prop is targeted by the user gaze. In standby state, hint animations are shown including glowing highlight, floating text and sound effects. The active state can be entered if the user further performs an action on the prop. In the active state, the prop animations like unfolding newspaper, rotating vinyl of a gramophone, opening a suitcase, accompanied by audio. Meanwhile, it may also trigger secondary animations including character animation and special effects. For example, after the gramophone begins playing music, Leo can turn to the audience and talk about his parents dancing in the living room with the music in the old days. When all the animations end, the prop automatically enters into the activated state. In this state, a new interface may emerge, e.g. headline news on the newspaper are displayed after the newspaper is spread; a new prop may show, e.g. a diary appears when the suitcase is open; or it may trigger the close animation automatically or manually and set the state back to inactive.

Fig. 9.

The diagram of the interactive prop’s module

4.2 Asset Development

There are four characters, 40 min full-body character animation, 15 props and their associated animations, several special effects and 20 UI elements required to be created for this AR app. The visual asset development follows the same rule as script development with regard to historical accuracy. Most references came from the National Holocaust Center and Museum and the online United States Holocaust Memorial Museum.

Character model and animation are the biggest part of asset development in this project. The character asset development is consisted of model part, animation part, sound design part and real-time asset part (see Fig. 10). The model part follows the latest Physically Based Rendering (PBR) pipeline in the game industry, including collecting reference, sculpting in Zbrush, re-topology in Wrap 3, cloth modeling in Marvelous Designer, normal map baking with Zbrush and Xnormal, texture painting in Substance Painter, and finally adjusting material in Unity3D (see Fig. 11). This pipeline is accessible and effective. It separates the shaping process into two independent steps, sculpting and topology instead of mixing them together as traditional way does. Moreover, Wrap 3 can efficiently complete the topology task as it can map a standard-topology character model onto the sculpting model instead of retopology manually. Cloth with natural folds is important to make a character visually believable and Marvelous Designer is the best tool which can precisely create cloth folds in a quick way. Substance painter is the most popular PBR texture painting tool, however, skin shader is different between Substance Painter and Unity3D. Standard material in Unity3D doesn’t support Subsurface Scattering (3S) effect and some third-party materials like LUX 3S material is not supported by Hololens hardware. Consequently, in order to make the skin as natural as possible, extra tweaking and adjustment need to be done for standard material in Unity3D.

Fig. 10.

The pipeline of character asset development

Fig. 11.

Pipeline of Leo’s model creation

Full body animation was capture with Vicon Mocap System in the Mocap Lab at Tongji University where we restored the layout of the rooms in Holocaust Center to ensure the holographic virtual character could dovetail the real-world space (see Fig. 12). The facial expression capture has been implemented with PrimeSense 1.0, a depth camera and Faceshift software. In order to use Faceshift, 51 expression morphers need to be made to meet the requirement of the internal expression manager of Faceshift (see Fig. 13). Students from the acting department of the university performed for facial motion capture to match the existing body animation.

Fig. 12.

Motion capture for scene 1(left), concept composition of mocap animation and living room (right)

Fig. 13.

Using Faceshift to perform facial expression capture

As to spatial sound, Hololens headphone can achieve spatial sound via Head Related Transfer Function (HRTF) technology, making the position and direction of an audio source recognizable in a real-world space. Spatial sound can strengthen the fidelity and direction of the sound effect, character’s voice and sound of props. Built-in 3D sound module in Unity3D and Microsoft HRTF Spatializer plugin can deliver the audio into headphones of HoloLens as spatial sound.

4.3 Lessons Learned

Key lessons learned from the development phase are as follows:

• Character animation merging and refinement issues. Motion Builder is the key tool to process the Mocap data. Motion Builder is optimized for rendering and modifying real-time character animations. For example, Motion Builder can easily handle a 15-min shot containing 3 characters’ animation and over 100,000 keyframes, which may lead other software crash or stuck. Besides, its built-in story mode can edit multiple character animation clips together with audios using a layered track structure as film editing software. Position and rotation of different animation clips can be re-aligned automatically based on the same bone, such as hip, foot, hand, etc., and animation clips can be blended by overlapping with each other on the track. Therefore, Motion Builder can merge and match the expression animation and body animation and voice acting quickly. Lastly, Motion Builder has a better integrated Forward Kinematics (FK) and Inverse Kinematics (IK) character rigging system, which can make the adjustment of the animation faster.

• Interactive path finding issue. Path finding is a common demand for interactive character. In this project, protagonist Leo need to find the audience and talk to them face to face. Unity3D has a Navmesh module for pathfinding, which can dynamically generate a ground surface for navigation based on the 3D scanning data from HoloLens, and character can avoid the obstacles and find the target on the generated ground surface. Though navmesh offers good result, it’s not a perfect solution as it requires extra computing. In order to save computing power, a simplified pathfinding module was developed using animator. It consisted of turning into the target, walking towards the target, and stopping in front of the target.

• Real-time shading issue. Due to Covid-10 pandemic, it is infeasible to conduct in-situ evaluation on site, thus we had to create the environment to facilitate an alternative evaluation off-site. And Fig. 15 shows the real environment and the real-time photorealistic rendering used for evaluation remotely. As HoloLens has limited computing power, it’s impossible to use complicated shader for interior scene and props. However, it doesn’t mean realistic image can’t be achieved with simplified shaders that only have diffuse and specular. Lighting, baked global illumination and post-process like bloom effect can help to build a realistic atmosphere with a simple diffuse shader (see Fig. 14).

  Fig. 14.

  

  A photograph of the physical exhibition room in the museum (left); real-time photorealistic rendering in Unity3D (right)

• Stabilization of the hologram in HoloLens issue. Hologram’s stability is a common issue in AR. Microsoft has developed and defined several terminologies to describe the hologram’s instability, including jitter, judder, drift, jumpiness, swim and color separation. In order to avoid the above issues, frame rate and frame-rate consistency is the first pillar of hologram stability. 60 FPS is the ideal and maximum frame rate for HoloLens, and 30 FPS or below can lead to obvious drifting or jitter issue according to our tests. Besides, a constantly fluctuating framerate is a lot more noticeable to a user than running consistently at lower frame rates. Furthermore, it’s important to choose the right spatial coordinate systems: Stationary Frame of Reference (SFoF) and spatial anchors. SFoF is a single rigid coordinate system suitable for a small room scale experience within 5-m diameter or moving objects like walking characters and floating UI. Spatial anchors are a multiple coordinate system suitable for world-scale experiences beyond 5 m, which can mark the important point in the real-world and each anchor stays precisely where it was placed relative to the real-world. In this project, spatial anchors coordinate system are used to stabilize the hologram, and these anchors can be persistent, which are stored and loaded back even when HoloLens restarts. Lastly, HoloLens performs a sophisticated hardware-assisted holographic stabilization technique known as reprojection. For HoloLens 1, Automatic Planar Reprojection technology is usually used to set a stabilization plane using information in the depth buffer. On Unity, this is done with the “Enable Depth Buffer Sharing” option in the Player Settings pane.

• Optimization for FPS issues. There are many possible reasons to slow down the program. The most effective treatments during our development include: 1) use release mode to deploy the program instead of debug; 2) set blend weight of skin to 2 bones if possible; 3) keep pixel shader as simple as possible (avoid to use standard shader) and use unit or vertex lit if possible; 4) use prebaked textures for shadows and lights instead of real-time; 5) keep texture size under 2048 and generate mipmaps; 6) turn off anti-aliasing and use bilinear filtering; 7) use instancing when ring multiples of the same mesh.

5 Interview and Discussion

After the development of the alpha version was completed, 9 people were invited to participant in the testing and interview. The testing was held in the Lab in Staffordshire University and the National Holocaust Center and Museum (see Fig. 15).

Fig. 15.

The Lab for testing in university lab (left), the exhibition room for testing at the National Holocaust Center and Museum (right)

The interview questions were as follows:

• Q1: Do you notice the characters or environment are unnatural? If so, what aspect is unnatural?

• Q2: Do you feel the interaction are uncomfortable? If so, what aspect is uncomfortable?

• Q3: How do you like the experience? Please describe it and give some examples.

• Q4: How do you dislike the experience? Please describe it and give details.

The main findings according to the common feedback from the participants include:

• The gaze & dwell input model is difficult to use for selection, as there is an issue of “observation vs. commit”. Because the FOV is narrow in HoloLens, users tend to put the choice in the middle of the view to read, which would accidently trigger the choice while reading. This is the reason why we prefer gaze & commit input model.

• The low framerate (<30 FPS) caused discomfort for 2 participants while others haven’t noticed this issue.

• 4 participants said that ambient sounds like raining, noise, street cars were too distracting and sometimes they couldn’t hear the characters’ dialog.

• 6 participants felt the FOV of HoloLens was too narrow and the edge of the display could be identified. It was strange to see the virtual character was trimmed by the edge of the display.

• One participant thought it was much more interesting to interact with virtual character like greeting and having eye contact other than watching a branched story.

• All participants thought it was interesting to have this mixed reality experience and to see these holograms of characters in a real environment.

• Several participants noticed that Leo can have eye contact with them and they thought it was interesting.

In summary, the feedback of the alpha version met our initial expectations. According to the interview, we could conclude that the overall experience was interesting even if the narrow FOV limitation was annoying. Moreover, gaze and commit were used as the input model other than gaze and dwell. Framerate was improved above 50 fps in accordance with official guidelines [18] and suggestions from the developer community [19]. The pitch of Leo’s voice acting was modified to make the voice younger, and it was later noticed that girls or women could perform a good job for a young boy’s voice acting if the boy actor of the right age couldn’t be found. A new sound manager module, which can adjust the ambient sound dynamically based on audience’s attention, was developed to solve the issues that ambient sound was sometimes too loud to hear the actors’ voice. It is also reasonable infer that characters’ full-body animation is more important than their material and textures according to the interview result. Therefore, it is important to assign more time to animation other than modeling and surfacing. Lastly, it is interesting that the narrow FOV can sometimes hide problems, e.g. audience usually tend to focus on Leo’s head and upper body while any problem of lower body is ignored as it’s usually out of the HoloLens display.

6 Conclusion and Future Work

The author developed an AR interactive narrative experience based on HoloLens and evaluated the programme with a small group of people. The overall feedback was positive. Besides, the development pipeline, interactive narrative strategy, input strategy, the design of the character and props module and AR related technologies were summarised. However, the evaluation was done with a small group via interview, more rigorous evaluation including quantitative methods with more subjects should be considered. Moreover, AR technologies are advancing very fast recently, during the development, HoloLens 2 has already published on the market while this study used HoloLens 1 to develop and evaluate. HoloLens 2 has great improvement in terms of FOV and input methods, which should be further studied in the future. Lastly, due to the lack of narrative work in the field of HMD-based AR, more empirical studies and explorations for interactive narrative in AR should be made in both industry and academia.