Keywords

1 Introduction

With the rapid development and advanced pace of state-of-the-art visualization technologies, Virtual Reality (VR) and Mixed Reality (MR) have been placed at the forefront of information visualization and interaction across a variety of domains. From an interaction and prototyping standpoint, this presents an opportunity for an innovative solution that draws upon existing methods and techniques to create novel immersive content. Storyboarding is a tool from the motion picture industry, which serves as a critical component in the film design process. Starting with a script, the director creates a sequential set of images, the storyboard, to depict a vision of how the film will look and be presented to the audience. Storyboards allow for the visualization of settings, event sequences, dialogues and even dynamic effects scenes and camera movements. Storyboards are typically drawn on blank forms containing several ‘frames’ along with scene name & shot number within that scene along with action notes or dialogue underneath. Depending on the artist’s talent, a sequence of images, dialogue text & descriptive action text can convey a rich story as well as the director’s technical intent. Storyboards are so effective at storytelling that they have evolved into a consumer genre of their own; Comics [2]. Comics are essentially refined and consumer-oriented storyboards with some parochial techniques. Considerable innovation within comics has led to a new media with broad consumer appeal. Using sequential imagery storytelling with pencils & paper, an entire film can be designed in enough detail for the crew & cast to produce most of its essential elements.

In Fig. 1, the storyboard for a short film entitled “Attack from Mars” [2] exemplifies all elements of a typical storyboard except for dialogue. It depicts scenes, their duration, and motion, which is often indicated with arrows. Dialogue, if present, would appear in the text beneath the image. In this example, six images provide the reader all that is required to know what the film will look like and how it will tell its story. It is important to recognize the benefits of this format; (1) it is intuitive, (2) it is spatially efficient, (3) it is easy to create with common drawing skills, & (4) it uses paper & pencil and does not require any technology or software to implement.

Fig. 1.
figure 1

“Attack from Mars” storyboard showing an image sequence relating a story.

Full size image

2 Prototyping Virtual Reality

The increasing popularity of VR has led us to the appraisal of new considerations for the VR/MR genre as well as new approaches to support the unique qualities of immersive & interactive 3D experiences. Films are typically a 2D medium where the Director has full control over the images that the viewer sees. Everything is centered around the single viewer perspective, so the storyboard concept has a good likelihood of looking like the final film product.

VR brings several new elements into consideration. For example, in VR, the user has control over their location and orientation within the virtual environment, making it possible for users to visualize story events from multiple potential points of view. However, this may present a complex temporal issue, as user exploration of a certain area, activation or a control may be required to start narrative elements within a VR experience.

Given that the user has control over their location, desired events may be occluded by barriers or at an unacceptable distance. The freedom of movement in virtual environments mean that scenario designers must be careful in their work, always mindful of the user’s freedom of movement.

Within VR, storyboards must be cognizant of the 3D spatial location of the user with respect to the virtual environment. It may also be necessary to show certain moments in time from multiple points of view. Currently, the consideration of multiple perspectives in VR storyboarding has yet to be standardized.

VR environments naturally represent highly immersive experiences and therefore require a variety of interactivity. The most basic interaction involves the user’s ability to change view direction by craning their neck or body. Many VR experiences will allow the user to move throughout the VR environment via walking, vehicles, flight or teleportation. Highly interactive scenarios may include the ability to interact with objects in VR [3]. To support these interactions, additional storyboarding elements may include any user interface or controls and how they function.

2.1 VR for Storyboarding?

The typical prototyping chain for VR starts with a concept, a storyboard and sometimes a 3D mockup, followed by construction of the VR prototype & then progression to a refined experience. Typically, storyboarding is done on paper or 2D drawing software as it is difficult to draw in VR [4]. This step builds a vision of the experience without reliance on technology. There have been VR storyboarding tools [5] which attempt to create a storyboard within VR under the assumption that more can be done within the media being targeted, but these applications found little success due to the technological reliance and inability to put the simple concepts and interaction order onto simple paper – the paper format is preferred for its superior ability to depict timing and flow. It has been shown that working in VR enhances a storyboard by giving more control of the camera and animated assets [6], though such tools are used to play with space and may be used to create flattened images to go into the traditional storyboard. The highest utility for those tools appears to be as a scene design tool after the storyboard phase is concluded. Additionally, storyboarding on paper can be extremely useful in providing design guidelines when introducing multiple modalities and controls and efficiently tying them to tasks performed within the experience. Thus, eliminating the need to build expensive prototypes, learn how to use VR storyboarding tools and extra time to revise and refine design requirements.

2.2 3D and Perspective View Storyboards

Three-dimensional storyboard frames (i.e. blanks) have been created, allowing for visualization of a scene with depth elements. Such storyboards show the VR environment and can also depict the user in relation to the surrounding environment. In Fig. 2, the user is facing a building, there is a building behind the user to the right and there are two horses behind and to the left, just outside the peripheral vision. The ground is a circular base with foreshortening to create a depth illusion. This base has three zones depicted in grey. The zone in front of the user represents their focal visual range while the side fields represent peripheral vision. The largest area depicts objects that are not seen unless the user turns their head or rotates their body.

Fig. 2.
figure 2

Perspective scene depiction for VR. The oval shape indicates depth.

Full size image

These depth-based depictions have the advantage of depicting relative positions of objects and the character within the VR scene [7] as well as the spatial relationships between objects. Sense of depth allows the viewer to mentally navigate through the space with a clear understanding of how scene elements are laid out and how the user may navigate the scene. However, the disadvantages of 3D views are that the views take up a lot of space on paper and tend to break the flow of action in the storyboard. For example, they may require putting in sheets that have one or two of these views on a single page. Such a view also deprives us of the detailed depiction from the user’s point of view (POV) and takes us away from the sequential action that relates action or narrative.

2.3 Combining 3D and Point of View (POV) Depictions

Storyboards have been designed [8, 9] to include both 3D and POV depictions on the same page to deliver the benefits of a 3D scene depiction along with a sequence of POV images. Figure 3 shows a scene depiction using a curved screen guide, demonstrating the users POV along with a guide for which elements will be in the peripheral vision. This approach succeeds in combining these elements and allows for several sets of scene/POV pairs to exist on a single page, preserving the sequential nature of the storyboard for the price of losing one third of the page’s frames compared to traditional storyboards.

Fig. 3.
figure 3

Combining Perspective & POV views [10]. In this example, 1/3 of space is lost to perspective views, though 3 scene frames & 6 POV frames will fit on a page.

Full size image

2.4 Annotating Traditional Storyboards for VR

Even with the availability of VR-focused storyboard blanks, the authors’ organizations continue to use traditional storyboard sheets for work despite the considerable volume of VR development our respective laboratories conduct. We have found that the space required by circular diagrams reduces the number of POV and other views available for action sequences, reducing the effectiveness storyboard, though as of now, we have no study results to back up that claim. Instead, we employ a technique to join multiple frames together to represent a single moment in time. In Fig. 4, we connect frames with a single line and cap the start & end of a single scene image sequence with triangular brackets. This practice lets us stick with old style storyboards and dedicate each frame to our desired use. We can use on for a map of the area, or as in this example, we have a POV view followed by side, top and perspective views. This simple technique is intuitive and allows us to dedicate as many or as few storyboard frames to the VR representation, as we desire. We could still make use of the spherical representation by drawing the shape into a rectangular frame. Complex scenes may still require a full-page design document to adequately depict the entire VR environment and all it contains.

Fig. 4.
figure 4

Traditional storyboard modified for multiple views per scene.

Full size image

3 Prototyping Mixed Reality

Mixed Reality (MR) and its subcategory Augmented Reality (AR) blends real-world elements with virtual objects and settings. In AR, information or characters visible through displays while vision of real-world surroundings is preserved. The broader MR category includes cognizance of the real-world to the point that the virtual objects interact or overlay real world objects, as they know their location and approximate shape. Real world elements in MR may include people, tables & chairs, floors and walls, for example.

In the example depicted in Fig. 4 with the woman facing a table with objects on it, we may assume that the room and the table are real world objects and that the sphere, cube and cone are virtual objects to be superimposed into the real-world space. How would that be best depicted? Our solution is to employ color in the storyboard [8]. We draw real-world elements in black & white. MR applications often assume some fixed elements to the real world to base the interaction on, such as a floor, wall or table. In this case, we assume presence of a table. The virtual elements we depict in blue. We might depict the user in orange to differentiate her from virtual characters (blue). If we wish to script motion, we might use green arrows. We employ red for user interface. Using a standard color symbology, our teams can review a storyboard and know instantly which elements are real-world, virtual, user or motion guides.

4 Conclusion

Time-tested pen & paper storyboarding principles are still the standard; especially when paired with new visualization, frame-pairing & depth indicator techniques. The effective methods are preferred, while technological solutions are optional. Such tech solutions may be considered if they provide proven benefits & time savings.