12.1 Introduction

Recent technological advances have increased the quality and accessibility of compelling, immersive virtual reality (VR) (Largent 2011), motivating its wider adoption in the domain of radiology and medicine in general. The ability to effectively and flexibly visualize segmented medical models as well as unsegmented image data makes virtual reality an attractive modality to complement a medical 3D printing program. This chapter presents an overview of virtual reality and its history, describes the current landscape of modern VR technology, and describes current and future medical applications including its relationship to 3D printing.

Virtual reality has been broadly defined as “a high-end user-computer interface that involves real-time simulation and interactions through multiple sensorial channels” (Largent 2011). Two hallmarks of virtual reality are visualization and positional tracking. The real-time visualization required for virtual reality has historically been achieved primarily through head-mounted devices (HMDs) that use small screens and lenses to cover the user’s visual field or CAVE Automatic Virtual Environments (CAVEs) that take the form of cube-like spaces in which images are displayed by a series of projectors (Burdea and Coiffet 2003). To relate the visual information being displayed to the user to a simulated virtual environment, the position of the user’s eyes (or head) must be tracked in 3D space. Full positional (six degrees of freedom) or rotational-only (three degrees of freedom) tracking have commonly been accomplished through the use of inertial monitor units (IMUs) (Burdea and Coiffet 2003), computer vision (Foxlin et al. 1998), laser-based tracking (SteamVR® Tracking 2017), magnetic tracking (Burdea and Coiffet 2003), or a combination of these technologies.

The terms virtual, augmented, or mixed reality have recently become buzzwords following the growing popularity of new consumer VR devices. These sometimes confusing terms are clearly explained and delineated by the concept of the reality-virtuality continuum first introduced by Milgram et al. (1994) and illustrated in Fig. 12.1. On one end of the continuum, there are environments consisting entirely of the real world: reality. On the other, environments that consist entirely of virtual objects: virtual reality (VR). Mixed reality (MR), then, is defined as a continuum between the two extremes where there is some combination of real and virtual environments—augmented reality (AR) being a subset. Augmented reality describes a simulation where the majority of the environment experienced is that of the real world, but with some amount of added virtual objects or environments. The less common concept of augmented virtuality (AV) describes a fully immersive virtual environment that has added elements of the real world (by using live video input, for example).

Fig. 12.1
figure 1

Illustration of the reality-virtuality continuum

Full size image

12.2 History of Virtual Reality

12.2.1 Early Milestones

While the concept of VR dates back to early science fiction writers, its history is rooted in the idea of an “experience theater,” described by Morton Heilig around 1950 (Burdea and Coiffet 2003). The focus of Heilig’s idea was a cinematic experience for users involving all the senses rather than just the usual 2D display with sound. Twelve years later, in 1962, Heilig introduced the Sensorama Simulator (US Patent # 3,050,870): an arcade-style device for a single user that featured displays of 3D video feedback (obtained by a pair of side-by-side 35 mm cameras), stereo sound, a moving chair, wind effects via small fans near the user’s head, and even odor producers. The Sensorama is considered the earliest archetype of immersive, multisensory technologies.

Heilig may also be the first to propose head-worn displays with his concept of a simulation mask. He was granted a patent for his concept in 1960 (US Patent # 2,955,156), which featured 3D analog displays encompassing the user’s periphery, optical controls, stereophonic sound, and smells. In 1961, Philco Corporation introduced their version of a headset device tethered to a closed-circuit television system that could be used by the wearer to transmit findings while navigating dangerous environments. However, it was Ivan Sutherland who is credited with producing the first example of a fully immersive head-mounted display (HMD; sometimes called the head-mounted audio-visual display). Released in 1966, and called the Sword of Damocles, Sutherland’s HMD used two cathode ray tubes to produce a stereoscopic display with a 40° field of view. The device was suspended from a ceiling-mounted cantilever—being too heavy to be supported by the wearer—which also tracked the wearer’s viewing direction via potentiometers. Sutherland later incorporated computer-generated scenes to take the place of analog images with his groundbreaking development of a scene generator that produced primitive 3D wireframe graphics. Introduced in 1973, Sutherland’s scene generator was capable of displaying 200–400 polygons per scene (frame) at a rate of 20 frames per second. These scene generators are the precursors to modern graphics accelerators—a key component of VR computer hardware.

Other important elements of immersive experiences followed shortly after the emergence of HMDs. In 1971, the first example of haptic feedback was demonstrated by Frederick Brooks Jr. and his colleagues. This development, as well as others, was incorporated into several iterations of military flight simulations in the 1970s and 1980s which was classified at the time. Other government agencies were also pursuing their own interests in simulators. In 1981, the National Aeronautics and Space Agency (NASA) created an HMD that used liquid crystal displays with optical controls to focus the images they produced close to the eyes. The initial NASA device was called the Virtual Visual Environment Display, or VIVED. Their successor system, called the VIEW for Virtual Interface Environment Workstation, was introduced in the late 1980s and boasted upgraded computer hardware as well as an interactive glove for manipulating wireframe objects that were spatially and mechanically tracked.

By the late 1980s and early 1990s, commercial VR systems began to emerge. The DataGlove, the same glove used by NASA’s VIEW system, was introduced in 1987 by VPL Inc. and was the first break from the standard keyboard and mouse computer interface tools. VPL Inc. was also the first company to release an immersive VR solution consisting of an HMD (called, interestingly, the EyePhone) that featured two LCD displays to produce stereoscopic images, each with a resolution of just 360 × 240 pixels. The HMD was used together with their previously released DataGlove, and their system was called the RB2 system (Reality Built for 2). It retailed for over $11,000.00, and the HMD weighed over 5 lbs. Nintendo later released an answer to the DataGlove in 1993, called the Power Glove.

While hand-worn and head-mounted devices were under development, other companies focused on improving VR hardware and software platforms. In 1991, Division Ltd. in the UK produced a scalable and integrated VR workstation to support their line of VR products. On the software side, the US company Sense8 in 1992 developed a library of VR-specific programming functions, called the WorldToolKit. This was followed by the Virtual Reality Toolkit (VRT3) software framework by Dimensions International in the UK.

12.2.2 Alternative Technological Approaches

While head-worn displays are currently considered the de facto standard for fully immersive VR and are the most practical technological solution for consumers, previous limitations associated with HMDs (e.g., weight) motivated the exploration of other VR system concepts. One popular example is the cave automatic virtual environment (CAVE) or its variations. A CAVE is a small room enclosed by whole-wall displays of virtual images produced by a series of video projectors. A stereoscopic 3D effect can be achieved through the use of positionally tracked active shutter glasses worn by the occupants and synced with the projectors. In active shuttering, the projected image alternates between the views for the left and right eye, while a shutter blocks the eye for which the view does not apply, producing a 3D perspective. CAVEs are commonly used in engineering, manufacturing, and construction industries to prototype designs.

12.2.3 Historical Applications in Medicine

The earliest applications of VR in medicine were centered around visualizing medical images and performing surgical planning (Chinnock 1994). Since then, medical applications of VR have expanded into the realm of medical education and training, facilitated communication (between clinicians or between clinicians and patients), and in a variety of therapies, including the treatment of phobias, PTSD, anxiety disorders, rehabilitation, and pain management. Interest in medical applications of VR has also been steadily accumulating. A recent search by Pensieri and Pennacchini (2014), for VR-related articles in the medical literature, uncovered nearly 12,000 publications as of 2012 using the most common search terms representative of VR applications in healthcare (but excluding “virtual environment,” “augmented reality,” etc.). Rather than focusing on the traditional applications of VR in medicine, the rest of this chapter will focus on the current landscape of VR technologies and how these technologies may be used to enhance the domain of 3D printing and the domain of 3D visualization in general.

12.2.4 A Technology Outpaced by Vision

Despite the pace of early development, as well as considerable amounts of media attention, VR companies in the 1990s failed to secure a widespread consumer base. Early systems were prohibitively expensive, with the fastest available graphics workstation by Silicon Graphics Inc. costing over $100,000, and were plagued with performance and reliability issues. As such, the VR industry remained small and largely contained to corporations, government institutions, and universities despite several attempts by the video game industry to generate interest in VR systems. Eventually, the rise of the internet claimed the public’s attention and, subsequently, interest in VR technologies waned when the few remaining companies failed to deliver on media hype (Stone 2006).

12.3 Modern Commercial Virtual Reality Technologies

12.3.1 Renewed Interest in VR

A new era of affordable virtual reality technology has recently emerged—driven primarily by the video game industry and enabled by breakthroughs in smartphone display technology, graphic processing units (GPUs), and tracking technology. VR recaptured significant public attention in 2012 largely due to the successful crowd-funding campaign for the Oculus Rift (Oculus VR, Menlo Park, CA) (Largent 2011; Kickstarter 2012). The campaign presented a prototype of a rotationally-tracked HMD using IMUs and smartphone displays. Following two developer kits and acquisition of Oculus by Facebook (Largent 2011), the Oculus Rift consumer version was released in March of 2016—consisting of a high-resolution, low latency head-mounted display. Six degrees of freedom positional tracking of the HMD is facilitated by a proprietary tracking system called Constellation which uses IMUs and optical cameras that track infrared (IR), patterned LED markers. Tracked handheld controllers were later released for the Rift in December of 2016.

While Oculus received the bulk of public attention throughout its development of the Rift, the emergence of modern VR technology resulted from the work of a number of players. One notable example is Valve Corporation (Bellevue, WA) who are credited with the development or discovery of a number of key components that facilitate immersive VR (e.g., the necessity of low-persistence displays) (James 2015). Following an early collaborative relationship with Oculus, Valve partnered with HTC Corporation (New Taipei City, China) to produce the HTC Vive—released 1 month after the Oculus Rift. The Vive was released with tracked controllers and uses a full room-scale, 360° tracking system called SteamVR® Tracking. SteamVR Tracking uses IMUs in conjunction with two “base stations” that regularly sweep the room with IR lasers (which are detected by photodiodes on the tracked objects) and boasts high-frequency submillimeter tracking accuracy within a 5 m corner-to-corner volume (SteamVR® Tracking 2017).

Together, the Oculus Rift and HTC Vive represent the first widely available, modern, PC-based, consumer VR platforms. However, the new landscape of VR devices is rapidly evolving with other offerings such as Razer OSVR, FOVE, MindMaze MindLeap, and Vrvana Totem which all present interesting technological variations (Largent 2011). With many choices available, and certainly more to come, early adopters of modern VR will likely be concerned with compatibility both now and in the future. To this end, Valve has made their SteamVR® software platform open to all hardware manufacturers through the OpenVR software development kit and application programming interface and have even gone so far as to freely license the use of SteamVR® Tracking so that any hardware manufacturer can make use of their tracking system (SteamVR® Tracking 2017; Lee 2017). The future of VR technology compatibility will also likely be greatly facilitated by the development of OpenXR: a cross-platform open standard for virtual reality and augmented reality applications and devices created in collaboration with a group of companies under the direction of the Khronos Group (Khronos Group 2017).

12.3.2 Mobile VR

Beyond advances in PC-based or “tethered” virtual reality technology, modern developments have also introduced a new domain of mobile VR driven primarily by smartphones. These devices take the form of custom lenses mounted in cases of various designs that hold compatible smartphones. Software is run on the smartphones themselves, and tracking—accomplished by relying on the phone’s internal IMUs or mounted IMUs—is generally limited to rotational (three degrees of freedom) only. Current examples of mobile VR at the time of writing are the Samsung Gear VR (Samsung, Seoul, South Korea), Google Cardboard (Google, Mountain View, CA) (simply a handheld cardboard shell with lenses), and Google Daydream (Wiederhold 2016).

Considering that the computational ability of smartphones is significantly less than that of high-end PCs and that mobile VR is generally limited to rotational-only tracking, the experiences available with mobile VR have been comparatively limited in capability to date. Despite this, mobile VR has already been used in medical roles such as anatomical education (Moro et al. 2017), ophthalmic image display (Zheng et al. 2015), surgical training (Gallagher et al. 2016), and patient education (Forani 2017).

With various classes of VR experiences available—from simpler mobile experiences to high-end PC experiences with external tracking systems—it is useful to distinguish between different levels of HMD-based VR experiences by the sophistication of their visualization and tracking. The most basic, perhaps, are 360° videos. These experiences are created from video recordings where a view in every direction is simultaneously recorded using an omnidirectional camera or a collection of cameras. The VR user then controls viewing direction with rotational-only head tracking (Forani 2017), and since the video is monoscopic and parallax is impossible, there is no perception of depth by the user. With more sophisticated video recording technology, 360° videos can be recorded with stereoscopic cameras adding the perception of depth to the video viewing experience. However, translation of head position is not reflected in the experience and interaction with the environment is not possible.

When the position and orientation of the user’s head is tracked in 3D space, the convincing sensation of being present in a fully immersive virtual space can be realized. However, this precludes the use of prerecorded video, and virtual experiences must now be generated in real time by a 3D rendering engine. Including tracked hand or controller positions increases the level of interaction available and creates an even more immersive experience (Cameron et al. 2011).

12.3.3 Augmented Reality

The new enthusiasm for virtual reality has also increased the attention given to augmented reality. This technology has recently taken the form of handheld experiences using smartphones and tablets where digital models are superimposed onto the real world (Moro et al. 2017); video pass-through headsets where forward mounted cameras are placed on the front of virtual reality headsets and stereoscopic video of the real world is superimposed with virtual images (Largent 2011; VRVana 2017; uSens Inc. 2016; Abrash 2012); and see-through glasses—most notably illustrated by the Microsoft Hololens development kit (Microsoft® 2017)—where virtual elements are superimposed on clear glasses or visors with additive blending (Largent 2011, Abrash 2012).

While augmented reality technology holds great promise for medical practitioners, and current solutions are being used by some groups (Cui et al. 2017; Weng and Bee 2016; Garon 2016), the communication from leaders in the field suggests it may be several years before augmented reality headsets see widespread proliferation (Brennan 2017). This is largely due to the current limitations and greater challenges that the technology faces compared to virtual reality.

For video pass-through AR, the experience is diminished by the fact that video has a lower dynamic range and resolution than real-world vision. Additionally, the eye is not free to focus on any part of the real world since focus is controlled by the cameras. The need to overcome latency introduced by capture, processing, and display of the real-world images can also be a challenge (Abrash 2012).

The challenges concerning perceiving the real world are bypassed in see-through AR methods where the real world is simply viewed directly. However, tracking for see-through headsets is generally accomplished through inside-out, computer vision solutions which introduce some latency, especially for mobile form factors. Since there is no delay associated with visualizing the real world, small lag in the positioning and visualization of virtual elements—which often must interact with real-world objects—is more easily noticed. See-through AR also faces the challenge of only being able to display virtual elements through additive blending, which means that visualization is necessarily translucent and pure black cannot be generated (Abrash 2012). Finally, current implementations of see-through display technology result in small fields-of-view for virtual element visualization, resulting in a limited ability to blend virtual elements with the real world in a convincing manner (Ren et al. 2016; Kreylos 2015).

12.4 Medical Virtual Reality and 3D Printing

Due to new levels of robust performance, accessibility, and low cost, the emerging ecosystem of modern virtual and augmented reality technologies promises to revolutionize the practice of medicine in ways that previous technological iterations did not. Modern computer graphics hardware allows for the real-time, fluid visualization of computationally intense medical data. New, cost-effective, and robust tracking systems open the door for intuitive human interactions with virtual medical models. Finally, advances in computer vision and holographic visualization technologies increase the accessibility of mixed reality tools for facilitating medical interventions.

While virtual reality has a rich history of being researched (see Sect. 12.2.3), until recently, medical VR applications have seen relatively limited clinical adoption. However, there is currently a booming interest in many different medical uses of VR. For example, the domain of medical training and education has seen a recent increase in publications (Matzke et al. 2017; Zilverschoon et al. 2017; Rahm et al. 2016; Hackett and Sttc 2013; Herron 2016). Much of what makes 3D printing attractive as a teaching tool can be applied to the visualization of medical models in virtual reality. What VR visualization methods lack in their inability to be interacted with as physical objects, they make up for in flexibility: animation, varying transparency, resizing, movable cut planes, etc. are all possible with the same sense of depth and 3D understanding that comes with handling 3D-printed models.

Virtual reality is also likely to make a significant impact on patient education. It has already been used to alleviate patient anxiety toward medical procedures (Forani 2017) and can be used, much like 3D-printed models, to explain pathology and medical details to patients (MediVis 2017).

Due to its ability to flexibly simulate the medical data related to patients or immerse clinicians in a realistic environment, there is a renewed interest in using virtual and augmented reality to improve surgery and surgical planning. Several systems for surgical training are currently available or in development (Osso VR 2017; BioflightVR 2017; 3D Systems 2017), and several systems for augmented reality-guided interventions are being researched or used (RealView Imaging Ltd. 2017; Baum et al. 2017) with many more likely to emerge in the coming years.

Of particular interest to adopters of medical 3D printing is perhaps the use of VR for medical image visualization (MediVis 2017; Surgical Theater LLC 2017; Cattin 2016; EchoPixel 2017; Vizua Inc. 2017). In contrast with 3D printing, virtual reality can be used to visualize unsegmented image sets through volume rendering (Zhang et al. 2011). Applying volume rendering techniques in VR is likely to be an active area of research in computer science since the computational requirements (two images for stereoscopy, high frame rate requirement, high resolution) for virtual reality increase the demands on what is already a relatively high computational load. More sophisticated volume rendering techniques (Dappa et al. 2016) will likely require modification or optimization before they can perform at a high enough frame rate for fully immersive VR. In addition to the realistic perception of depth and scale that virtual reality provides, the use of handheld tracked controllers allows for intuitive manipulation of medical images as illustrated in Fig. 12.2, which shows the use of a handheld visualization plane being used to interact with a CT-MRI fusion.

Fig. 12.2
figure 2

Example of interacting with volume rendering image sets (fused MRI and CT) using a handheld virtual plane to navigate through the image set in any arbitrary orientation

Full size image

VR can also be used to visualize segmented medical models. The STL or other object files generated for 3D printing take little to no effort to import into accessible 3D rendering engines such as Unity (Unity Technologies, San Francisco, CA) or Unreal Engine (Epic Games, Cary, NC). With VR system plugins for these engines being freely available, there is very little overhead for developing simple medical VR applications for research or clinical use. The flexibility that virtual reality provides when interacting with 3D models provides a useful parallel avenue to 3D printing for the clinical use of medical models (see illustration in Fig. 12.3), and a wide range of innovative and impactful VR applications will likely develop from this new creative space.

Fig. 12.3
figure 3

Example of interacting with medical models in virtual reality illustrating the benefit of controllable variations in transparency

Full size image

Virtual reality may well become a facilitator for future medical 3D printing practices. Recent software developments outside the medical domain have already shown a diverse number of examples of VR effectively facilitating sculpting and modeling (Oculus VR LLC 2017; MakeVR 2017; Brinx Software 2016) with the resulting models often being physically realized with 3D printing (MakeVR 2017; Brinx Software 2016; Strange 2017). It is easy to imagine that with the ability to effectively visualize 3D scan sets and intuitively manipulate 3D models, the medical model creation workflow could be greatly enhanced by virtual reality.

12.5 Conclusions

Previous iterations of virtual reality technology suffered from premature enthusiasm and mostly failed to live up to expectations. However, a recent confluence of technological innovations has led to a new environment of rapid development and growing adoption which suggests that, this time, VR is here to stay. Forward-thinking medical professionals would do well to pay close attention to what promises to be both a strong complement to 3D printing and a transformative technology in its own right.