1 What is VR?

Virtual reality (VR), sometimes referred to as immersive computing technology (ICT), provides a unique way to interact with the ever-growing digital landscape. VR is often described as a set of technologies that enable people to immersively experience a world beyond reality.

A number of core VR technologies have arisen over the years that synergistically enable a person to experience a virtual environment. Display technologies come in a variety of modalities and sizes, each with a goal of delivering information to our senses, particularly sight, hearing, and touch. While smell and taste displays have received understandably less attention, displays for sight, hearing, and touch have progressed considerably.

Visual displays come in almost any configuration imaginable. Commonly, virtual reality facilities utilize one or more of the following: a single large projection screen (i.e., powerwall), multiple connected projection screens (i.e., CAVE® Cruz-Neira et al. 1993), stereo-capable monitors with desktop tracking, and head-mounted displays (HMDs).

Audio displays can be headphones, a single speaker, or a full surround sound system. Sound localization makes it possible to simulate sound moving or coming from a location within a virtual environment.

Interacting with a virtual environment is a critical component of many VR applications. Tracking systems of a variety of mediums (optical, magnetic, ultrasonic, inertial, etc.) enable the position and orientation of physical objects to be calculated within a physical space in real time. This becomes especially valuable when calculating the correct viewing perspective for the user. Coupled with gesture recognition algorithms, tracking systems allow natural body movements to be translated into functional interaction techniques (Mitra and Acharya 2007). Handheld controllers allow users to navigate and manipulate objects within the virtual world (Bowman et al. 2008). To enhance interactions, haptic devices provide force feedback through physical manipulators resulting in a stronger understanding of how objects in a virtual environment physically interact (Laycock and Day 2007). Other means of providing feedback to the user, such as vibration, wind, temperature, and pressure, can also be incorporated within the virtual environment.

At the core, virtual reality is a human experience. The technology is purposefully designed to take advantage of the human information processing system—to mimic how we interpret the world around us. As the famous Harry Houdini describes: “What the eyes see and the ears hear, the mind believes.” The technology supplants information from reality with that of the virtual world. Computer algorithms simulate the virtual world, displays render the simulation to our senses, and it is our minds that put the pieces together to form the experience.

When done well, virtual reality experiences convince users that they feel physically located within the virtual world or feel a sense of presence. Producing a sense of presence sets VR apart and takes traditional computing interfaces to the next level. While creating a sense of presence is not a requirement for VR applications, the sense of presence has emerged as a core differentiator receiving the attention of considerable research (Witmer and Singer 1998; Bowman and McMahan 2007).

1.1 Where did VR come from?

Both the hardware and software that enable VR have seen significant growth and adoption since VR’s conception nearly 50 years ago. The original vision was conceived by Ivan Sutherland in his 1965 essay, “The Ultimate Display” (Packer and Jordan 2002). In this publication, Sutherland describes a display that conveys information not only to the eyes, but to the ears, nose, mouth, and hands. He proposed a number of technologies, that had yet to exist, to support the ultimate display: 3D interaction devices, dynamic perspective rendering, haptics, and eye/gaze tracking. Sutherland states: “The Ultimate display would, of course, be a room within which the computer can control the existence of matter.” His vision set the stage for virtual reality research.

Over the next thirty years, technology matured with noteworthy prowess. In the 1980s, Jaron Lanier formed VPL Research, the first company to sell virtual reality devices. Others followed with software and hardware advances. However, it was not until the early 1990s that the necessary technical capabilities for Sutherland’s fantasy would begin to sprout. In 1993, John A. Adam published an article entitled “Virtual Reality is for Real” (Adam 1993) outlining the current state of VR technology. Adam described VR use at Caterpillar, Chrysler, Boeing, NASA, and research at a number of universities. He concluded that the performance was not sufficient to support truly immersive experiences, but because of the potential benefits, industry was starting to quietly investigate VR technology. Adams stated that, “much work is needed,” as the technology “almost works.” The mid-to-late 1990s saw growing interest with multiple surveys covering the increasing capability of the technology (Sturman and Zeltzer 1994; Bowman 1995). As the technology performance became more usable, interest in industry use of VR grew. Whyte et al. (1999) conducted a survey of the house building industry finding that the vast majority of participants surveyed believed that VR could be potentially useful.

It took only a matter of years for virtual reality to make noticeable adoption in industry. Fred Brooks, Professor of Computer Science at the University of North Carolina at Chapel Hill, reported on this adoption through an NSF-sponsored international survey of industry in 1999 (Brooks 1999). The publication, “What’s Real About Virtual Reality?”, provided an overview of production-stage, “users doing real work,” applications. He described his experiences at British Airways, Warsash, Daimler-Chrysler, and NASA. To close the article, Brooks described key challenges for VR researchers. First, the end-to-end system latency must be decreased to maintain immersive experiences. Next, complex 3D models (>1 million polygons) need to be rendered in real time. Finally, haptic simulations need to offer more realistic feedback. Brooks concluded that virtual reality had finally arrived and that it, “barely works.”

Since Brooks published his survey, research in virtual reality has flourished. Both the industrial and academic communities have contributed to a large knowledge base encompassing technical innovation as well as application-experience-based insights. As the technology developed more surveys emerged from the literature covering motion capture (Moeslund and Granum 2001), haptics (Varalakshmi et al. 2012), tracking systems (Rolland et al. 2001), tracking calibration (Kindratenko 2000), high-resolution displays (Ni et al. 2006), and presence (Schuemie et al. 2001).

While it has taken some time, virtual reality has been adopted in a variety of industries serving many needs. Numerous industries have employed VR to help design, develop, and evaluate early concepts before resorting to high-cost physical prototypes (PWC 2016). The medical community has made impressive strides in using the technology as a training platform to expose novice medical professionals to high-risk and difficult procedures (Liu et al. 2003). Architects and interior designers benefit from experiencing virtual spaces before construction (Mobach 2008). Simulating uncomfortable situations has been used to help treat a variety of phobias (Krijn et al. 2004). Complex abstract data can be explored and better understood through advanced visualization (Dam et al. 2000). All of these examples illustrate simulation of various situations where decisions need to be made. Virtual reality has enabled people to make decisions in completely new ways.

Although many technical challenges still exist, many that Brooks identified have been overcome. Today’s computational resources can render highly complex models at sufficient frame rates to support interactive displays. Position trackers are faster, smaller, and more accurate. Thanks to the commercial gaming community, virtual reality systems can now be constructed at much lower costs.

1.2 Motivation

We have arrived to a point at which virtual reality works. The supporting technology and software are mature, stable, and, most importantly, usable. Simply put: VR works! Therefore, the next step is to ask: How is VR being used in industry today?

There are several research publications that describe the potential of VR, or how research results in techniques and devices that support the use of this technology in industry, or compare the requirements of different software and hardware, but the authors found that a survey of actual industry experiences using VR was lacking. This paper describes the results of an NSF-sponsored research project with the goal to understand how virtual reality is being leveraged to empower industry innovation. More specifically, understanding how VR helps people make decisions is at the core of this work. Four motivating questions (Table 1) are the focus of this study. In an attempt to explore these essential questions, a survey of industry was conducted. The survey consisted of on-site interviews with VR practitioners. From the data collected, we can gain a clearer perspective on the current state of the art and use this knowledge to guide future research directions.

Table 1 Motivating questions
Full size table

1.3 Scope

What counts as VR? For the most part, the authors adopted the definition of VR put forth by Brooks (1999): “I define a virtual experience as any in which the user is effectively immersed in a responsive virtual world.” More specifically, the survey concentrated on applications in which real users are using VR for the benefits experienced (Brooks calls them “production applications”). Recently, Zimmermann (2008) describes use cases at Volkswagen in the context of the product design process of the automotive industry. The research presented here focuses on multiple facilities in many industries, concentrating on how VR supports decision making as part of the design process.

In defining the scope of the survey many decisions were made in an effort to focus the research. Unarguably, the entertainment and video game industries have paved the way for multiple low-cost consumer technologies, including virtual reality. However, as the goals of video game applications are often for entertainment, they will not be treated here. Furthermore, simulators have evolved in parallel but separately from traditional VR. As the literature has covered the success of simulators at length, they will not be included in this survey. Finally, training applications using VR were not the focus of this survey. The decision-making focus of the survey is defined as decisions that are made in the product design process, not the decisions made by someone in a training situation. This survey concentrates on how virtual reality is used to aid in decision making with respect to product design.

2 Methods

A survey of industry was conducted to better understand the current state of the art of virtual reality applications and technology in industry. Through a series of on-site visits to VR facilities, VR users and practitioners were interviewed.

To begin, a variety of methods were used to generate a list of industries using VR. First, the literature was reviewed to identify industries who have participated in past surveys, and by extension, were likely to participate again. Next, websites of major VR technology manufacturers were examined for customer testimonials and success stories. Personal knowledge and relationships between the authors and industry practitioners were also leveraged. Sometimes interviewees would suggest other VR facilities or organizations for the authors to visit (i.e., snowball sampling). The industry list evolved throughout the duration of the survey, eventually growing to a total of 50 VR facilities.

Once an initial draft of the list was complete, the authors reached out to personnel at several of the facilities by email or phone call. Once a response was received, a conference call was scheduled to discuss the details and determine whether the facility would be a good addition to study (see Sect. 1.3). Finally, an on-site visit was scheduled. These visits occurred during the fall of 2014 and extended into the spring of 2015. Only two companies participated solely through conference call interviews.

Each visit consisted of two parts. First, a tour and a demonstration of the facility were given. The demonstration provided the researchers with a better understanding of the context in which VR was used at each facility. Second, the researchers conducted interviews with participants. Depending on schedules, participants were interviewed individually or in a group. A semi-structured interview protocol with specific and general questions was used. Participants were encouraged to highlight important aspects of their VR experiences. This approach ensured that participant responses would help to answer the questions listed in Table 2, while maintaining the opportunity for unexpected themes to emerge. Researchers began with open-ended introductory questions and moved toward more specific inquiries. During the discussions, researchers captured participant responses and observations in notebooks. When possible, the interviews were audio recorded for later analysis. Some interviews were as short as 20 min, and some were as long as several hours; however, most lasted approximately 45 min.

Table 2 Interview questions (sample)
Full size table

3 Results

After each visit, the researchers met to discuss and summarize the interview results. Interview notes were integrated into cohesive documents. These discussions allowed researchers to develop a shared understanding of observations and participant responses. If available, audio recordings were studied and any new observations were noted.

A systematic coding procedure, described by Corbin and Strauss (2014), was used to analyze the data. For each VR facility, open coding was executed to conceptualize each section of the notes. Next, to identify themes across facilities, axial coding was done. Finally, selective coding leads to the results reported here.

The results of the study are presented in five parts. First, information about the participating VR facilities is presented in Sect. 3.1. Second, hardware configurations and popular software are reported (Sect. 3.2). Third, in Sect. 3.3, VR use cases are described in the context of decision-making categories. Next, stages of the current VR use process are detailed in Sect. 3.4. Finally, general insights and larger implications are described in Sect. 3.5.

3.1 Participants

In total, researchers reached out to 35 VR facilities and 25 responses were received. Over the course of the survey 18 on-site visits were conducted as well as remote conference calls with two VR facilities. A variety of domains were represented across the participating organizations (Table 3).

Table 3 Domains of industries
Full size table

Across all of the visits, 62 people were interviewed either individually or as part of a small group. At each organization, several people with differing responsibilities interacted with the virtual reality facility. The authors identified five unique responsibility categories: maintainer, operator, user, builder, and manager (Table 4). It was important to interview not only the managers but also the users. During analysis it became apparent that some interviewees interacted with the facility in several responsibility categories. In fact, most participants had multiple roles. Table 5 shows the number of people interviewed who had responsibilities within each category.

Table 4 Participant roles
Full size table
Table 5 Responsibility categories
Full size table

3.2 Technology and software

The survey revealed several types of VR facility configurations. CAVEs and HMDs were noticeably most common. Surprisingly, many facilities supported not only a CAVE, HMD, or powerwall, but often a combination depending on functional requirements. Table 6 lists the total number of configurations surveyed by type across all facilities.

CAVEs and HMDs typically track only one viewpoint, allowing one person to be the “driver” of the experience. In addition, with HMDs there is only one display so that others cannot see what the primary tracked person sees. We found that it is becoming quite common to share the perspective of the tracked user, whether in a CAVE or an HMD, by using a powerwall or larger television to mirror the user’s display and viewpoint (Fig. 1). This allows for better communication between the user, in the VR system, and other team members watching nearby. At one facility, a person stands in the CAVE wearing an HMD. Their position is tracked, and the CAVE walls are used as display systems for multiple perspectives and information. Both CAVEs and HMDs have been criticized for being a single user experiences; however, combining CAVEs and HMDs with auxiliary display systems can transform them into collaborative design spaces in which multiple users can efficiently interact.

Fig. 1
figure 1

Hybrid HMD and Powerwall system at John Deere. Photograph courtesy of John Deere

Full size image

The survey was performed during 2014–2015. At that time, the Oculus Rift was just becoming a commercial device. There was mixed reaction to the potential for this grade of consumer VR devices to impact virtual product design. Lockheed Martin, Boulder, CO, is just one facility that was exploring early models of the Oculus Rift as replacements for higher-end HMDs. VR personnel at Ford were less enthusiastic about the potential of the Rift because one of their key application areas relies on presenting highly rendered images, with little display lag, to the user in an HMD. These two separate views highlight the important need to match hardware and software to the anticipated specific use cases in order to achieve a successful outcome.

Optical tracking systems seemed to be the most common tracking systems. A few industries also used magnetic tracking. We were surprised to find that sound did not play a large role in many of the virtual experiences surveyed.

There is growing interest in portable VR systems that can be taken on the road. Value has been found in bringing the system to the users. Many utilize low-cost VR hardware including: stereo televisions, Microsoft Kinect, Oculus Rift, and Nintendo Wii Remotes. Figure 2 shows a portable system at Idaho National Laboratories in Idaho Falls, ID. This system consists of a portable screen, a short throw stereo projector, and optical tracking with the software running on a laptop. The entire system fits inside two carrying cases.

Fig. 2
figure 2

Portable VR system at Idaho National Laboratories

Full size image
Table 6 Hardware
Full size table

Software varied more widely than hardware configurations. While several facilities preferred authoring custom applications using VR toolkits, most facilities employed a 3D visualization suite to simplify the process of getting geometry into the virtual environment. Table 7 lists the most common software packages encountered during the survey. In most cases, facilities used one or more software packages depending on their needs.

Table 7 Software
Full size table

3.3 Real-life use cases

Throughout the interviews people were encouraged to share real-life use cases. Many commonalities and patterns emerged through storytelling. VR is being employed to enhance a variety of design activities throughout the design process. At some facilities, the use of VR is scheduled far in advance as an integrated part of the design process, while in other instances it is used to address issues as they come up (i.e., ad hoc). Many of the use cases participants described occurred during the conceptual and early phases of design, however, not always. While not an exhaustive list, the salient categories with specific use cases are described here.

3.3.1 Visibility/viewability

By far the most common scenarios described centered around evaluating the visibility of a human given a particular setting or posture. Cases in this category seek to answer the simple questions: What can I see? What is blocking my visibility? While it seems easy to evaluate visibility using analysis software on a desktop workstation, evaluating visibility during movement and interaction becomes a much more difficult challenge—one for which VR has shown to be well suited.

Many people from multiple automotive manufacturers told stories of evaluating driver visibility. As there are commonly three sets of pillars in most vehicles, it is important to understand how their size and placement influence the driver’s view of the outside environment. Having larger pillars may result in safer vehicles; however, it comes at the price of reduced driver visibility. We saw multiple use cases of designers using either HMDs or CAVEs and sitting in a virtual vehicle and moving about the space naturally to get a true sense of the visibility afforded (Fig. 3).

Fig. 3
figure 3

User wearing an HMD with perspective overlaid at Ford. Photograph courtesy of Ford Motor Company

Full size image

Engineers at the General Motors Design Lab investigate the influence of veiling glare from the instrument panels onto the driver's side window. Lighting algorithms have advanced to the point where light reflections can be accurately calculated and rendered. Sitting in their five-walled CAVE in a real vehicle seat provided a strong immersive experience. The display resolution combined with the lighting simulation was very compelling. Using this technology, designers can better understand how the instrument panel influences driver visibility during night driving.

Evaluating visibility became especially important at the Rock Island Arsenal during the design of a vehicle mounted gun turret. Soldiers were brought into a four-sided CAVE environment to test out the visibility of a gun turret atop a Humvee. Visibility plays a large role in keeping soldiers aware of their surroundings, so the view had to be right, while still maintaining a protective enclosure.

Visibility evaluation in the context of motion and interaction was also very common. Ergonomic engineers at the Ford Ergonomics Lab don an HMD to understand operator perspectives during a transmission docking task. At one point, manufacturing engineers wanted to shorten the studs (visual guides) involved in docking the transmission. However, shorter studs made them more difficult to see. VR provided an opportunity for engineers to ensure that the studs remained visible during the docking process.

3.3.2 Ergonomics/reachability

While seeing the environment is important, we found many cases where interacting with the environment was also important. Numerous interviewees described scenarios of how VR was being used to measure the impact of physical tasks on human operators. One ergonomics engineer summarizes with a question: “How’s someone going to posture themselves to do this technique [task]?”

At Ford Motor Co., ergonomic engineers are using VR to establish design criteria related to the maximum allowable assembly force to install various hoses. Armed with an HMD, physical props, and force sensors, ergonomic engineers estimated the forces required to install hoses given certain human postures. They used the results of their VR experience to set design specifications for external suppliers on the maximum force required for installation. Incorporating feedback from assembly operators is critical to successfully determine these specifications: “We bring the operator from that workstation in the factory into the lab. He buys off, so the transition goes smoothly.” Data gathered from ergonomic evaluations in VR are often used as design parameters for external suppliers: “Most studies are about setting a target.” Virtual reality in this context provides a method of ensuring people of many heights and strengths can complete assembly tasks safely.

At one of the Case New Holland VR laboratories, ergonomic engineers leverage a large stereo powerwall display to evaluate the reachability of door handles within a vehicle buck. During one design process, several suggested door handle locations were subsequently discarded because they resulted in uncomfortable and dangerous positions for the driver. With the large wall configuration, Case New Holland can also use VR to evaluate reachability of an exterior door handle from a ground-level stance (Fig. 4) for some of their larger products.

Fig. 4
figure 4

An author checking out the interior of a large tractor at Case New Holland

Full size image

Engineers at Caterpillar in Peoria, IL, looked at how a vehicle servicer might access filters in the context of guardrails. Immersed in a wide, four-sided CAVE, users try different ways of gaining access to the filter panel while avoiding the guardrails. Lessons learned from this activity are used to validate and/or modify the geometry to support easier maintenance, thus reducing customer maintenance costs.

Near Detroit, MI, designers and engineers at the Tank-Automotive & Armaments Command (TACOM) evaluate the reachability of instrument panel configurations during interior vehicle design. CAD models can be quickly rearranged within the virtual cockpit to find the best controls arrangement to support the task. Operators can move around the interior of the vehicle and evaluate the various configurations.

3.3.3 Packaging

VR has long been praised for its ability to communicate a sense of space within a virtual environment. A variety of stories have been told surrounding the use of VR to help plan the organization of large spaces. Whether it be a cockpit or a large room, controls and tools must be placed at logical locations to best support the underlying tasks.

Advanced Concept Engineers at TACOM described a scenario in which post-production vehicles needed to be retrofitted with newer equipment. With a team of engineers and soldiers, they evaluated different configurations of equipment in the context of real-life scenarios. Design meetings with slide shows often result in people asking: “Why did you put this here?” Then a little later when they were in the VR environment: “Oh, I understand now.” Another engineer expands: “[with CAD] They understand a little, but after being in here [CAVE] they understand more deeply.” Not only do they determine whether it will fit, but also whether the configuration best matches the tasks (Fig. 5). Early experience in VR influences how designers and engineers experience production-stage prototypes. One engineer described an experience of sitting in a production vehicle after experiencing it virtually: “I’ve been here, this is surreal...if I turn around I’ll see the radio, yep, there it is!”

Fig. 5
figure 5

An author checking out the driver’s seat at TACOM

Full size image

Designers at PSA Peugeot Citroën use a three-sided CAVE to investigate the potential placement of controls inside vehicle designs. Understanding how controls and instruments are located in the overall architecture strongly influences the cohesive feel of the interior.

Design engineers at the Rock Island Arsenal are charged with configuring portable maintenance shops for deployment on the battle field. They use virtual mockups to organize the mills, drills, lathes, and other tools within the confined space. Engineers put together multiple packaging options for tools and then virtually walk through tooling scenarios to ensure the workers have an efficient and safe work place.

3.3.4 Aesthetic quality/craftsmanship

Advancements in high-resolution graphic rendering have improved so much that it is now possible to evaluate an object’s aesthetic quality interactively in a virtual environment. Improvements in lighting and material properties enable a near realistic product to appear in virtual space. Several visualization packages offer photo-realistic renderings of 3D models.

Craftsmanship engineers at the Ford FiVE Lab use an HMD to understand the aesthetic qualities of 3D vehicle designs (Fig. 6). In one scenario, the rear seats of a car model were folded forward and the designers meticulously inspected gaps that might allow customers to see internal components. Or in another scenario, engineers asked: “We need to change the shape of the end cap on the instrument panel, what will that look like?” Vehicle models with different interior materials can be loaded into the virtual environment and compared for look, feel, and personality. The space between components also speaks to the craftsmanship. Engineers often investigate parts that, “don’t look like they talk to each other.” It is important to be viewing geometry at true scale to understand the impact of gaps. “Sometimes we get stuck at 1/10 of a millimeter [in CAD],” an engineer at Ford explains, “but it doesn’t really matter.” The HMD allows the designers and engineers to take on real-life postures and visual perspectives. As one engineer explains: “I sit in the car like it’s real. I can open the door, open the glove box, look under the seat.” Designs can be quickly changed and reevaluated virtually.

Fig. 6
figure 6

High-resolution rendering of a Ford Mustang. Photograph courtesy of Ford Motor Company

Full size image

Engineers at the General Motors Engineering VR laboratory stepped into their CAVE to take a look at the front grill of a truck. They wanted to know what could be seen from the outside: “Can I see the AC condenser through the grill?” after a moment, “Yep, we better paint that black then.” Their experience in VR allowed them to notice and fix visual aspects of the design that they may not have seen until production. The VR laboratory manager explained, “This [VR] has basically eliminated 3D prototyping...We can render this [model] without chips on the floor.”

3.3.5 Storytelling

Many of the examples thus far have concentrated on the design of a particular product. However, VR has also been used to tell stories in which a product is the main character. Often these scenarios are preprogrammed with specific viewpoints that can be accessed interactively and controlled during the viewing session. In this way, the lead storyteller can move the participants into advantageous positions to show a particular view.

Development engineers at TACOM described meetings with five to ten people in the CAVE where one engineer, who led the meeting, used animation and dynamic viewing to describe use scenarios of a proposed vehicle design. One moment the team was looking out of the driver’s side window and, with a push of a button, they were standing outside the vehicle while the lead engineer told the next part of the story. This setup allowed designers to communicate design intentions in context of real-life scenarios. A TACOM engineer describes the difference between desktop and virtual reality understanding: “you can put it on a 2D chart, but until you see it in VR [you will not understand].”

In a similar way, design engineers at Lockheed Martin Space Systems loaded up design concepts to virtually walk through possible assembly or maintenance situations (Fig. 7). Using a CAVE or set of wireless HMDs, engineers can simultaneously visualize and talk about interaction opportunities before having physical parts.

Fig. 7
figure 7

Users walking through scenarios at Lockheed Martin Space Systems. Photograph courtesy of Lockheed Martin

Full size image

3.3.6 Abstract data visualization

All of the scenarios so far have been visualizations of real-life objects. However, there are multiple cases when it is useful to visualize data that might not have a visual representation in real life.

A senior scientist at the National Renewable Energy Laboratory (NREL) demonstrated a VR application for analyzing wind wakes from turbines (Fig. 8). Scientists use the interactive simulation to better understand how wakes interact with each other. Leveraging this information can ultimately result in better turbine design and placement. In another demonstration, a world of red and blue dots appeared. In efforts of creating more efficient solar cells, a material’s morphologic properties are represented abstractly in the virtual environment. For this case, the data itself did not have any real-life representation. However, the visualization provided the scientists with rich qualitative information which they used to further investigate the space quantitatively. Immersive visualization can be very different than traditional 2D techniques on a desktop workstation; in some instances causing scientists to rethink the way they approach the data. “It’s a whole new paradigm.”, a laboratory manager explains, “They [scientists] have to untrain from using the desktop.” The visualization helped inform the scientific inquiry process.

Fig. 8
figure 8

Wind turbine simulation at the National Renewable Energy Lab (NREL). Photograph courtesy of NREL

Full size image

Lidar data is loaded into a four-sided CAVE at the VR laboratory at Idaho National Laboratories (INL) to understand changes in geospatial information (Fig. 9). Lidar is a remote sensing technique that uses light and radar to determine distances of objects. At INL, scientists use Lidar to capture spatial landscape data. In the CAVE, one set of Lidar landscape data is displayed as green dots. A second set of Lidar landscape data, collected at the same location but at a different time, is displayed using red dots. The superposition of these two landscape data sets allows easy visual detection of changes in the landscape over time.

Fig. 9
figure 9

Users investigating landscapes with lidar data at Idaho National Labs (INL). Photograph courtesy of INL

Full size image

Engineers at John Deere use virtual reality to explore computational fluid dynamics (CFD) analyses. Often, the large number of particles and stream lines displayed in the post-processing of the data make it difficult to interpret using only a traditional monitor display. Large screen virtual reality systems provide the ability to show the data on a larger viewing display, and position tracking supports the natural interaction of people as they explore this fully 3-dimensional data. Engineers remarked that they use the system when they are trying to interpret the analysis results themselves and also when they need to communicate the results to others to support decision making.

3.3.7 Communication across disciplines

People from various backgrounds and expertise are called upon to solve issues that arise during the design process. Communicating within and across disciplines presents many challenges. Each discipline uses its own communication tools, e.g., spreadsheets, graphs, data tables, 3D models, and other data visualizations. Getting a diverse group to fully understand a particular issue and contribute input from each perspective is a key to achieving a good design. Survey participants have shared several stories of how VR has influenced their communication processes internally with team members and externally with other departments.

At Case New Holland, people from engineering, marketing, and industrial design come together in the VR laboratory to communicate design goals and concerns. Each person on the team provides input on the design issue that is the focus of the VR session. The immersive facility enables people with varying design goals to communicate across disciplines within a shared experience. Departments can work together to understand the impact of form and function on product branding.

Another way that VR is used for communication is to provide a scenario for experts to communicate with managers who are not as intimately involved with the product as the design experts. Engineers at John Deere use the technology to demonstrate and describe design attributes to managers. They have found that seeing the design in full size and exploring some of the design issues using natural human motions provides a rich environment for communication compared to using more traditional methods of data communication such as CAD, Excel and PowerPoint.

Virtual reality technologies are being leveraged to investigate questions across a variety of categories. The limited set of categories presented here is by no means exhaustive. As VR sees further adoption in the future, the number of use case categories can only increase. Regardless of use case, many VR facilities follow a similar set of steps for the preparation and execution of VR reviews. That process is the topic of the next section.

3.4 VR use process

As the hardware becomes more reliable and the software more approachable, VR systems are becoming easier to operate. However, given the intricate interplay of the technologies involved, VR is still not a turnkey system. People may one day be as familiar with VR systems as they are the modern desktop workstation; however, we are not there yet. The process involves multiple support staff with varying abilities (Table 4).

During the interviews, participants were asked to outline the steps involved during a common VR use case scenario. Processes from all the participating facilities were modeled. Similarities and differences between the processes were extracted during analysis. When combined, the general process is shown in Fig. 10. While the process was rather general, the implementation of each step varied from company to company.

Fig. 10
figure 10

VR use process

Full size image

Step 1: VR request There were several approaches to initiating or scheduling a VR session. In some organizations, the use of the facility is scheduled automatically as part of the official design cycle. At the Ford ergonomics VR Lab, for example, ergonomic engineers must assess human effort in different postures through VR evaluations. More often than not, the facility is scheduled as needed when critical issues arise during the design process. Once the technology has proven useful, certain groups within the organization become regular users of the facility. “We don’t have to sell once they get here and see the value”. A VR laboratory manager at General Motors explained. Individual contact with the laboratory operator or manager seems to be a key aspect of scheduling the VR facilities.

Step 2: Model acquisition Once the VR facility is scheduled, a builder collects the relevant geometry. This is often accomplished by accessing the PLM system or through email. In some cases the full geometry of a subsystem is used, and the builder can easily obtain the entire assembly from a design database. In other cases, finding the exact specific models within the database can be difficult if the builder is not familiar with the particular product or naming convention. The builder communicates with the user to ensure the correct geometry is identified and collected.

Step 3: Model preparation Most people interviewed suggested that VR sessions require anywhere from several hours to multiple days for model preparation. Model preparation varies from company to company and from use case to use case. In the quickest cases, raw geometry can be converted and displayed in a Spartan environment within a few hours. If model conversion and enhancements are required, such as the addition of color, texture, material properties, and lighting, then the process can require multiple days.

The amount of model conversion seemed to be a function of how well VR was integrated within a company. The more established the use of VR was, the less preparation was required. For instance, at Caterpillar, CAD models are automatically converted to a file format that can be directly read into the VR software whenever the geometry is modified in the design process (multiple times a day). This automation helps reduce preparation time by removing the model conversion step from the rest of the process. Depending on the size and complexity of the CAD files, the conversion process appears to be the most time-consuming stage of model preparation. Many participants described having a conversion process involving multiple steps and file formats. For the majority of cases, getting the models into the correct format is only the first step.

Once the correct file format has been reached, additional steps may be required before the geometry can be loaded into the virtual environment. Based on the needs of the design review, a subset of the CAD model is selected and exported. Reducing the number of subassemblies helps users concentrate on specific concerns while in the virtual environment. The complexity (number of polygons) may have to be decreased depending on the computational resources available. A trade-off between graphic quality (fidelity of rendering) and real-time interaction must be considered when preparing geometry for virtual environments.

Step 4: Build virtual environment Once the models are ready, the virtual environment must be created. While some facilities preferred authoring custom applications using a VR toolkit, the majority surveyed reported using a commercially available software package (Table 7). Building the environment consists of multiple steps.

It is critical for the VR support staff to understand the session goals in order to build a sufficient scenario. When an ergonomics engineer at Ford is preparing geometry, he specifically asks: “What do you need to do the job?” and enhances the models to support the task. For many engineering scenarios, inquiries surrounding space claims, ergonomics, and interaction only require that the geometry is life sized and positioned accurately within the virtual environment. However, other VR practitioners are interested in questions that go beyond size and position.

Models that need to be manipulated freely must be identified ahead of time. Users at Boeing export models as independent files to ensure that they can be manipulated within the virtual environment. Other times models must be grouped so that they can be manipulated as a single model. There is a clear need to be able to select and manipulate any number of models in the environment interactively; however, not all software packages provide this feature.

Manipulating objects with full six degrees of freedom often makes sense for assembly scenarios; however, some objects, a car door for instance, are physically constrained to other objects. Adding kinematic constraints to VR simulations can be an arduous task. Using the desktop interface, a builder must manually select the manipulatable models, the type of constraint, and axes of movement. If the builder is unfamiliar with a product, they will need to reach out to someone else for help. “Because I’m not specific to any machine, I’m not an expert on it,” a builder at Case New Holland explains, “I’ll have to ask a question about how the movement is setup.” As larger products tend to have many moving parts, adding kinematic constraints to the experience becomes time-consuming. Finding a way to automate this process could greatly reduce the time it takes to prepare model interaction in virtual environments.

Engineers at Boeing, Ford, and Deere all use physical props to enhance ergonomic evaluations. Physical props, attached with smaller tracking markers, must be meticulously aligned to the virtual environment. One ergonomist discussed a complicated calibration procedure for a full body tracking suit. First, the optical tracking cameras are calibrated. Next, numerous markers are placed on the human subject. Each marker must be given an associated name in the software as part of a template system. To complete the setup, the software requires training through simple human gestures. Only after all these steps are completed can the ergonomic evaluation take place. At Deere, the process of aligning physical vehicle seats and controls within the virtual environment has been standardized and appears seamless. Multiple vehicle bucks are stored near the VR facility that can be quickly swapped in and out of the virtual environment during a VR design session if needed.

Outside of object manipulation, animating models also adds value to VR design reviews. Because many models move, it is important to see how that movement influences visibility and interaction with other parts of the product. Craftsmanship engineers at Ford investigate internal visibility with animations. The engineer explains: “We want to watch things as they open. You can’t see a clip when it’s open or closed, but when it’s opening you can see it.”

Builders require a clear understanding of the VR session goals in order to best support decision making. When the geometry is loaded and the interaction is configured, the virtual environment is ready to be tested.

Step 5: Proof-of-concept with user To ensure everything is configured as requested, the builder and or operator (often the same person) meet one-on-one with the user to demonstrate the VR application. If the user finds any issues, alterations are made before the final VR session or design review. This was a common practice across all facilities.

Step 6: VR session Once the user has approved the experience, the VR session is held. Before the users arrive, the operator prepares all the equipment and ensures the software is up and running. Participants reported that approximately five to ten people attend a VR session at a time. For larger groups, turns are taken to explore different parts of the virtual environment. Commonly, VR sessions are led by one or two users and normally last about an hour.

At Lockheed Martin, participants described CAVE reviews in which a small group of users would enter the CAVE and trade-off the tracked glasses while exploring the virtual environment. Other CAVE scenarios involved one person in the CAVE with others seated outside the CAVE watching the interaction. For the most part, HMDs were used by a single user or traded off between users during the session. However, at Lockheed Martin’s Colorado facility multiple HMDs were networked together allowing several people to experience the VR environment simultaneously within the large position tracked area. Ford has found significant benefits from a configuration where one person wears an HMD, while others could watch the external display to understand what the HMD user was seeing. At John Deere a user sits in a vehicle buck with an HMD with a large screen in the background (Fig. 1). The first person perspective and the third person perspective are projected on the screen. Multiple team members can stand in the area to view the projection display while interacting with the user wearing the HMD.

With HMDs, a tracking system is required; however not all of the powerwalls at these facilities had tracking systems. Some facility managers mentioned that position tracking detracted from the goals of the VR session. At a Lockheed Martin facility in Palmdale, CA, an operator sits in the back of the room and controls the virtual environment from a desktop workstation. The meeting leaders simply describe what they want and the operator makes it happen.

At this point operators and builders may be asked to make alterations to the virtual environment or change which geometry is loaded. The rapidity of changing the scene is highly dependent on the model preparation and environment building process.

Throughout the sessions it was common for users to take notes on notebooks or smartphones. The manager at the John Deere Product Engineering Center commented that their ability to capture screen shots of the scene interactively by pressing a button on the wand allowed them to accurately capture key findings from the VR experience. These images were later used to document the outcomes from the VR session.

Step 7: Outcome summary Documenting discussions and outcomes is an important part of every meeting. When the VR session was complete the user leading the meeting might send out a summary of their experience making special notes as to what needed to be done. Many times images and notes from the meeting are used to document the VR session and share with others not in attendance.

3.5 Key findings

Multiple themes presented themselves during data analysis of interview recordings and notes. High-level insights the implications are presented here.

3.5.1 VR is another tool in the toolbox

For many years VR has been praised as a panacea, a solution to many problems, while at the same time it has also been described as a solution in search of a problem. However, neither of these seem to be the case. VR’s use is strategic and calculated; it is a well-defined tool in a toolbox. VR allows people to make decisions about environments in increasingly natural ways. Specifically, it is used to create believable environments in which we can effectively make decisions about realities that have yet to exist, to predict future outcomes. A VR laboratory manager at General Motors explains: “To see them in the totality of the vehicle and to enable that decision-making process in the context of what it’s going to look like, either in the assembly plant or in real life at the dealer show room, this is the only way to do it.” Questions of spatial relationships are easily attended to in virtual reality: What can I see? How can I interact? How does this feel? Investigating human–product interaction is also a place where VR shines, especially when physical prototypes are unavailable. Viewing and interacting with objects at true scale seems to be one of the strongest affordances VR offers. Many participants spoke of finding immense value in seeing an object in real size within the context of the intended environment. As one participant at PSA explains: “Real size is very important, VR doesn’t make sense if we don’t have real size.” While not a good fit for everything, VR has found its identity as a valuable tool in the toolbox.

3.5.2 High frame rates are not always necessary

Being able to render the virtual environment at a high frame rate has always been a desirable quality of VR applications. Having a higher frame rate is particularly important for interactive simulations when there is a lot of head and body movement. A trade-off between geometry complexity and frame rate continues to exist. Application authors must ask themselves: Which is more important, geometry detail, or a high frame rate?

Many participants described building applications with this question in mind. Depending on the needs of the user, an application is authored with a detail-centric or interaction-centric (frame rate) approach. For instance, if a static viewpoint or particular perspective is to be evaluated, a lower frame rate may be sufficient. “Someone who is used to gaming will say ‘this is slow’,” an engineer at TACOM explains, “We’re not trying to react to things, or interact with things, we just want to show something.” In another use case, when users work closely with the geometry, manipulating and interacting with it, to investigate pressing design problems, they often get caught up in the task and find themselves unconsciously adjusting to low frame rates. “When people get involved in the context of the design, issues [like frame rate] tend to go into the background.” A VR manager at John Deere clarified.

3.5.3 Importance of high-quality rendering varies

The need for detailed renderings depends on the questions being investigated. The questions participants shared spanned a myriad of topics. Over the course of the survey, the questions users were exploring in VR often fell into two general categories: visual issues or functional issues.

The first group consisted of visual designers, craftsmanship engineers, industrial designers, and like-minded individuals. Questions from this group often surrounded the aesthetic qualities and styling of the geometry, paying special attention to the interplay between color, texture, and material properties. Having high fidelity or photo-realistic renderings was understandably important. Additionally, users in this group used VR to learn about how the form of the geometry gave it personality. Using VR with highly detailed models, designers and engineers in this group could investigate how a final product might look like in its entirety.

The second group was made up of more traditional engineers who concentrated on the fit, form, and function of the geometry, almost always from a product design perspective. Having high-fidelity geometry was not a priority for this group. However, in order to fully engage form and function questions, the geometry must be physically representative of the design with respect to size, orientation, and position. Questions surrounded how a product could be assembled or disassembled on the manufacturing floor. Simulating product function using animation helps to forecast future functional challenges. Ergonomic inquiries are better understood through life–size interaction between a human and the virtual model. Engineers were less interested in the aesthetics of the geometry and more interested in whether the models fulfilled the technical specifications.

This dichotomy was also articulated when it came to software. VR users needing high-quality renderings typically embraced a particular software, while engineering-centric questions might be better tackled with a different software package. The importance of high-quality renderings is strictly a function of the questions being explored. High-quality images are valuable, but not always required for decision making.

3.5.4 Starting a VR facility from the ground up is difficult

Many interviewees described the challenges in starting up and maintaining a VR effort in their companies. It appears there are two major hurdles that must be overcome before a VR facility can flourish.

First, while the cost of establishing a VR facility continues to decline, industry-level VR facilities require considerable capital to establish, operate, and maintain. Many of the interviewees described long processes of garnering internal support and buy-in from management. In many of the success stories, there was an internal champion who pushed for the adoption of VR. This person had a clear vision and strong belief in the value of VR and communicated that to management to eventually get buy-in. A second challenge arises once the facility is built.

Once constructed, there is the challenge of technology adoption within the company. To achieve adoption, the internal champion had to show the engineers and other users that added value could be obtained by using the facility. The initial question of “Why leave my desk when I can do all my work from here” needed to be answered. Humans, being creatures of habit, tend to avoid breaking away from established processes. A unique approach at one facility was to assign tasks to a “VR sales person.” Instead of relying solely on the internal champion to encourage use of the facility, another individual, a person with considerable design experience within the company, is part of a traditional design team but also has the responsibility of encouraging others to try VR. At this facility, he typically approaches his colleagues on an informal basis, coffee in hand, and suggests where VR can be useful. Perhaps after several interactions he might offer to take their geometry and set up an initial VR work session. Interestingly, he indicated that it only takes a single visit in a VR facility to communicate the value and benefits achievable to employees who are trying to solve design-related problems. This approach has been very successful in broadening the use of the technology throughout the company. At other facilities, it was clear that in order for the technology to thrive, the internal champion must bring others on board who believe in the value of the technology and are willing to promote it within the company.

Both of these challenges stem from a single underlying principle unique to VR. VR’s value is difficult to comprehend without experiencing it firsthand. Not only is it difficult to understand, but it is also hard to communicate. It is easy to describe an experience of brushing teeth because almost everyone is familiar with the activity; however, fewer are familiar with immersive VR. So, how does one describe the value of VR in the course of gathering internal support? In the ideal world, employees from one company could visit the VR facility at another company. Unfortunately, as VR is still an emerging technology, many companies keep their practice private to maintain competitive advantage.

3.5.5 A new career path is emerging

Virtual reality is unique in that it combines a set of diverse technologies to form a complex and powerful experience medium. The installation, operation, and maintenance of that medium require skills that are equally diverse. Currently, these responsibilities are carried out by multiple individuals with varying areas of expertise (Table 5). As VR continues to permeate industry the need for people with VR-specific skills will rise. Knowledge and experience with displays, tracking systems, audio systems, and interaction technologies help form the technical background needed to manage VR systems. Beyond the hardware, knowledge of software and virtual environment design is critical for developing effective 3D worlds.

3.5.6 Measuring return on investment is worthwhile

Keeping track of facility usage can be a useful way to estimate return on investment. Several VR facilities surveyed implemented extensive spreadsheets to record laboratory activity. The spreadsheets captured a variety of information. Recording the who, what, why, and when of VR sessions, however, is not always enough. The most detailed record systems estimated potential gains based on findings and outcomes. As finding issues early in design is a core goal of VR, it is also important to calculate cost avoidance. At Lockheed Martin Space Systems the impact of specific findings is estimated. “This is where it gets a little tricky,” the laboratory manager explains, “there is always the question of - well would I have found that issue if I didn’t use VR?” Usage records together with prototype cost data were used to justify upgrade costs to multiple facility technologies at General Motors. Showing, with records, that VR has saved both time and money is a proven method of increasing confidence in the technology.

4 Closing Thoughts

The last twenty years have seen remarkable strides in the advancement of virtual reality technologies. John Adams wrote that VR, “almost worked,” in 1993. Fred Brooks reported that it, “barely works,” in 1999. Since 1999, virtual reality has flourished thanks to rigorous efforts of academic and industry research communities. VR can be found bringing value and benefits to a myriad of disciplines. It works!

This survey has uncovered a variety of real-life examples of how VR is being used to support stronger understanding through immersive experiences. Facility configurations are as diverse as they have ever been. VR practitioners in industry are molding the traditional definitions of VR in search for the best immersive experience for their needs. VR has shown to be a strategic tool in the toolbox ready to tackle a variety of challenges. While not suited for every problem, this research shows VR’s strengths at investigating a variety of questions pertaining to: visibility, ergonomics, packaging, aesthetic quality, abstract data visualization, storytelling, and across discipline communication.

Table 8 Brooks’ research challenges (Brooks 1999)
Full size table
Table 9 Research challenges
Full size table

4.1 Moving forward

There are considerable milestones to reach before Sutherland’s iconic vision for the ultimate display can be realized, but the community has taken a leap in a promising direction. At the conclusion of his survey, Fred Brooks proposed a number of open research challenges. Many of the challenges have been overcome, while others remain. Table 8 lists the challenges that Brooks outlined in 1999 and summarizes the current state of technology advancement.

To close each interview, participants were asked about their hopes for future VR technology, applications, and experiences. Responses varied from enhanced technological capabilities to ways to streamline the VR use process (Fig. 10). Participant feedback was categorized. Major themes are summarized in Table 9.

The recent explosion of affordable consumer-grade VR hardware coupled with powerful software gaming engines holds great potential for the future of VR in industry. The development of high-performance graphics cards, better tracking devices, better HMD optics, etc., which has been a result of the drive to produce consumer devices has contributed to the overall advancement of hardware and software for VR all across the board. The price point of the Oculus Rift, and other consumer HMDs, may open the door for many new facilities to explore the use of VR for product design and manufacturing. However, the demands of some successful industrial applications of VR may not be fulfilled by a VR device highly optimized for consumer gaming. As these devices begin to move into industry, the applications that are successful and those that are not will begin to emerge.

It is important to also recognize the affect consumer VR will have on the skills of people entering the industrial workplace. As VR becomes just another gaming tool for the general public, more and more people will come to the workplace with VR experience and knowledge of the capabilities of VR. It is not inconceivable that the next generation of assembly line workers and engineers will look at a 3D CAD drawing and ask where the VR hardware is so they can view it immersively.

4.2 Summary

This paper reports on a survey of virtual reality in industry. First, a list of VR facilities was generated. The authors reached out to 35 VR laboratories through phone and email. In the end, 18 on-site visits were conducted along with conference calls with two other VR facilities. In total, 62 people from varying disciplines and backgrounds were interviewed. Real-life use cases are presented within overarching categories and high-level themes detailed. New updated research challenges were outlined for the community. Results indicate that VR has flourished in the last twenty years. The knowledge base has been expanded greatly thanks to VR professionals from both the academic and industrial communities.