Executive Summary

This chapter deals with the following topics:

  • Basics and advanced techniques of Virtual Reality

  • Providing insight into how engineers benefit from using Virtual Reality (VR) technologies

  • Describing functioning, benefits, and limitations of VR technologies in practice.

Quick Reader Orientation and Motivation

The intention of this chapter is:

  • to give an overview of VR technology in Virtual Product Creation as driver and enablers for Digital Transformation in engineering

  • to present VR technology as part of Virtual Product Creation from a practitioner’s point of view to analyze the need and usefulness for day-to-day industrial work practice

  • to give instructions on how to use VR technology

  • to explain models, frameworks, and mathematical representations that help to grasp the internal working modes of VR technology.

In 1995 Milgram et al. [1] asked the following question: What is the relationship between augmented reality and virtual reality? As an answer, they defined the Reality-Virtuality (RV) continuum as shown in Fig. 13.1. This represents a seamless continuum between a real environment and a virtual environment. The space between the two extreme environments is called Mixed Reality.

Fig. 13.1
figure 1

Reality-virtuality continuum as defined by Milgram et al. [1]

Full size image

The real environment, also designated simply as reality, is the concept at the left end of the continuum. This includes any environment that consists of real objects, regardless of whether a person is present in this environment or if it is viewed from outside, i.e. on a screen. Such an environment is always bound to the rules of physics.

The opposite right end of the continuum is called virtual environment and can be represented by the concept of Virtual Reality (VR). The virtual environment provides an artificial “world” in which the user is completely submerged. A virtual environment is an environment that consists of virtual objects that are simulated by computer graphics or specific simulation engines, which might emulate some aspects of the real physical world by means of digital models and algorithms. Such an environment, however, is usually not bound to all stringent rules of physics.

Any environment between these two extremes is called a Mixed Reality (MR) environment in which real and virtual objects coexist. Mixed realities can be subdivided into Augmented Reality and Augmented Virtuality. Augmented Reality (AR) augments a real environment with virtual objects. Augmented Virtuality (AV) adds real components to a virtual environment [1].

According to LaValle, the term Virtual Reality (VR) refers to the induction of a targeted behavior in an organism by means of artificial-sensory stimulation, while the organism has little or no awareness of an illusion felt [2]. He also defined the following four key components:

  • Targeted/desired behavior: the organism has an “experience” designed by the originator. Such an experience may be, for example, flying, walking or watching a movie.

  • Organism: potentially any kind of living being that can be stimulated by artificial stimuli and thus put into a virtual reality, be it a human, fish, monkey or a fruit fly.

  • Artificial-sensory stimulation: through the power of technology, one or more senses of the organism are hijacked and their normal input replaced by artificial stimulation.

  • Awareness: during the experience, the organism is unaware of the stimulation and feels present in the virtual world. This is accepted as normal.

This definition includes any artificial stimulation. This takes cases such as watching a movie, listening to music or looking at a painted picture into account. Today’s understanding of virtual realities, however, tends to exclude such cases and refers exclusively to those realities created by computer simulation. Sherman et al. [3] framed the following definition, which is here assumed to be the valid definition of Virtual Reality:

“Virtual reality is a medium composed of interactive computer simulations that sense the participant’s position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed or present in the simulation (a virtual world)” [3].

They also defined four key elements of a Virtual Reality experience: a virtual world/environment, immersion, sensory feedback in response to user input, and interactivity [3].

Immersion refers to the (subjective) feeling of a person to be in a fictional environment. This can be an environment that produces reading a book or watching a movie in the minds of readers or viewers. However, these areas do not allow the person involved to interact with the environment, which is why one speaks here above all of a mental state.

In Virtual Reality (VR), however, the user has possibilities for interaction, all-round vision and movement in the virtual environment. Therefore, immersion in this area is defined more precisely as a deceptive effect triggered by a technical, artificial stimulation of the sensors or sensory organs, which the user is not conscious of and which produces a real sense of the virtual environment. Immersion can be subdivided into mental and physical immersion in this area. The mental immersion describes the above-described state of strong involvement, the absence of doubt/disbelief, the perception of the environment as real. Physical immersion, on the other hand, describes the physical entry into a medium, the synthetic stimulation of the senses of the body through technology [3].

However, the terms differ in the bibliographical references found. For example, Slater describes immersion as only physical and he defines it as a measurable property of a system to the extent in which the sensory information corresponds to reality. On the other hand, he describes mental immersion as presence, a subjective feeling of the human that is triggered by their immersion in the virtual world. Thus, immersion or physical immersion enhances the mental immersion or sense of presence of a human in a virtual environment [4].

Another essential part of VR systems is the ability to interact with the virtual environment and its objects. Depending on whether the stimulation of the human senses can be influenced by the actions of the user, one distinguishes between interactive VR systems (closed-loop) and non-interactive VR systems (open-loop). In the case of a closed-loop system, the user has the ability to interact with the system, apart from using movements. For example, by voice commands, heart rate or body temperature [2].

Virtual Reality was invented by Ivan Sutherland who developed the first head-mounted display in 1968. 20 years later, virtual realities were first extensively introduced to research and film culture as an admired vision of the first digital revolution in the 90 s. With the use of a display in front of each eye the user was brought into an immersive virtual environment and was able to interact with it through the motion of their head. Because of the lack for computational power and small high resolution displays the technology has then been unnoticed for a long time. Immersive virtual environments were meanwhile made possible with high resolution beamers that projected the images on up to six faces of a cube around the users. This technology is called CAVE which stands for: Cave Automatic Virtual Environment. In 2016, the first consumer suitable HMDs reached the marked and major game engines like Unity and Unreal started to support them.

VR became more and more popular in recent years. Today it is used in all kinds of fields from engineering via games and movies through to medicine. Virtual Reality provides functionality to look at huge models such as airplanes, cars or other kinds of machines and other products in a 1:1 scale before they are built, to communicate with people who are on the other side of the globe with the possibility of spatial referencing or even to interact with the same object. Overall, Virtual Reality uses the three “I”s, immersion, imagination and interaction, to provide a realistic understanding of the virtual 3D models for the users (see Fig. 13.2).

Fig. 13.2
figure 2

The three “I”s: immersion, imagination and interaction

Full size image

13.1 Engineering Understanding of Virtual Reality

VR solutions provide powerful technology elements in the context of extended Virtual Product Creation scenarios. Engineers and IT experts integrate VR solutions into numerous industrial processes to achieve state-of-the-art digital connectivity and intelligent virtual support.

Primarily, VR technology for engineering is used in design reviews, production and assembly planning as well as in training. In general, VR can support tasks such as:

  • Interactive visualization interfaces to support engineers, workers and costumers during prototype reviews or planning activities,

  • Support of context-aware activities and of product user assistance as well as training,

  • Visualization of a product in a fully computer simulated environment,

  • Visualization of product and process-relevant virtual geometries and information.

Engineers are starting to appreciate Virtual Reality as a new convincing review and working environment where 1:1 scale representation of technical systems, products and manufacturing facilities can easily be comprehended and spatially be assessed. Designers and Engineers, however, are oftentimes still afraid how to become proficient in interacting and manipulating Virtual Reality scenes and, therefore, they often still depend on specifically trained Virtual Reality technology experts. Just recently, it becomes apparent that the future of engineering will need specific advanced VR support in order to better understand, study, analyze, assess and alter functional connectivity reasoning of technical systems and components as part of sharply rising model-based systems engineering and integration and data linkage engineering.

In addition to industrial VR applications, VR technology can add value in several further areas like sales (e.g. customer product presentation and configuration), healthcare (e.g. surgery training and support) and consumer market and culture (e.g. games and exhibitions).

13.1.1 Why Does an Engineer Use Virtual Reality?

An engineer uses VR applications e.g. to visualize digital CAD data no longer just on a 2D screen but in an immersive way allowing them to percept and interact with it in 3D and at original scale. This means that the engineer and other stakeholders can gain a more lifelike impression of the product before any physical prototypes have been created to help to comprehend the overall design and assembly situation of the product and to detect design issues earlier on in the product development process.

VR is also used as a decision-making tool to provide answers to questions in fields such as visibility and viewability evaluation, ergonomics and reachability analysis and review of the aesthetic qualities of the product. VR is also effective as a communication tool, which can be used for storytelling to explain use cases as well as a means to convey information between different disciplines such as engineering, marketing and design. One example at Bombardier is to conduct Virtual Reality based design reviews for train development as shown in Fig. 13.3.

Fig. 13.3
figure 3

Virtual reality based power wall design review at Bombardier with active 3D shutter glasses tracked by an outside in tracing

Full size image

Virtual Reality is especially used in earlier phases in the product development. It enables engineers in the concept, design and verification phase to e.g. conduct ergonomic studies, assessments if the product can be assembled, CAE flow studies and simulations and it can also be used as a communication platform to inform others about the current state of the development status (compare Fig. 13.4).

Fig. 13.4
figure 4

Circumstances for engineers to use virtual reality

Full size image

In summary, Virtual Reality is a technology that enables engineers and managers to experience the product, to validate the product in early stages and to use it as a communication platform to identify faults in the product.

13.2 How Does Virtual Reality Work?

The virtual world is an imaginary space or a description of a collection of digital objects in space and immersion refers to the immersion into an alternate reality or point of view. The sensory feedback is the feedback from the overall VR tracking system e.g. based on the user’s physical position and one form of the user`s interactivity may be the ability to control the computer-based virtual world. A VR-scene consists of different objects/surroundings that are three dimensional and the experience is different from normal computer screens as the depth information can also be perceived. The digital objects in a VR-scene are basically computer-generated graphics that are designed in advance (compare Fig. 13.5).

Fig. 13.5
figure 5

The virtual reality user is supported by the VR technology stack

Full size image

One very crucial requirement from VR technology is that it should provide real-time conditions. Only then, an experience comparable with reality can be obtained. From a visual perspective, a motion can only be recognized by the eyes of a human being if the frame rate of the screen can deliver at least 24–25 frames per second. However, if the user interacts dynamically as well, i.e. is not in stand-still situation, the frame rate for pictures delivered by the computer should be processed by the visual device with a frame rate of meanwhile 75–90 Hz. Why is that? The reason for a much higher frame rate (respectively a lower latency) is that a couple of different physical conditions of the images need to be distinguished with respect to the human eye and brain recognition: the refresh rate (of the same images) and the frame rate (of individually different images). According to [5] the critical refresh rate starts as slightly under 50 Hz and an absolute flicker free recognition for the human being needs a frame rate of almost 100 Hz. For HMDs the refresh rate plays a more important role which drives a fast reaction time of the HMD screen. The visualization latency, however, is just one contributor of the overall VR latency: in addition to the contribution of the human tracking latency, the VR scene simulation (computer) latency and the connectivity latency (of all sensors and the compute system) need to be considered and controlled meaningfully in order to avoid human recognition distortion.

In a VR Environment the HMD or the tracking glass is usually tracked by one or more cameras through infrared pulses. These cameras are connected to a computer which analyses the information and spots the two displays in the headset precisely. The matching stereoscopic perspective of the user referring to the location and orientation of his head can then be rendered and the video data is transferred to the displays with a cable or the information is transmitted wirelessly. This creates a three-dimensional impression for the user which has an immersive character if a frame rate of minimum 60 frames per second (meanwhile a higher frame rate is anticipated as explained earlier) and a field of view of at least 100° are used.

Furthermore, the VR environment can be affected by additional trackers and controllers. They can track the movement of the users’ body to display it in VR or the user can manipulate the environment in an interactive manner. Some HMDs also support eye tracking. With an infrared controller, the direction of the user’s eyes can be monitored in order to adapt the virtual world, save render power or direct additional mechanical parts of the HMD.

13.3 Virtual Reality Technologies

Virtual Reality requires a rich set of technologies and system architecture in order to provide a rich immersive and interactive engagement for the users with the 3D model scene. The following sections will step by step provide an insight to these technologies.

13.3.1 Setup of the Overall Virtual Reality System Architecture

A virtual reality system consists of different components: input devices translate and deliver the user inputs such as location of the user or interactions to the VR engine. Such input devices may be, for example, 3D glasses (passive or active), head-mounted displays and/or controllers with associated tracking system (see Fig. 13.6).

Fig. 13.6
figure 6

System architecture of a VR system

Full size image

The VR engine then creates the visualization of the virtual environment based on the current location and orientation of the user. Engineering software customizes the VR-application, for example for haptic and feedback calculations. These calculations can be used to feedback the forces to the input devices to allow real-time force and ergonomic validations.

In order to bring an image of a virtual environment to the displays of the headset, a real-time rendering engine is needed. Major Virtual Reality application providers have developed their own real-time rendering engines. However, from 2015 onwards the influence of gaming environments has also been increasingly notable. The two major applications for such engines are currently Unity and the Unreal Engine. They are both game engines that had already been widely used for game development before the introduction of VR. At the beginning of the VR trend both engines decided to broaden their user base by making their software usable free of charge if the user is without revenue through game sales. This way, engineers from various fields started working and researching with virtual reality scenarios. The support for individual headsets, controllers and trackers is implemented by open source SDKs or handled through official partners like SteamVR. For developing VR experiences and interactions Unity is using C# where Unreal is based on visual scripting and C++.

13.3.2 Head Mounted Displays, 3D Glasses, Projection Displays

For a long time, the stereoscopic viewing for the user was exclusively enabled by passive filter or active shutter 3D glasses that produced two separate images for the right and left eye. This was achieved while looking at a display that offered two images (one right and one left, compare Fig. 13.3). Head Mounted Displays (HMD) were long time too heavy to allow for convenient carrying and were used in specific applications in industry with limited success (too heavy, field of view limited to 100–110° maximum).

A head-mounted display (HMD) is similar to a pair of glasses or a helmet worn on the head display device. This may or may not allow visibility to the outside world, i.e. be transparent or cover the entire field of view. The first HMD was developed by Ivan Sutherland in 1968 [6] (see Fig. 13.7). In his accompanying publication “A head-mounted three-dimensional display” Sutherland described the mode of operation, whose basic concepts can still be found in today’s HMDs. Thus, head-mounted displays have two (one eye each) built-in screens that represent images of a virtual environment or virtual objects so that the user gets the impression of a three-dimensional space. A so-called tracking system allows the tracking of the position and orientation of the user’s HMD or head in space. By means of this information, the images displayed on the screens and thus the perspective of the user in the virtual space or on the virtual objects are adapted.

Fig. 13.7
figure 7

First HMD developed by Ivan Sutherland in 1968 [6]

Full size image

Since 2012, first modern HMDs were unveiled. Nowadays, many different models and approaches to VR headsets have hit the market. While the display of the first prototype (Oculus Rift DK 1) had a resolution of 1280 × 800 pixels, state-of-the-art headsets can now reach up to 7680 × 2160 pixels (Pimax 8 K). The increase from 640 × 800 pixels per eye to 3840 × 2160 pixels per eye makes the pixels almost imperceptible for the user. For the immersion the field of view is equally important. The first-generation HMDs in the 90s and 2000th used to have a FOV of 100–110° only, which corresponds to the stereoscopic binocular FOV of a human. Yet the borders of the screen are still visible in the two monocular fields of view. For a full coverage of the monocular visions a FOV of at least 200° is needed, which the newest models of Pimax already achieve. To bend the light according to the FOV typically different Fresnel lenses are used. The lenses are essential because the eye of an adult human cannot focus a display that is located three to seven centimeters away from their eyes. Virtual reality glasses are a form of HMD covering the entire field of view of the user. Such glasses allow the user through this technique to look around and move in the virtual environment analogous to the real world [3]. An example for such virtual reality glasses is the HTC Vive that was developed by the technology group HTC and the software company Valve in cooperation with each other. It has two built-in AMOLED 3.6 “screens (one for each eye), each with 1080 × 1200 pixels (together 2160 × 1200 pixels) and a refresh rate of 90 Hz. Together these screens offer the user a field of view of 110°.

In addition to HMDs, other technologies allow users to immerse themselves in virtual worlds. Such a technology is for example a CAVE (computer aided virtual environment). As shown in Fig. 13.8, a CAVE is a room in which three or four of the walls are illuminated by projectors in such a way that the illusion of a 3-dimensional virtual world is created for one person in this room who is looking through passive or active filter glasses [7].

Fig. 13.8
figure 8

CAVE examples: left: design review of a tooling machine; right: structure analysis of vehicle side door; bottom: 5-sided CAVE system (4 surrounding ones plus bottom)

Full size image

Figure 13.9 provides an overview of the different visualization techniques for virtual reality.

Fig. 13.9
figure 9

Different types of visualization technologies to enable virtual reality

Full size image

13.3.3 Tracking

In the early days the tracking of the PC-based HMDs and shutter glasses changed from inside-out tracking with build-in motion sensors to outside-in tracking with cameras. It was possible to make such progress because a higher tracking precision could be achieved with tracking cameras from outside. Today’s headsets usually use two tracking cameras that send and receive the infrared pulses with infrared LEDs and lasers. Nevertheless, the most recent developments from the HMD manufacturers are coming back to inside-out tracking because the VR setups can then be offered at a cheaper price. The tracking works with visible light and the base stations are not needed anymore. However, this means that the tracking will not completely work in the dark.

There are two different ways in which the detection of movements in the three-dimensional space of HMDs works (compare Fig. 13.10). With outside-in tracking, the position of the user in space is recorded via external cameras or signal transmitters [8]. As examples, the Oculus Rift CV1 or the HTC Vive can be mentioned here, since both systems require external hardware for position detection. The advantage of these tracking systems is the high tracking accuracy of six degrees of freedom and thus a stable measurement of the position in space. In contrast to this, there is the Inside-Out-Tracking. With Inside-Out-Tracking the HMD autonomously detects the position in space by internally installed sensors. This can be done based on stereoscopic camera images or by using sensors. Examples of HMDs with Inside-Out-Tracking are the Oculus Quest, or the Windows Mixed Reality glasses. The disadvantage is the lower accuracy, as well as a higher probability that the controllers required to use the HMD are not correctly captured. The advantage of this technology is that no additional tracking hardware, besides the HMD, is needed. It is also possible with this technology to use self-sufficient HMDs without connection to a computer.

Fig. 13.10
figure 10

Tracking technologies for virtual reality systems

Full size image

Latest tests of tracking systems for VR systems have revealed that there exists a wide range of tracking accuracy between 0.1 and 5 mm [compare 8]: outside-in tracking systems provide the highest accuracy with 0.1 mm whereas the range of inside-out tracking systems vary between 0.69 and 5 mm incl. observations of a position drift cause by the motion controller.

13.4 Human Interaction with VR

Four main functionalities that are necessary to interact in VR are navigation, selection, manipulation and system control to change the system preferences (see Fig. 13.11).

Fig. 13.11
figure 11

Interaction techniques in virtual reality

Full size image

Through various technical approaches, humans are able to interact with virtual environments. Early types of VR interaction used game controller devices that were already used for 2D games. For simple applications such as zooming, choosing & picking, selecting or discarding they fit well by offering simple motion and selection operation with e.g. a joystick or specific buttons (compare the over the air connected fly stick in Fig. 13.3 and a cable-based navigation and manipulation in Fig. 13.12).

Fig. 13.12
figure 12

Cable-based navigator/manipulator device in a VR design review

Full size image

As described by Stark et al. (compare [9]) new interaction research and demonstration solutions have been developed already in terms of modeling, sketching and designing technical products in Virtual Reality. The idea is to enable designers to directly model in true 3D space by offering different types of interactive tools as part of the VR system environment. Figure 13.13 shows a range of VR based interaction technologies which have been developed at TU Berlin and Fraunhofer IPK in Berlin, Germany. These CAVE or projection screen-based interactions offer an interesting mix of direct visible physical interaction devices while being immersed in virtual reality.

Fig. 13.13
figure 13

Interaction devices for sketching and modeling in VR CAVE space top: sketching lamp design with line and surface modeling devices; middle: application specific input devices with function selectors; bottom: interactive surface modeling devices and bare hand modeling based on camera based technologies

Full size image

However, the conventional and the tailored manipulators have the big disadvantage that the controllers are not visible while wearing a Head Mounted Display (HMD) and their position in 3D reality cannot be used. Special VR controllers for the consumer market solve this. They are tracked like an HMD and can be visualized as a virtual representation so that the user always knows their position and the software or system in which the controllers are used can also work with this position data. They can be used to point to things, control units only by moving them through the air and they still support the use of buttons and touchpads. With a simple (scalable) tactile feedback of the whole device, the user can receive a simulation of tactile feedback of their actions.

Even though this type of controller already enables simple intuitive gestures and gives the user the feeling of actively interacting with the world with their hands, the user is still not able to use their fingers naturally and individually. The entire hand is always grasped around a controller. The HTC Vive controllers have “grip buttons” on their sides, but they do not make a difference between grabbing and releasing the hand, because they have to be pressed much too actively and the controller still has to be held after ‘grabbing’.

The next generation of consumer VR controllers further develops this grabbing behavior. For instance, the Valve Index Controllers as well as the upcoming Pimax Sword Sense Controllers support the recognition of single fingers and can be strapped to the hand, so the hand can be released and the controller recognizes this by its sensors.

In addition to the controllers that are widely used, there are systems that enable users to bring their hands into Virtual Reality. These systems can be divided into optical Systems like LeapMotion or Intel Realsense and haptic or non-haptic data gloves (see Fig. 13.14). Whereas optical systems have the advantage that the user does not have to put on additional technical equipment, they do not provide stable tracking data. This can result in wrong hand and finger poses. Data glove on the other side can provide stable data but the user has to wear them in addition to the HMD.

Fig. 13.14
figure 14

Leap motion camera equipped HMD to track hands for VR based assembly planning; bottom: real person; top: virtual assembly scene

Full size image

Furthermore, with the improvement of AI-technology, voice interaction is also a powerful interaction technique. With this, users can e.g. in design review situations document their decisions by just speaking them and adding them to the part. Thus, one has a direct relation between the part and the decisions of the design review, which makes it easier to implement the changes afterwards.

A solution for better haptic feedback are the Tactical Haptics Reactive Grip Controllers. They do not support finger detection, though. Two plates on the grip complete the haptic feedback. One of them lies under the fingers and the other one under the palm. These get shifted a little bit up and down, mostly in opposite directions to simulate shear and friction forces. In addition, the controllers are able to vibrate and they can also stick together with different magnetic attachment points on their upper area. This allows the user to combine them for several use cases, it combines the two one-handed grips to one two handed grip or it generally emulates devices with two grips that are arranged in a fixed ratio to each other.

13.4.1 Development and Use of VR Applications

The development of VR applications is similar to those of traditional real-time 3D applications and mostly takes advantage of the same tools. The 3D interaction and perception constitute the biggest differences. This becomes apparent during testing VR applications. In traditional applications, a developer can typically use the same display for both development and testing and in many cases even the same input devices such as keyboard and mouse. In order to test a VR application properly, however, the tester or developer has to use a HMD or projection-based VR test environment and tracked controllers to closely resemble the experience of the end user. This also means that, depending on the hardware used, a tracking set-up may have to be present in the tester’s workspace.

Applying a VR application to a specific use case is, in many cases, a process that still involves staff support to operate VR hardware and software and prepare the VR scene as well as the model that should be investigated. Depending on the complexity of the use case the time required for model and scene preparation can vary. Especially if complex materials and textures have to be applied or if kinematic constraints have to be set up, the process can take multiple days. When physical devices are to be employed in the use case, the tracking of these also has to be set up and they have to be aligned with the virtual content.

13.5 Use of VR for Engineering Working Tasks

In engineering working tasks Virtual Reality is primarily used in product-decisive phases for virtual and hybrid prototyping (see Fig. 13.15). It offers the possibility to experience the geometric, technological, interactive and physical characteristics of a product as well as its special impression.

Fig. 13.15
figure 15

Engineering working tasks in virtual reality

Full size image

Engineers can walk around a virtual 1:1 scaled version of their future product, look at special parts of it, evaluate it and change certain (predefined) features such as scale, detail, annotations on/off, position and orientation, cut-off plane, culling etc. Therefore, new ideas and engineering issues can be visualized rapidly and functionality experienced early.

Furthermore, it is possible to review products remotely together with other engineers around the globe or to show future product ideas to customers.

Virtual Reality can also be used to train new employees. Thus, an experienced engineer can record a workflow and new employees can watch the recording later and train the process using the virtual version of the actual product, e.g. the machine.

13.5.1 Technological Limitations

One of the main limitations in VR applications is the fidelity and level of detail of the simulation presented to the user. These limitations are mainly caused by the display hardware and computing power available to the user. The resolution of most consumer level HMDs, e.g., is not high enough so that the user cannot notice the grid of pixels anymore, which decreases the sense of immersion. On the contrary, a higher resolution of the displays will increase the demand on graphics hardware, which can negatively affect the frame rate resulting in a decrease of immersion sense, too. To mitigate this, various methods are conceived. One of them is to take the optical properties of the headsets into account when rendering the scene to selectively decrease the resolution on the edge of the field of view. Another method is to project previously rendered frames again to artificially increase the framerate.

Other type of limitations to VR applications are present in non-immersive real-time 3D applications, which indirectly amplify and increase demands in resolution and framerate present in the final VR applications. This includes the geometric complexity of the scene, the visual quality of the rendering, physics simulations and collision detection. Especially when CAD models are to be rendered in VR, these limitations come into effect: compared to typical video game scenes CAD geometry has an extremely high level of detail and little optimization regarding its complexity which result in both, heavy demand on VR hardware fidelity and power. In addition, in a CAD context higher precision it is typically required in physics and collision calculations compared to games. Figure 13.16 provides an understanding of the trade-off necessary between geometry level of detail and level of real time interaction for VR applications.

Fig. 13.16
figure 16

Qualitative relation between type of VR application in respect to geometry level of detail over level of real time interaction

Full size image

Moreover, the size and weight of the HMDs or of other VR glasses constitutes another limitation to the user experience because these factors can cause discomfort in the user after extended periods of use.

13.5.2 VR Applications

Coming a long way from using an “expensive and difficult to use” Virtual Reality exclusively in research only or deploying it mainly for assessing 3D design shape and early virtual prototypes in industry, Virtual Reality technology and applications nowadays become step by step a day-to-day solution in engineering which will boost the understanding and interaction with 3D immersive models and other collaborative working meetings. The following sub-sections provide an overview of typical Virtual Reality application patterns.

Engineering Review Activities with Virtual Reality

The motivation to use VR for design and engineering review activities is to reduce time and cost since virtual models enable the avoidance of building the product or parts of the product physically with expensive tooling. Thus, the idea and goals for digital and virtual reality-based reviews is to study, analyze and evaluate if the reviewed product or parts of the product fulfill the rich set of requirements that were defined during the development project. To conduct an engineering review it is necessary that the relevant stakeholders participate in such meeting. Therefore, it is necessary that VR can be used in multiuser settings. The advantages of using VR in engineering review situations are manifold, e.g. it is possible to:

  • create and review photorealistic designs

  • review the product in a 1:1 scale

  • review different design alternatives in direct comparison to each other

  • include the product or factory line in the future environment which is embedded also as virtual model in the VR scene

  • conduct acceptance tests of product uses, factory workers and service personnel due to the high sense of immersion.

Peer-to-Peer and Team Interaction with Virtual Reality

As already mentioned, in Virtual Reality it is necessary that more than one user can participate in a design review. When using CAVE systems, it is possible that more than one person is in the CAVE and use appropriate glasses so that they can see three-dimensionally. However, only the user with the tracking target on the glasses perceives the right perspective. Thus, the other participates in the design review have to try to get the same perspective as the main user with the tracking target. This can be achieved by standing very closely together.

In addition to that, by using Head Mounted Displays, it is not possible for others users to see in 3D what is displayed on the HMD. Due to the small amount of cost and existing software, it is possible to use them in multiuser-settings. This means, that several users wear an HMD and everyone can see the same models, products or environments. To see who else is in the multiuser VR-session the users are usually presented by avatars. The advantage of a setting in everyone using a HMD is also that everyone has controllers and can freely navigate through the VR-environment (see Fig. 13.17).

Fig. 13.17
figure 17

Multi-user application of Fraunhofer IPK

Full size image

Consequently, contrary to CAVE and powerwall systems, the perspective is not limited to only one person. There exist several software applications, that support CAVE-systems as well as multiuser use cases, e.g.: IC:IDO, WeAre, STAGE, Virtalis and MiddleVR. Whereas Virtalis and IC:IDO are built on proprietary engines, WeAre, STAGE and MiddleVR are built on GameEngines, which enable companies to rely on existing multiuser technologies from the gaming sector. As an advantage, those companies can integrate new technologies quickly as soon as the GameEngines provide new features.

In summary, these applications enable companies to:

  • conduct meetings and workshops with long distances between the participants,

  • reduce cost and time to travel, as it is possible to share more information than just on 2D screens (PowerPoint) like in ordinary 2D desktop viewing applications,

  • increase the development time, as reviews can be conducted more quickly and with the relevant information which is needed to make decisions,

  • improve the efficiency for interdisciplinary teams in complex projects, as it is easier and more intuitive to present and describe the current state of the project.

Due to the distance between the users, however, it becomes necessary to address the topic of data security. Before using collaborative VR/AR software in business contexts it is mandatory to investigate which data are shared between the different users and how secure these data are. This is especially important for 3D-models. In the future more powerful computer hardware and streaming solutions will reduce the risk for exchanging 3D models prior to Virtual Reality reviews and might also help avoid the process of preprocessing 3D geometry in order to reduce the geometry level of detail.

Besides using VR only for design reviews, it is also possible to use Virtual Reality technologies in creative product design processes. For example, it can be used to design models of any kind dependent on the functionality and aesthetics (see Figs. 13.13 and 13.18).

Fig. 13.18
figure 18

Sketching at Fraunhofer IPK

Full size image

VR enables designers and all other users to design the products in a 1:1 scale in the environment immediately as the product will be used in the future. This is especially a big advantage in comparison to existing CAD or other design modeling tools (CAID or CAE modeler). The VR based systems are mainly used in early design processes to generate different design alternatives and first sketches. These applications address different use-cases. Whereas some tools are mainly used to generate sketches with primitive objects, other tools address freeform surfaces.

Next to creating and designing new models, it is also possible to use Virtual Reality for factory layout planning: Fig. 13.19 shows a use case in which a user can design a factory layout in Virtual Reality based on a construction kit. The construction kit has also integrated the degrees of freedom of each part of the factory production line. Using VR for the factory layout planning, enables companies to assess immediately the path workers would walk or investigate ergonomics. This use case has been leveraged within the publically funded research project VIB-SHP (“Virtual Commissioning with Smart Hybrid Prototyping”), compare [10].

Fig. 13.19
figure 19

Interactive VR-environment for factory layout planning [10]

Full size image

13.5.3 Summary of the Technology’s Benefits and Main Trends

In summary, Virtual Reality is a technology, which can be used in all phases of the product development process. The wide spread of use-cases as well as the low cost to start using the technology due the Head Mounted displays (in comparison to costlier powerwall and CAVE installations) leads to a more widely use in industrial applications. Nevertheless, one has to take into consideration that for different use-cases it is necessary to use different software or hardware tailored to the specific purposes.

Currently, the main trend is to increase the usability of VR-software so that it is easier for everyone to use VR applications. In addition, it is mandatory that the VR applications can be easily integrated into the product development process and into the IT infrastructure of the different companies. The latter is particularly nowadays an impediment to use applications that are based on game engines, as the data interfaces for industrial applications are still to be developed.