Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

How would a driver react if faced with a hazardous situation? When is it best to alert the driver about unforeseen danger? What is the best way to interact with the driver in order to achieve the desired reaction? Is it possible to develop a near-realistic driving simulator? The above questions can be answered by analyzing the driver’s behavior within a driving situation taking into account other drivers, the road conditions, and the car dynamics. This is important since human error is one of the principal factors that lead to road accidents, and is attributed to increased mental workload, mostly caused by distractions such as operating devices or eating while driving. The most effective way to understand how drivers master situations which could lead to an accident would be to place them in that same situation. Placing drivers into a real driving situation in order to evaluate their behavior is too dangerous whereas testing environments such as crash test environments are very time and cost intensive. Therefore, driving simulators are commonly used for research purposes in monitoring the driver’s behavior and for design and evaluation of new Human–Machine Interaction (HMI) concepts. Mostly Two-Dimensional-Driving Simulation (2D DS) environments are used in the automotive industry and for automotive research. These conventional driving simulators often lack the immersion of the driver into the driving scenario, and hence do not reveal the natural reaction and interaction required to understand the drivers’ behavior.

Emerging technologies in the field of Virtual Reality (VR) from the area of consumer electronics and increasing processing power offer potentials for new highly immersive Driving Simulation (DS) concepts, for example in the development and evaluation of new Advanced Driver Assistance Systems (ADAS).

Most of the leading automotive manufacturers use VR in various phases of their development process for example; Ford, for interior and exterior design (Ford 2015; Howard 2014), Audi, for car configuration (Robarts 2015), Renault for research and development (Renault 2014), Lexus for test driving of virtual prototypes (Eedelstein 2014) and BMW for car development (BMW 2003), just to name but a few.

VR DS with a Head-Mounted Display (HMD) such as the Oculus Rift or the HTC Vive provides a different DS perspective to conventional DSs used in the development of automotive products. Users feel present in the computer-simulated environment due to the wide field of view, stereoscopic 3D effect and extremely low latency offered by novel HMDs. This leads to the sense of presence which means that the test driver is physically and mentally involved in the driving experience for a natural reaction and interaction with the system. Therefore, it is possible to collect and analyze data from complex and critical driving situations in a controlled environment without endangering the life of the driver or destroy the prototype.

In order to realize a driving simulator with a HMD, the device has to be integrated into the control loop of the DS. In most cases, a driving model is implemented, which consists of integrated traffic scenes and real driver input hardware like gas pedal, brake pedal, and steering wheel. A VR driving simulator as basis for future automotive HMI applications is expected to solve the problem and challenges ADAS researchers’ face of scientifically proving the reliability and safety of HMI concepts. This is due to lack of proper analysis at the early stage of development before the physical prototype is developed. The virtual prototype enables fast prototyping and early analysis of a concept to unforeseen circumstances without waiting for the physical prototype. The quality of the product and hence, a better user experience is also expected to improve considerably. This is because it is a user-oriented development where considerable number of variants can be shown, modified, and assessed at a very early phase of the process, thereby reducing the time and costs required for the overall process (Volkswagen 2015).

1.1 Motivation

The level of immersion experienced in real time using VR devices such as head-mounted displays due to the stereoscopic three-dimensional (3D) effect and the wide field of view is expected to enhance development of driving safety functions. Therefore, a DS with integrated HMD would facilitate rapid prototyping and introduction of new HMI-features since all changes could be made directly on the virtual prototype until expected result is achieved. This would serve as basis for a physical prototype. This is very important in the automotive industry with high level of competition, need for cost-reduction, and globalization, and most especially as vehicles are getting highly connected and interactive giving room to more driver distraction concerns.

Another benefit of 3D DS development is flexible, user-oriented adaptation of functions and HMI components in the context of a recursive, user-centered development for an enhanced user experience. Implementation of new ADAS, for example, an adaptive-cruise-control (ACC) or adaptive collision-control could be rapidly and flexibly evaluated using a complete VR approach with regard to usability and acceptance without the use of real hardware which are usually very expensive. Some VR DS systems are composed of only the Logitech steering wheel and pedals, 3D models, sound systems, and a HMD, the surrounding environment are all virtual objects. DS in a 3D environment with space depth and reliable distances offers the user an immersive opportunity to drive in an environment almost similar to a real driving vehicle. This environment is safe and risk free for the test driver and the prototype.

Furthermore, best tools for developing and interacting with 3D applications could be discovered through reviews and benchmarks. Likewise, solutions to tackle the unpleasant effects of simulation sickness could be disclosed, implemented, and evaluated. Finally, through the integration of a real car and a motion platform to simulate motion cues, user tests of new concepts could be performed in an ergonomic environment similar to reality, the limitations of the automotive concept could be disclosed, and effective users’ feedbacks collected for future work in eliminating these limitations on the physical prototype and subsequently, the actual product.

1.2 Overview

Section 15.1 introduced this work, describing the motivation and a brief description of the purpose of a VR DS. Section 15.2 discusses the state of the art of the key technologies in a VRDS. A historical background, characteristics, and applicability of VR and DS in various industrial sectors is described in detail. This chapter is finalized by a short summary of both technologies.

Section 15.3 describes the key points in the realization of a VR DS, the concept, tools, and implementation of a prototype. This chapter also provides a detailed description of the steps required for the development of the simulator. These steps include; VR Simulator concept, modeling of high resolution 3D objects, prototype development, and interaction concepts. This is followed by the selection of an appropriate motion platform for the simulation of motion cues.

Section 15.4 illustrates the shortcomings and limitations of simulated environments. The limitations of systems using VR and DS technologies are discussed and suggested solutions stated.

Finally, Sect. 15.5 presents the conclusion of this work. This section also states the challenges and limitations faced by automotive components suppliers regarding reliability of the developed applications and also the limitations connected with the use of 2D DS, and why VR technology could be a better solution.

2 State of the Art

2.1 Virtual Reality

VR as it is mostly called is a 3D multi-sensory highly interactive artificial environment which gives the user the sense of spatial presence through total immersion (VirtualReality 2009; Ni et al. 2006). The feeling of immersion could arise as a result of the user feeling isolated from the real environment but being part of the simulated environment, and being able to interact and manipulate virtual objects in a natural way which provides the illusion of moving in the virtual environment. Depending on the level of immersion experienced, the sense of spatial presence could be achieved through multiple sensory channels interaction and stereoscopic 3D effect. The sense of presence in the virtual world is reached, when the user perceives the virtual world as real and believes to be in a different location rather than the actual real environment.

VR is an old technology that originated from research on 3D interactive graphics and vehicle simulation in the 1960s and 1970s. The term VR is associated with the following terms; computer-generated graphics, virtual environment, stereoscopic 3D, real-time tracking, highly interactive, and multiple sensory channels. The virtual environment could be real for example the simulation of a real flight cockpit for training pilots, or an imaginary flight cockpit for gaming.

VR is classified into various categories based on the level of immersion it offers. These ranges from Non-immersive for example desktop visualization of 3D objects, Augmented VR for example Head-up Display (HuD) to Immersive VR achieved with the help of a HMD, for example, HTC Vive or a CAVE (Cave automatic virtual environment). The CAVE, which was discovered in 1992, is a projection-based system which uses 3D images projected on the walls and floor of a cube-like room to create an immersive experience. This technology immerses users into the virtual world with scenes projected on every corner of the room and surround sound, they could also navigate freely with the help of 3D glasses. Therefore, the CAVE provides a high level of immersion without disconnecting the user from the real world and, it enables multiple users to share the experience and move and interact freely in the virtual world. However, when compared to HMDs, it is very expensive to set up and requires an entire room.

HMDs such as Oculus Rift, Samsung Gear VR, and HTC Vive are currently the most used VR devices on the market (Fig. 15.1). Unlike the CAVE which involves a room and projectors, HMDs are affordable, easy to transport, use small display screens worn close to the user’s eyes and move with the viewer, and provide complete immersion in the virtual world by disconnecting viewers from their real environment, therefore providing a totally immersive VR experience.

For a fully immersive experience in the virtual environment, extra control devices are necessary for an optimal interaction with virtual objects. These devices aid to navigate freely within the virtual world. For example, the Virtualizer (Cyberith 2014) aids to move freely (walk, jump, sit, and run) in the simulated world, tracking the position of head, hand, or the entire body. Conventional control devices like Mouse, Trackball, and Joystick are not very effective with HMDs because the eyes are completely covered and users especially non-gamers, find it difficult to locate the position of the navigation buttons. Data Gloves with inbuilt sensors on fingers track the user’s hand and finger motions and could be used to manipulate virtual objects. For driving simulators, Logitech steering wheels and pedals with force feedback are widely used to enable a realistic driving experience. Other tracking devices such as the Kinect, are used to scan and create user’s avatar and this is mapped to the user’s motion in order to improve the level of immersion experienced (Aitpayev and Gaber 2012).

Fig. 15.1
figure 1

Oculus Rift and HTC Vive virtual reality head-mounted display

Full size image

2.1.1 Historical Background

This subchapter illustrates some of the VR highlights of the past such as the inventions of two VR pioneers namely; Morton Heilig who invented the Sensorama machine in 1957 and Ivan Sutherland, who invented the Ultimate Display headset in 1965. Stated in this chapter are also some of the reasons why VR was not widely used in the past decades though the potentials it offers in various industrial sectors were very obvious then, until the rebirth of VR in 2012 by Palmer Lucky who founded the Oculus Rift.

Morton Heilig, a cinematographer, who had the vision of an interactive 3D in the 1950s, invented the multi-sensory motorcycle simulator known as the Sensorama in 1957. The simulator enabled users to watch 3D films, which was later patented in 1962 (Morton 1962). The Sensorama combined all sensory channels by generating visual scenes of the city driven, engine and city sounds for auditory stimulation, smell of exhaust and food and seat vibration in its interaction concept in order to give its users the illusion of being part of the virtual environment. Apart from the generated vibrations of the seat, the machine also generated wind for an immersive experience of the visually perceived scenes. Heilig understood in his research of “The Cinema of the future” that in order to recreate reality, all sensory channels that influence human perception of the reality has to be taken into consideration and not just sight and sound (ArtMuseem 2000). This is actually what makes VR so complex to achieve. Though it is difficult to say for sure who invented VR, most papers refer Morton Heilig as the father of VR because of his contribution in insuring a near-realistic virtual experience through his multi-sensory interaction concept. The Sensorama was however not a big commercial success but laid down a blueprint for future 3D interaction concepts (Fig. 15.2).

Fig. 15.2
figure 2

The Sensorama motorcycle (Morton 1962)

Full size image

Another great invention of the 1960s in the area of VR is the Ultimate display HMD by Ivan Sutherland in 1965. The Ultimate display was part of the PhD dissertation concept of Ivan Sutherland in 1963. The concept was later developed to the Sword of Damocles, which is claimed by some researchers to be the world first HMD to track the head in real time. Just like the Sensorama, it is an interactive device which generated acoustic, olfactory, taste, and tactile stimulation. Sutherland on his research on the world of 3D graphics described his headset as a window in the artificial environment. He described the screen as a window through which one sees a virtual world but, the challenge still remains to make that world look real, act real, sound real, and feel real. This challenge as he rightfully quoted then has not changed. Though the Sensorama and The Sword of Damocles devices achieved the ultimate goal of VR by placing the users in a virtual environment, the Sword of Damocles goes further to track the user’s head movement which was correctly mapped to the stereo view in real time (Fig. 15.3).

Fig. 15.3
figure 3

The Sword of Damocles HMD (Sutherland 1965)

Full size image

Other devices and applications followed such as the Visually Coupled Airborne Systems Simulator (Kocian 1977) which is a visual system which comprises of a helmet with a tiny television tube and imaging optics meant to resolve visual presentation problems of flight simulators by imposing HuD contents in the pilot’s view. However, due to software and hardware limitations, side effects such as simulation sickness, high cost of setting up a VR labs, lack of accurate head-tracking, computers with limited processing power for HMDs in the past decades, VR was not successful and not widely applied. Nevertheless, this marked the beginning of an era which would change the way users interact with their systems and basis for the development of better virtual devices such as research into 3D interactive concepts with virtual objects through stimulation of all sensory organs and just like Sutherland rightfully stated, making it feel real, act real, sound real, and look real for the users of virtual environments. Morton Heilig and Ivan Sutherland also greatly inspired many works in the area of VR, highly interactive computer graphics, and HMI concepts.

2.1.2 Industrial Application of Virtual Reality

The psychological effects VR devices have on its users by creating a sense of presence in the virtual environment and at the same time, disconnecting them from their real surroundings could provide a natural form of interaction and thus, enhanced HMI concepts. For example, a test driver could be confronted with a dangerous traffic situation and reacts with panic although the environment is safe and absolutely risk free. This in turn, could create new forms of HMI concepts applicable and profitable to several industrial sectors. Though at the mention of VR, most people associate it directly with gaming only; VR could be applied in various industrial sectors from the design phase to marketing and maintenance of products and services. VR enables 3D visualization for a better understanding of concepts and how they interact. A virtual prototype of a new concept could be developed and tested on its expected functionality. This enables the discovery of discrepancies or safety challenges at the early stage of development.

That is especially essential for a sector like the automotive industry and other sectors that carefully need to avoid safety critical situations or situations with significant distraction of the driver from his primary task. Hence, the offered solutions need to be thoroughly evaluated before being introduced into a product and a proper evaluation is performed on a real environment. This allows a 3D perception of structures and provides new and risk-free methods of evaluation of critical traffic situations in the automotive industry without necessarily taking users to the road especially at the early stage of the development. It also reduces the number of prototypes developed because the prototype is not exposed to any physical damage and all changes could be performed on the virtual prototype.

Most of the leading automotive manufacturers use VR in various phases in their development process (refer to Table 15.1).

Table 15.1 Few automotive manufacturers who apply the VR technology
Full size table

VR could be applied to develop virtual architectural models of houses before they are built in order to foresee damages and prepare adequate preventive measures. Virtual prototypes enable potential customers to step into the computer-generated building and interact with their future house before it is built instead of just watching a non-interactive image. This has the advantage that users can visualize, explore, and create a better understanding of the house they had in mind, therefore achieving a better customer cooperation and satisfaction.

In the medical field, VR is mostly used for training and therapy. An example is the training of medical students to perform virtual surgery and get acquainted with the practice before the real surgery is performed on real patients. The stereoscopic 3D and high resolution displayed by HMDs presents detailed near-eye resolution of medical imaging diagnostics. Through the simulation of the entire physiology of the human body and organs, doctors could easily diagnose problems and chose the appropriate therapy faster. Therapists may also use the technology to treat people who are afraid of heights and needles with the help of a virtual application and virtual needle therapy. The surgeon using a HMD may have a complete simulated view of the surgery and detect new and easier ways to perform the surgery. For medical students learning how to operate, the best way would be to start with virtual patients. With the use of self-avatar, stroke patients could be stimulated to move disabled body parts by seeing their body do the movements in the virtual world.

In the Air Force, VR is applied for training of complex tasks and techniques without putting the users in any kind of risk. A full simulation to replicate a real-life situation, for example a simulated aircraft with all elements like in a real cockpit and place pilots in that position in order to improve their skills in a controlled environment. Table 15.2 shows some industrial application areas of VR.

Table 15.2 Some application areas of virtual reality
Full size table

2.1.3 Summary

VR is a real-time simulation that offers a high-end user interface that tricks the brain with the illusion of handling in the computer-generated world rather than operating from outside the virtual world. The main objective of VR is to create a real or imaginary world and objects to represent real-life situations, which are easy to operate and understand by the user for example, driving simulators to represent complex traffic scenarios. Furthermore, VR devices like HMDs with its real-time position tracking ability, stereoscopic 3D, near-eye resolution, and ultra-low latency provide a natural form of interaction within the virtual environment due to the multi-sensory channels. The good news is, most of the obstacles that hindered the advancement of VR have been resolved or its resolution in progress, and the devices are very much affordable ranging from 99 US Dollars for Samsung Gear VR, 599 US Dollars for the Oculus Rift to 799 US Dollar for the HTC Vive. Those who cannot afford the renowned devices mentioned above, could still experience full immersion using the ultra-low-cost Google cardboard and many other cardboards affordable from 5 US Dollars onwards.

As amazing and captivating as VR may seem, it still has a long way to go before its full integration into industrial processes is achieved. Creating an accurate replica of reality required to fool the human brain demands high computing power with appropriate programming. However, the experience and potential it offers, shows that it is worth the trouble since users get so captivated and spend a lot of time marveling at the virtual environment, which in turn would create awareness about the product or service in question. It is recommended to make the experience as interesting as possible by creating near-realistic scenes for experiences meant to replicate real-life products or services. The time spent in the virtual world should be limited because not every user is comfortable with HMDs and simulation sickness could occur at any stage during the experience. With the rapid emergence of new technologies such as better graphic cards, computer processing power, 3D modeling tools, it is expected that content building, which is still a major setback and other limitations would be easily dealt with in the future. VR should complement and not replace development processes or physical prototypes.

2.2 Driving Simulation

DS is a representation of driving scenarios and its complexity in a computer-simulated environment. It is used for research, development of new products, future enhancement for a better product quality, product verification with regards to reliability, and system robustness. It is composed of several highly performing subsystems for example, visual assets, motion system, interaction tools, and car dynamics which are integrated to reproduce near-realistic traffic models in real time. According to a research carried out by the Center of accident of the Monash University in Australia on the application of driving simulators in the prediction of changes in real-world crash risk, driving simulators are important and essential for pre-evaluation of interventions not yet tested (Rubin-Brown et al. 2009).

New automotive HMI components that are often derived from CE products are normally meant to be tested in real vehicles, on a real road, and with a real driver. To test a new component for its reliability and safety for the road could be very costly and risky and it should be repeatable. For this reason, driving simulation just like any simulation system is used to represent a real-world model of such concepts for example product or process model. The models are recreated on a computer to better understand, analyze, design, and easily manipulate virtual prototypes of complex systems mostly developed in Matlab/Simulink (Robinson 2004). Most of these simulations are running on computers with vehicle models implemented in a close loop. The use of these models in place of real vehicles and physical prototypes of automotive components, saves time, work, reduces cost, and avoids unforeseen hazardous situations which could arise from an unproven technology. In order to achieve a realistic driving experience in a virtual environment, it is important to have a realistic description of all factors involved in a driving scenario for example traffic situation, the road and driver behavior, when creating a simulation model for a vehicle (Gühman et al. 2012). Likewise, it is crucial to simulate speed and acceleration correctly when assessing ADAS for a realistic and correct simulation of vehicle behavior (Kemeny 2014).

The two main types of driving simulators which are widely used for research purposes are HIL (Hardware in the Loop), which is a form of real-time simulation mostly used in testing complex embedded systems with integrated real test components for example an electronic control unit (ECU), and SIL (Software in the Loop), which consists of only software model in a close loop. During a HIL simulation, a mathematical model of the process normally referred as the Plant such as a vehicle model or an engine is simulated and integrated with the real device to be tested. The test object is manipulated to accept the simulated model as a real object. Since the real environment where the hardware is supposed to run is a simulated model, a possible damage to the plant such as car engine is avoidable and therefore, reduces cost and risk (Halvorsen).

2.2.1 Choosing the Right Driving Simulator

The choice of a driving simulator system depends on the purpose and project for which the simulator is required. Driving simulators range from very simple and compact systems to highly sophisticated and multi-million US Dollars systems such as the Dome. The advanced and highly immersive Dome driving simulator engulfs a real vehicle, offers a visual field of up to 360° and has high performance motion simulators of 6–9 degrees of freedom (DOF). The Dome is so far the most sophisticated and advanced driving simulator meant for research and engineering purposes.

The next category of driving simulators are the so-called full-scale driving simulators which use a real vehicle or a semi-vehicle in order to improve comfort or ergonomics by presenting the user with a real vehicle components and dashboard. In order to reflect the behavior of the real vehicle, a motion platform is integrated, for example, the D-BOX motion simulators which move the entire car while it reproduces in real time the motion cues which are correctly mapped to the visually perceived motion to the driver (D-BOX 2016) (Figs. 15.4 and 15.5).

Fig. 15.4
figure 4

D-BOX actuators (left) and AMS motion platform (right)

Full size image
Fig. 15.5
figure 5

Audi A8 driving simulator with D-BOX actuators

Full size image

The last type of driving simulator to be considered in this chapter is the compact driving simulator. An example of this type of driving simulator system is the A3 from the Atomic Motion System shown in Fig. 15.6. Compact driving simulators are low-cost simulators which could accommodate extra screens attached direct to the system or on the wall. Steering wheels and pedals such as the Logitech G27 with force feedback could be easily integrated. Compact as the name states, it is easy to transport for instance, to fares and for customer demonstration. Though most come with integrated motion simulators, this could be easily integrated depending on the compatibility to third party products.

Fig. 15.6
figure 6

Atomic motion system compact driving simulator

Full size image

3 Realization of the Virtual Reality Driving Simulation

3.1 Concept and Architectural Design

The idea of integrating VR to a DS is meant to speed up the development process of automotive components through early evaluation with an interactive virtual prototype. These virtual prototypes drag the users into the system thus enabling an immersive and natural interaction with a product yet to be built. This will in turn reduce the number of iterations through reducing the number of physical prototypes developed since all changes could be made directly on the virtual prototype (Coates et al. 2002). The most important aspect of the immersive quality provided by a VR DS is the feeling of being part of the simulated environment which encourages the acceptance of this artificial environment as real. This provides room for a realistic testing of critical safety systems at the early stage of development when the concept is still not yet ascertained. The acceptance of the virtual environment as real has the advantage that the feedback provided by test drivers relates very closely to a realistic driving experience on a real road. This feedback could be effectively used as a basis for the development of the physical prototype and thus, saving production costs and time.

In order to develop a virtual driving simulator which could be effectively integrated in the development cycle of automotive components, specific requirements, and steps have to be considered and effectively implemented. These involve the selection of VR-specific tools such as simulator software with VR devices integration possibility for example 3D engines such as Unity3D and Unreal Engine 4. Most important is the correct simulation of the virtual environment contents such as 3D models of cars, landscapes, roads, and the vehicle control model. The visual quality of the environment determines if the artificial environment is accepted as real or not since the appearance of pixels as a result of low resolution definitely does not promote realism. Therefore choosing appropriate tools for VR systems plays a major role especially when it involves complex systems like a driving simulator with high performance meant for the evaluation of safety functions.

The process of selecting the tools should be done through benchmarking of the performance of the tools into consideration and recommendation of VR device manufacturers such as Oculus VR for systems using Oculus Rift (Oculus 2016). Though the use of renowned driving simulation software such as SILAB would be the most appropriate because of the many features and ready-made virtual environment which has passed the test of time, 3D simulation tools compatible with head-mounted displays are favored because most renowned head-mounted displays support Unity3D and Unreal Engine 4. It is however recommended to consider the integration concept and limitation factors of all components especially Third-Party tools, for example input and output devices such as Logitech steering wheels and pedals when considering 3D engines because not all have the ready plugins. Unity3D has support for most Third-Party tools.

The process of driving coupled with the different driving maneuvers in order to perform the needed driving tasks, yield different types of motion cues which are essential for the driver’s effective and intuitive maneuver of the traffic situation. The visual cues are not enough when driving since the motion felt is needed in order to connect the driver’s body to the vehicle mechanics. This makes it a must to reproduce the most important visual motion cues by selecting a high performance motion platform in order to simulate the required motion cues. The integration of the chosen motion platform poses a big challenge because a 1:1 scaling of the motion implemented and felt is necessary. For example, the driver should be able to differentiate about a vibration felt on a smooth surface and that felt on a rough road surface in order to react appropriately. Therefore, it is essential to reproduce the motion accurately since research has proven that no motion at all is better than bad motion. This is because if the motion does not represent the visually perceived motion, it could result in the wrong maneuver and also simulation sickness especially in a vigorously moving environment like driving simulators.

The concept of a VR DS is centered on the Human actor. The Human actor wears the head-mounted display, controls all input devices, and experiences various feedback based on the inputs. During driving, the input devices send signals to the simulation model. The simulation model accepts this signal and calculates the corresponding feedback. It then provides an adequate response to the output devices. This response goes back to the Human actor. The system works continuously in a close loop until the application stops. This simple concept encourages early validation of the components and system as a whole. Test drivers could experience real-time simulation of new safety products behavior and enables a near-realistic interaction. This would reduce the development time since discrepancies are detected and readily taken care of at the early stage of development. The development of a virtual driving simulation should be model-based and should support easy integration and reusability of components which could arise as a result of changed requirements. Figure 15.7 above illustrates this concept.

Fig. 15.7
figure 7

A VR driving simulation centered on the Human actor

Full size image

3.2 Implementation

The implementation of the VR DS at the Center of Competence HMI, Robert Bosch GmbH in Leonberg in cooperation with the UniTyLab of the University of Applied Sciences in Heilbronn is based on two driving simulation concepts, namely the Compact and the Full Scale driving simulators. While the Full Scale composed of a semi-Audi A8 with full integrated components and real car cabin, and a D-BOX motion platform, the compact simulator consisted of the Atomic Motion Systems A3 (AMS A3). This work will however focus on the implementation of the AMS A3 Compact driving simulation system because the full-scale simulator system is still under development. Figures 15.5 and 15.6 show the semi-Audi A8 full scale and the AMS A3 driving simulator mock up.

In order to develop a complex VR application such as a driving simulator, a high-end computer with a powerful graphic card is recommended. The central processing unit (CPU) used for the prototype development is the Intel Core i7-5960X which has an eight core processor unit and an Asus X99 mainboard which enables the addition of multiple graphic cards. Two GeForce GTX 1080 graphic cards are selected for an optimal graphical performance since this is very essential for VR application. Other components such as the RAM and disk drives are selected based on the application to be developed and benchmark.

Both Unity3D and Unreal Engine 4 are considered for the implementation of the simulator software. Based on the benchmark carried out on a previous work, Unity3D was considered because of its large user community for easy problem tackling and availability of most third-party plugins. Unreal engine 4 on the other hand, provides a higher and better visualization performance of the virtual environment for a near-eye visualization and also the Blueprint technology for non-developers. The implementation also involves the modeling of the 3D assets for the virtual environment such as roads, traffic signs, landscapes, and many more assets that make up a realistic driving environment (Fig. 15.8).

Fig. 15.8
figure 8

3D model of a typical German country road (left) and traffic model (right)

Full size image

The prototype development consists of two phases; the first phase consists of the simulation of the traffic environment and a car model, taking into consideration an accurate simulation of the vehicle dynamics. This phase also involves the integration of control devices such as Logitech steering wheel, shifter and pedals, and a 3D audio for the surrounding and engine sound. The second phase consists of the integration of the motion system to the simulator software and the implementation of dynamic head-up display features for various use cases which was actually one of the main objectives of this simulator concept. Head-up displays enable the driver on a real road to focus on the road and therefore avoiding distractions since all the necessary features are projected on the windscreen. The easiest implementation of head-up display could be done as a standard floating GUI text. This is however not enough for complex and dynamic features. Unreal engine 4 on the other hand with its Blueprint technology provides a faster and better solution for head-up display contents. Figure 15.9 shows prototype without head-up display and with head-up display.

Fig. 15.9
figure 9

Wrong (left) and correct (right) display of a BMW Dashboard

Full size image

Finally, an appropriate haptic feedback device for hand-finger tracking and when necessary, with upper or full body tracking so that drivers could handle freely and intuitively like he would in a real car and also interact with clickable interfaces like in a real car infotainment system. Considered for this simulator is the Manus VR Glove which tracks the user’s hand movements in real time, with the help of sensors inside the glove (MANUS VR 2016). Figure 15.10 shows the most important steps and functions to be considered when implementing a VR DS. In addition to the steps illustrated below, the choice of a VR—ready computer with high-end processing power and graphics is very important.

Fig. 15.10
figure 10

The workflow of the simulator implementation

Full size image

3.2.1 Driving Scripts

The driving script is the main controller in this driving simulator. Due to the fact that 3D Engines offer only wheel colliders and simple Physics, it is therefore very hard to get a realistic driving model. This is not enough to develop a realistic driving simulator with all the complexity involved in a driving scenario. In order to implement a realistic car model, the driving script was implemented in JavaScript and C# in Unity3D which consists of transformer object for the steering, wheel colliders for the Front Left and Front Right wheels, vehicle dynamic parameters, for example, engine maximum Torque and controller functions.

The scripts consists of a Start function, where the center of mass of the rigid body is initialized, an Update function, which is constantly reading the inputs from the driver and transforming the resulting movements to the wheels and steering of the moving vehicle and a graphical user interface function, which updates the display values in the car cockpit. Figure 15.11 shows the driving script control loop.

Fig. 15.11
figure 11

Driving script control loop

Full size image

Finally the driving script is integrated into the Unity3D VR DS project. During this integration, the Front Right and Front Left wheel colliders of the car are mapped to the respective wheel collider variables of the driving script as well as the steering transform object.

3.2.2 Control Devices

In order to achieve a fully immersive experience and give the driver the illusion of presence in the driving virtual environment, a near-realistic interaction with virtual objects is essential. These devices aid the user to move freely (touch, feel, walk, sit, run, and jump) and manipulate objects like in real-life. Conventional control devices like Mouse, Trackball, and Joystick are not very effective with head-mounted displays because the eyes are completely covered and for non-gamers, might be difficult to locate the various keys or buttons needed for specific navigations. Driving simulators third-party hardware such as Logitech G27 steering wheel and pedals with its force feedback transmission used to control the direction and speed of movement provides an optimal interaction when accurately implemented for a realistic driving impression. The control flow between the Logitech device and the 3D vehicle model simulation is realized in the 3D engines with the help of Logitech Plugins and Assets. The system could however be configured in a way that, if the Logitech is not connected, the system automatically starts using the keyboard as the driver input. Also very essential for a better interaction and user experience are hand and finger tracking devices such as Data Gloves with inbuilt sensors for tracking the test driver’s hand and finger motions in real time and also to manipulate virtual objects such as clickable interfaces (Fig. 15.12).

Fig. 15.12
figure 12

MANUS VR Glove for hand tracking

Full size image

3.2.3 Three-Dimensional Modeling and Animation

Three-dimensional models are a vital component in the development of a VR DS. The 3D models should consist of 3D stereoscopic display information reflecting the objects in the real world. Third-party assets could be used except the needed models are not available or not good enough for the required purpose. Unity3D asset store and Unreal engine 4 offer a good number of models for easy integration. Most 3D engines support .fbx and .obj assets. These 3D models were downloaded from the Unity Asset store. Most models were retextured using the Substance painter texturing software. Most assets come with very high polynomial numbers which could slow down rendering and also affect the performance of the implemented software negatively, some parts of the asset for example the car model could be removed by a 3D-Artist and any resulting issues could be easily fixed (Fig. 15.13). It is very essential to add AI vehicles and animated pedestrians in order to give the illusion of a congested traffic model.

Fig. 15.13
figure 13

Model reduction of parts not visible by the driver

Full size image

3.2.4 Visualization Components

This consists of the head-mounted display, desktop monitor, or any other display monitors. Most important is the choice of a high-end graphic card and a renowned head-mounted display like the Oculus rift consumer version and HTC Vive. The visualization components interact with the simulation through the head-mounted display SDK which provides the integration package with Unity3D or Unreal engine 4, Library containing all the Prefabs, and plugins necessary to create and run a VR application. The main prefabs are Player Controller and Camera Rig. The camera controller is positioned in front of the driver, so that the actor wearing the device can be following the vehicle movement. The tracking sensor of the device is placed in a suitable position in order to accurately track the position of the device.

4 Challenges

It is quite complicated to replicate a real traffic situation because of the complexity of traffic conditions. It is even more complicated to develop a VR DS which the human brain has to accept as real. Though VR is not a new technology, its application is associated with a lot of limitations which must be overcome in order to explore the benefits of the technology. Few of these limitations associated with VRDS are; the lack of accurate motion cues which correlates the vehicle mechanic to the human body, lack of haptic feedback when the driver operates interfaces that should give him a clue of the next action without necessarily looking at the button, low resolution which does not correlate with reality because of the appearance of pixels, a traffic environment which does not correlate to reality, and the inconsistency in the speed driven and the speed felt.

All factors that influence the driving conditions must be taken into consideration when implementing a VRDS to be applied on a real product development. For example, the behavior of the vehicle must be consistent with the driver’s expectations if the driver is expected to accept the environment as real. The punishment for not achieving the aim, which is to accurately fake reality, could result to simulation sickness. Simulation sickness which is a form of motion sickness, experienced by a person exposed to simulated environments.

Simulation sickness could be caused by sensory conflict as a result of a conflict between the visual perception of the simulated motion and the motion felt (Karl et al. 2013). People exposed to simulated environment like VR and DS could experience general discomfort which could influence the user’s mental and physical state during evaluation sessions on a simulated environment. Therefore, it is relevant to find a solution on how to eliminate or reduce these symptoms to minimum. Almost all the limitations of VR and DS induce simulation sickness. For example, the lack of a motion platform or inappropriate motion cues could lead to simulation sickness because of conflicting information perceived visually and felt by the inner ear. While some researchers argue that little motion, appropriate or not is better than none, other studies suggest that no motion is better than a badly simulated motion. Dr. Kemeny however presented a motion cueing algorithm in his research titled Motion cueing algorithms for a real-time automobile driving simulator, which takes into consideration driver’s attention and the road information in order to reduce false motion cues presented to the driver (Kemeny 2012). It is therefore essential to integrate motion simulators to generate the visually perceived motion in the form of vibration and tilting which gives the feel of a real car. The vibration is synchronized with simulated engine revolution per minute (RPM) for an improved perception of the virtual speed, while tilting offers a vestibular stimulation to suppress simulator sickness during acceleration, braking, and steering or cornering (Weinberg and Harsham 2009).

High latency which reflects a delay between an action such as pressing on the gas pedal and the response such as the car moving accordingly in the visually perceived world poses a big problem associated with all VR applications. While some platforms offer accurate Vehicle Physics for a near-realistic vehicle simulation model, it still remains near-realistic and the discrepancy could be felt by an attentive and well-experienced driver. This could limit the level of immersion experienced. Another big challenge is the integration of audio systems that reproduces the correct vehicle sound and surrounding traffic sounds. These are just few of the problems accounted during the development of a VR DS prototype. Every DS is different and is faced with task or system-specific problems which could be solved accordingly.

5 Conclusion

As vehicles become more complex and connected, there is need to implement solutions that avoid and reduce any form of distraction to the driver. DS provides a computer-generated environment for various research purposes in the automobile industry. The application of DS is preferable to real-life experiments on the road because it is cost-efficient and enables rapid prototyping but most especially, it is a controlled environment and totally risk free for the users. This facilitates rapid development of virtual products for evaluation of new concepts before the physical product is produced. Emerging technologies in the field of VR from the area of consumer electronics and increasing processing power offer potentials for new highly immersive evaluation concepts through the combination of a VR device to a driving simulator. The integration of a HMD to a DS is to simulate a realistic environment of imaginary or real objects which is easy to operate and accepted by the user as real. VR and DS use the same technology to render user interaction and the sense of presence in the virtual environment and are highly interactive computer-generated graphics, which offer a certain level of immersion in the virtual environment and in real time (Kemeny 2014). Because of the above-mentioned reasons, both technologies could easily be combined to develop a highly immersive VR driving simulator framework, for rapid prototyping of HMI concepts using highly performing HMD or the CAVE for multiple user experience.

It is very essential to implement the following factors correctly in order to have the full benefits of a VR DS. First, the modeling of high-quality visual assets which represent real vehicle and traffic models is of paramount importance. Second, congested traffic models which consist of all components that make up a real traffic situation such as artificial intelligent (AI) cars with drivers, traffic lights and signs, landscape, pedestrians have to be correctly simulated and animated where necessary. Since the aim of VR is to impose realism to the human brain, there is need for a natural form of interaction such as the use of an instrumented glove with haptic feedback for hand and finger tracking, high-resolution graphic quality for a pixel-free visualization, correct engine, and surrounding sounds. Most important is the 1:1 mapping of the motions seen and felt because the motion cues enables the driver to have the right reaction, for example, slow down when driving on a rough surface. The correct implementation of the factors mentioned above due to the combination of high-end visualization, real-time computer simulation and a multi-channel interaction would also help to reduce the occurrence of simulation sickness.

Virtual environments are highly captivating at a first try but, in order to remain captivated into this illusion of realism created by VR, the virtual environment has to be properly simulated to reflect the corresponding real world it is meant to represent. If the technology is applied to evaluate a product, the product must be simulated in its correct size and behavior, a high graphical representation and a near-realistic interaction concept established. This also involves the addition of appropriate virtual objects that make up the simulated product such as animated pedestrians and traffic lights into a driving simulator in order to represent a real traffic environment.

A VR DS would enable easy evaluation of emerging technologies still under research such as the autonomous and highly connected vehicles. For example, unforeseen issues which could arise from the behavior of the human driver and the autopilot could be investigated with regards to autonomous vehicles. Although DS and VR are two technologies are highly associated with simulation sickness as mentioned above, these technologies are however very important in the evaluation of new products. VR device manufacturers and researchers are working on possible solutions to reduce or completely eliminate these limitations. Many solutions have been studied and suggested by researchers such as reduction of the field of view, addition of visual assets, integration of motion platforms, high-end graphic cards and many more. Most of these solutions are application-specific. While complex applications like a DS might consider the correct implementation of a traffic model, vehicle control model, and most of the solutions mentioned above, applications which are only meant for vehicle interior design might focus on the graphical performance of the system.

The challenges of creating a VR application with a high graphical performance and an intuitive interaction concept which is convincing and acceptable to the human brain as real for total immersion in the artificial world are very high and should not be overlooked. However, as VR evolves, graphic cards and CPUs are getting more powerful by the day, and HMD manufacturers are also improving the optics of their devices because it plays a major role on the visuals. Controls devices are also getting better and cheaper, for example the MANUS VR glove for just 250 US Dollars. VR DS could effectively be integrated at the early stage of development to complement conventional 2D DSs and real-road evaluations in the development and evaluation of future automotive HMI concepts.