Keywords

1 Introduction

Towards multi-modal user-centered human-machine interfaces (HMI) in automotive research and development, it is essential to consider the user in early stages. Multi-modality includes visual, auditive and haptic stimuli during development.

These HMIs are designed, e.g. to direct and enable users to understand important information faster and act properly in relevant situations. A good HMI conveys confidence and trust to the user, therefore needs comprehensive testing to be used on a larger scale. However, testing is a resourceful component as well as time, money and labor consuming. Adding the higher risks of testing on open roads and the difficulty to replicate trials, industry and science has shifted to more advanced and safer ways to build experiments.

Within this context, virtual reality (VR) has been a common tool in the automotive sector to design, develop and evaluate new external HMI (eHMI) since the 1990s with Cave Automatic Virtual Environment (CAVE) [1]. Since then, designers and test architects have been able to test human-centered user interaction in a risk-free environment while showing prototypes in early development stages. Today’s hardware with advanced and fast multi-processors is able to output high-resolution stereoscopic 3D-rendered images on small head-mounted display (HMD) with additional auditive and haptic feedback. These components help bringing a more immersive experience to the user and setting up a new environment detached from the physical world while being more efficient, smaller and less expensive than its predecessor.

Using VR reduces development time and cost but also comes with new challenges to build real-world-like scenarios. Instead of creating a potentially dangerous situation and compromising safety, users may sit in a laboratory perceiving auditive signals to immerse into an virtual world. Different approaches have been described to conduct automotive eHMI development in such environments but with little analysis of the overall setup.

To extract important key points and common practices, we are going to take a closer look at automotive eHMI development procedures in VR and compare experimental hands-on practices. Our analysis is based on the implementation and execution of the evaluations published in each paper. For that, we are going to take a closer look in our paper on VR simulation key points that need to be taken into account.

2 Concept of Immersion and Interactivity

To foster authentic feedback in VR, an environment has to be created to trigger realistic user perception and reaction. Walsh and Pawlowski [2] worked out a set of dimensions for VR user experience (UX) in information system research based on cumulative literature review. With multiple studies supporting the same pillars [2, 4], we will use this as our concept reference and concentrate on immersion and interactivity as being largely responsible for VR experience and the well-being of users [3]. The two dimensions will be our main focus when assessing the studies to point out strengths and weaknesses.

Immersion is the degree of isolation from the real world, i.e., from multi-modal stimuli according to Slater and Wilbur [5]. Unlike the subjective feel of presence inside VR, immersion corresponds to the objective stimuli put onto the users’ sense to generate the degree of immersion. The more we are disconnected from the real world the more we feel immersed into the virtual world. The level of immersion is a product of visual, audio and haptic input or output. In 1992, Steuer assessed the field of immersion and placed breadth (varieties of sense, i.e., visual, audio, haptics) and depth (e.g., frequency, resolution, field of view (FOV), detail) as the key factor for an immersive experience [6]. Common practices uses devices to shield physical environmental input and overlay virtual ones [6]. To assess the objective stimuli onto study participants, we are going to take a look at the hardware setup and 3D environment contributing to the immersion.

Interactivity refers to how the user can modify and interact with the objects and environments of VR. Interaction can be done using a variety of input devices like hand-held motion controller, steering wheel or other sensor inputs. On one hand, it enables user interaction and direct feedback while on the other hand it amplifies the feel of presence. To map interactivity to HMI development, we assess the tools used to generate such interactivity.

3 Automotive HMI Development in VR

Previous studies addressed the concept of driver simulation and its design for near-realistic UX, giving recommendations when creating a virtual automotive study [7,8,9], i.e., high-fidelity setups and multi-modality for better immersion. Most studies agree on the advantages of VR over conventional testing environments like risk reduction and reproducibility and encourage to use VR for automotive development and early prototyping [7, 9,10,11].

In 2019, Colley et al. [12] investigated seven automotive studies in VR using simulation criteria derived from the discussion paper by Winter et al. [9]. They concluded that the usage of HMD should be limited below one hour. Furthermore, the researchers suggested more motion opportunities like actual walking and the presentation of questionnaires within VR. They also pointed out to consider the design of urban environment regarding sociocultural differences in VR to vary experimental outcome. The conclusion will be taken into consideration while assessing our findings in this review.

Hock et al. [13] proposed a checklist of eight points to consider in a driver simulation study. Considering our scope immersion and interactivity for eHMI, i.e., the view point of traffic participants, we concentrate on two out of the eight points: 1) simulator sickness (i.e., low motion speed, short VR exposure, for better readability we will refer to this as continuous immersion) and 2) simulator training (some form of familiarization for participants to get accustomed).

After defining our criteria to assess the goal of our study, the upcoming sections will cover the study selection.

4 Paper Selection

To focus on more recent and more affordable HMD technology used, we considered for a better comparison current studies from 2017 to currently April 2020. We identified a total of six studies on automotive eHMI prototype evaluation in VR from our database search (ACM Digital Library, IEEE, Scopus, Springer Verlag). Considering the emerging field of research both of automotive eHMI in automated vehicles (AV) and HMDs, The selected studies are only about eHMI development using modern HMDs and study design in VR. The criteria allow us to give a valid and focused view for comparison between all reviewed studies.

For the sake of simplicity, we assume that all studies used VR-ready hardware to back up the computational power. Because of that, variations on computational hardware will be neglected unless feedbacks reported some lag or technical issues. A more detailed list of the study setup is shown in Table 1.

Table 1. Setup and procedure of eHMI development studies in VR
Full size table

5 Results

Now that we have our identified studies accumulated, we can assess the studies against the background of VR dimensions [2] and simulation criteria [13].

Immersion: In terms of resolution and detail, the level of immersion created by the 3D engine and displayed onto the HMDs is mostly similar as depicted in all studies. The studies showed a typical western urban environment from a pedestrian’s point of view showing at least all relevant characteristics of a street (i.e., sideways, lane markings) and some building structures. The lack of descripted devices used in [17] in addition to an unspecified hardware failure makes this study in terms of hardware specification difficult to compare. Aligned with user feedback across all studies, the immersive experience can be described as positive. Since setups are not fully described in detail (as shown in Table 1), we are summarizing the approximated setup as sufficient in terms of FOV, refresh rate, resolution and level of detail. Sound was implemented by [14, 16, 17] using audio environment with dynamical changes depending on vehicle speed and distance. Two studies [15, 18] reported using headphones which can be attributed to the integrated headphones of the HMD and may imply a sound environment. In the after-study questionnaire by [14], participants attributed increased immersion due to the combination of visualization and sound which is align with prior research [5, 20]. Due to insufficient description, we cannot make a statement regarding the sound environment in the other studies. Questionnaires are administered before and after studies and therefore are not presented during VR sessions as suggested by [13]. However, considering the short duration of VR experience for users, we did not find any immersion drawbacks or need to implement within-VR questionnaires.

Interactivity: Regarding locomotion, [14] allowed for a 9 × 3 m2, [19] for a 4 × 7 m2 area, resembling a road section where participants could move. Controllers were used in [15, 16] to determine a movement decision by pressing a button on the motion controller. In the study by [17], participants used gestures to signal the AV to stop. However, an unspecified technical problem occurred during the recognition which limits the results. [18] did not offer any interactivity as the participants had to fill out a survey to report their experience.

Simulator sickness: Questionnaires measuring the user’s well-being help researchers to evaluate potential discomfort during their study. Conducted questionnaires to assess level of immersion and simulation sickness showed little to no discomfort in user feedbacks [14, 16, 19]. Some participants experienced increased discomfort [16] but no abortions were documented [14, 17, 19]. The time spent in the VR was documented between 20 and 40 min which is within the recommended limits stated by [13]. Differences in duration showed no significant changes across all identified studies.

Simulator training: Familiarization or warm-up period was given in almost every study. So most researchers do practice some kind of user acclimation to the VR. [15] suggested no longer than 30 min experiencing VR at a time, [19] conducted their study within 40 min per participant and [16] recommended taking a break every 5 min. It is shown that the learning curve helped users to get familiar and comfortable within VR. Feedback on the VR experience and level of immersion therefore was reported positively and is in line with prior research.

Overall, all researchers concluded a positive experience with their VR setup, praising the safe execution over real-life scenarios and confirmed to use it again. Although slightly different in setup constellation, studies share similar HMD specifications. All setups provided a visual environment with currently common 3D engines like Unity or Unreal Engine, providing potentially high-resolution images for the HMD [15,16,17,18,19]. Conference Name: ACM Woodstock conference Conference Short Name: WOODSTOCK’18 Conference Location: El Paso, Texas USA.

6 Discussion

Our goal was to assess studies in automotive eHMI development in VR to understand and filter for common applications. The studies had some descriptive gaps, making it difficult to understand certain setup structures as well as having a comprehensive view on the study design. The lack of multi-modal immersive stimuli in [15, 17,18,19] makes it challenging to compare immersion factors to other studies, especially when the differences in hardware use can be neglected. To our surprise, little effort was done to stimulate more senses, i.e., embodiment or locomotion for a better feel of presence in VR.

Besides multimodality, the analyzed studies seem to focus on ready-made and easy-to-use hardware and software solutions. Since hardware and software available on the market is still in an early product phase, it is hard to create a multimodal experience. However, as described in Sects. 2 and 3, we would still suggest to take more senses into consideration than just visual immersion to allow more reliable results and stable user well-being during the assessment. Overall, we tackled different aspects of VR and simulation structures and found frequent similarities between implementation and theory.

We do understand the lack of necessity to provide virtual interaction as studies often concentrate on the premise of observing the decision-making of traffic participants choosing to cross a street or not. Because of that, most studies disabled locomotion or even a virtual body representation. This is align with the conclusion [13] came to in their review study. Even without the need to take any action inside the VR, based on studies like [16], we encourage more interactivity to increase participants’ UX and give them a sense of presence through, e.g., walking or hand movement.

Regarding the simulation criteria, we found little to no drawback. Most studies introduced a warm-up phase to get familiar with the VR. The extended familiarization period presented in [19] showed a rewarding effort to lower feel of uncertainty and discomfort which we believed can help inexperienced users in VR. The consideration of sample was difficult to conclude as little was reported to show heterogeneity and overall user profiles. As for our examination, we believe this might not be as relevant for our paper as this study dimension has not produced abnormal study results and participants were mostly selected randomly.

Considering our challenge to show eHMI evaluation done in VR, we came across multiple limitations in our study: Firstly, it is important to point out that the automotive eHMI studies found and considered in our review are mostly of visual nature, making the perception of eHMIs highly dependent on visual input while auditive, haptic and interactive factors mainly serve the purpose to increase level of immersion. Depending on the assessing eHMI, auditive, haptic or interactive elements might be highlighted more. In general, we would suggest future studies to further take multi-modal stimuli into consideration to not only increase the feel of presence within a virtual situation but also to emphasize genuine reaction. Secondly, most study designs were methods to evaluate a specific eHMI design. Therefore, the validity is difficult to confirm or disprove by just criticizing the VR setup. Furthermore, as new technological devices reach the market, they become more accessible. Advanced VR interaction tools like VR gloves or gait recognition are still limited in their usage and need more time to be tested to reliably enhance virtual interactivity.

7 Conclusion

We assessed six different eHMI development VR setups against the background of VR dimensions and simulation criteria recommended in research to gain a comprehensive immersive experience for user studies. The scenarios built and setups made an overall immersive and profound baseline to evaluate new eHMI designs and test users in near-realistic environments. Towards our attempt to extract ideas and lessons learned, we distilled the following recommendations:

  • Let the user get familiar with the system and observe the user’s well-being during the study to ensure reliable results;

  • Build a near-realistic visual VR environment, backed up by a multi-modal environment to enable better interactivity and proprioception within VR;

  • Decide to use immersion factors depending on the evaluating eHMI;

  • Document setup description precisely and mention factors contributing to the VR experience of the user (i.e., hardware specification and study procedure).

Overall, all studies in our review support the usage of VR as a promising addition to early prototyping and they advocate the advantages of immersive virtual environments. Further studies are needed to provide a more in-depth look into eHMI-specific traits when building a virtual test environment for participants. The results suggest a high potential in future automotive eHMI development studies in VR with higher immersion and better multi-hardware setups.