Abstract
Augmented Reality in vehicles through Head-Up or Windscreen Displays offer numerous opportunities to develop road safety or user experience applications or both. These opportunities are further expanded in the context of automated driving, where human drivers are partially freed of the safety-critical driving task to attend to other activities, so-called non-driving related tasks. This brings new human factor challenges, such as lack of situation awareness when the driver is asked to take control of the driving vehicle. There are methodological challenges when undertaking user experience design or human factors research in this domain because of the current development status of future technologies. Prototypes of windscreen displays and automated vehicles are difficult to access and expensive. Therefore, we have developed an affordable virtual reality automated driving simulator that implements real-world driving videos for interface prototyping and testing. In this chapter, we discuss two concepts (dynamic windshield tinting to communicate certainty and an Augmented Reality video screen movements to communicate intention) to extend the windshield with additional driving relevant information, which were rapidly prototyped and evaluated in our Virtual Reality simulator. Following the user experience design approach, we report on qualitative findings from a user study design iteration (N = 16) about those concepts. We reflect on the identified strengths and weaknesses of our design concepts, which were critically influenced by participants’ experience and trust in vehicle automation, as well as the methodological process. Lastly, we report on valuable lessons learnt from developing, improving and implementing our prototyping, simulation and evaluation tool for Augmented Reality applications in automated driving.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Change history
01 January 2022
Correction to: Chapter “Inception of Perception—Augmented Reality in Virtual Reality: Prototyping Human–Machine Interfaces for Automated Driving” in: A. Riener et al. (eds.), User Experience Design in the Era of Automated Driving, Studies in Computational Intelligence 980, https://doi.org/10.1007/978-3-030-77726-5_18
Notes
- 1.
https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety, retrieved 30. June 2020.
- 2.
https://varjo.com/products/xr-1/, retrieved 30. June 2020.
- 3.
Second life and online virtual environment.
- 4.
Photogrammetry is a technique of capturing objects and environment in three dimensions with topological and geographical data respectively.
- 5.
https://www.microsoft.com/en-us/hololens, Retrieved 30 June 2020.
- 6.
https://www.magicleap.com/en-us/magic-leap-1, Retrieved 30 June 2020.
- 7.
https://www.oculus.com/rift/, retrieved 30 June 2020.
- 8.
https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool, retrieved 30 June 2020.
- 9.
Oculus controls were not used in this study.
- 10.
The IPD is the distance between the left and the right eye. The lenses can be moved together and apart to adjust that.
- 11.
Make transparent based on a unified (predefined) color value a.k.a “Green Screen” or “Blue Screen”.
- 12.
The videos can be recorded stereoscopic and then played back on the video-players individually for each eye to receive depth.
References
Abdi, L., Abdallah, F.B., Meddeb, A.: In-vehicle augmented reality traffic information system: a new type of communication between driver and vehicle. Procedia Comput. Sci. https://www.researchgate.net/profile/Lotfi_Abdi/publication/291013097_In-Vehicle_Augmented_Reality_Traffic_Information_System_A_New_Type_of_Communication_Between_Driver_and_Vehicle/links/56e6da7908ae9e803e7bee6d.pdf (2015)
Agarwal, R.: Validation of a pedestrian simulator for interaction between pedestrians and autonomous vehicles. In: de Winter Joost, H.R. (ed.) Master thesis, Delft University of Technology. https://repository.tudelft.nl/islandora/object/uuid%3Ae1581f7c-aa7e-4606-991b-bbaf7940c79c (2019)
Alvarez, I., Rumbel, L., Adams, R.: Skyline: a rapid prototyping driving simulator for user experience. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 101–108 (2015). https://doi.org/10.1145/2799250.2799290
Azuma, R.T.: A survey of augmented reality. Presence: Teleoper. Virtual Environ. 6(4), 355–385 (1997). https://doi.org/10.1162/pres.1997.6.4.355
Barić, D., Havârneanu, G.M., Măirean, C.: Attitudes of learner drivers toward safety at level crossings: Do they change after a 360° video-based educational intervention? Transport. Res. Part F Traffic Psychol. Behav. 69, 335–348 (2020). https://doi.org/10.1016/j.trf.2020.01.018
Brooks, J.O., Goodenough, R.R., Crisler, M.C., Klein, N.D., Alley, R.L., Koon, B.L., Logan, W.C., Jr., Ogle, J.H., Tyrrell, R.A., Wills, R.F.: Simulator sickness during driving simulation studies. Accident; Anal. Prevent. 42(3), 788–796 (2010). https://doi.org/10.1016/j.aap.2009.04.013
Chatzidimitris, T., Gavalas, D., Michael, D.: SoundPacman: audio augmented reality in location-based games. In: 2016 18th Mediterranean Electrotechnical Conference (MELECON), pp. 1–6 (2016). https://doi.org/10.1109/MELCON.2016.7495414
Craig, A.B.: Understanding Augmented Reality: Concepts and Applications. Newnes. https://market.android.com/details?id=book-7_O5LaIC0SwC (2013)
Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V., Hart, J.C.: The CAVE: audio visual experience automatic virtual environment. Commun. ACM 35(6), 64–72 (1992). https://doi.org/10.1145/129888.129892
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: arXiv [cs.LG]. http://arxiv.org/abs/1711.03938 (2017)
Endsley, M.R.: Designing for Situation Awareness: An Approach to User-Centered Design, 2nd edn. CRC Press. https://market.android.com/details?id=book-eRPBkapAsggC (2016)
Endsley, M.R., Kiris, E.O.: The out-of-the-loop performance problem and level of control in automation. Hum. Factors 37(2), 381–394 (1995). https://doi.org/10.1518/001872095779064555
Feierle, A., Beller, D., Bengler, K.: Head-up displays in urban partially automated driving: effects of using augmented reality*. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 1877–1882 (2019). https://doi.org/10.1109/ITSC.2019.8917472
Gerber, M.A., Schroeter, R., Vehns, J.: A video-based automated driving simulator for automotive UI prototyping, UX and behaviour research. In: 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ‚19). Automotive User Interfaces 2019, Utrecht, Netherlands (2019)
Gerber, M.A., Schroeter, R., Vehns, J.: A video-based automated driving simulator for automotive UI prototyping, UX and behaviour research. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 14–23 (2019). https://doi.org/10.1145/3342197.3344533
Gerber, M.A., Schroeter, R., Xiaomeng, L., Elhenawy, M.: Self-interruptions of non-driving related tasks in automated vehicles: mobile versus head-up display. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–9 (2020). https://doi.org/10.1145/3313831.3376751
Goedicke, D., Li, J., Evers, V., Ju, W.: VR-OOM: Virtual Reality On-rOad Driving siMulation. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 165:1–165:11 (2018). https://doi.org/10.1145/3173574.3173739
Guo, C., Sentouh, C., Popieul, J.-C., Haué, J.-B., Langlois, S., Loeillet, J.-J., Soualmi, B., Nguyen That, T.: Cooperation between driver and automated driving system: Implementation and evaluation. Transpor. Res. Part F Traffic Psychol. Behav. 61, 314–325 (2019). https://doi.org/10.1016/j.trf.2017.04.006
Häuslschmid, R., Osterwald, S., Lang, M., Butz, A.: Augmenting the driver’s view with peripheral information on a windshield display. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, pp. 311–321 (2015). https://doi.org/10.1145/2678025.2701393
Häuslschmid, R., von Bülow, M., Pfleging, B., Butz, A.: Supporting trust in autonomous driving. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 319–329 (2017). https://doi.org/10.1145/3025171.3025198
Heinrich, F.: Vorhersage der Fahrerbelastung während der Fahrt. https://elib.uni-stuttgart.de/handle/11682/2967 (2012)
Helldin, T., Falkman, G., Riveiro, M., Davidsson, S.: Presenting system uncertainty in automotive UIs for supporting trust calibration in autonomous driving. In: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 210–217 (2013). https://doi.org/10.1145/2516540.2516554
Horswill, M.S., McKenna, F.P.: Drivers’ hazard perception ability: Situation awareness on the road. In: A Cognitive Approach to Situation. https://www.researchgate.net/profile/Frank_Mckenna2/publication/271767727_Drivers’_hazard_perception_ability_Situation_awareness_on_the_road/links/57b2261d08ae95f9d8f4cb38/Drivers-hazard-perception-ability-Situation-awareness-on-the-road (2004)
Jung, T., tom Dieck, M.C., Lee, H., Chung, N.: Effects of virtual reality and augmented reality on visitor experiences in museum. In: Information and Communication Technologies in Tourism 2016, pp. 621–635 (2016). https://doi.org/10.1007/978-3-319-28231-2_45
Kawulich, B.B., D’Alba, A.: Teaching qualitative research methods with Second Life, a 3-dimensional online virtual environment. Virtual Reality 23(4), 375–384 (2019). https://doi.org/10.1007/s10055-018-0353-4
Kelley, J.F.: An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Infor. Syst. Secur. 2(1), 26–41 (1984). https://doi.org/10.1145/357417.357420
Kern, D., Müller, M., Schneegaß, S., Wolejko-Wolejszo, L., Schmidt, A.: CARS—Configurable automotive research simulator. In: Lucke, U., Kindsmüller, M.C., Fischer, S., Herczeg, M., Seehusen, S. (eds.) Workshop Proceedings der Tagungen Mensch & Computer 2008, DeLFI 2008 und Cognitive Design 2008, pp. 256–260. Logos Verlag (2008)
Kim, H., Gabbard, J.L.: Assessing distraction potential of augmented reality head-up displays for vehicle drivers. Hum. Factors 18720819844845 (2019). https://doi.org/10.1177/0018720819844845
Kim, H., Miranda Anon, A., Misu, T., Li, N., Tawari, A., Fujimura, K.: Look at me: augmented reality pedestrian warning system using an in-vehicle volumetric head up display. In: Proceedings of the 21st International Conference on Intelligent User Interfaces, pp. 294–298 (2016). https://doi.org/10.1145/2856767.2856815
Kim, S., Dey, A.K.: Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 133–142 (2009). https://doi.org/10.1145/1518701.1518724
Klüver, M., Herrigel, C., Heinrich, C., Schöner, H.-P., Hecht, H.: The behavioral validity of dual-task driving performance in fixed and moving base driving simulators. Transport. Res. Part F Traffic Psychol. Behav. 37, 78–96 (2016). https://doi.org/10.1016/j.trf.2015.12.005
Körber, M., Baseler, E., Bengler, K.: Introduction matters: Manipulating trust in automation and reliance in automated driving. Appl. Ergon. 66, 18–31 (2018). https://doi.org/10.1016/j.apergo.2017.07.006
Kunze, A., Summerskill, S.J., Marshall, R., Filtness, A.J.: Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics 62(3), 345–360 (2019). https://doi.org/10.1080/00140139.2018.1547842
Kunze, A., Summerskill, S.J., Marshall, R., Filtness, A.J.: Augmented reality displays for communicating uncertainty information in automated driving. In: Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 164–175 (2018). https://doi.org/10.1145/3239060.3239074
Langlois, S., Soualmi, B.: Augmented reality versus classical HUD to take over from automated driving: An aid to smooth reactions and to anticipate maneuvers. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 1571–1578 (2016). https://doi.org/10.1109/ITSC.2016.7795767
Lee, J.D., Moray, N.: Trust, self-confidence, and operators’ adaptation to automation. Int. J. Hum. -Comput. Stud. 40(1), 153–184 (1994). https://doi.org/10.1006/ijhc.1994.1007
Li, X., Schroeter, R., Rakotonirainy, A., Kuo, J., Lenné, M.G.: Effects of different non-driving-related-task display modes on drivers’ eye-movement patterns during take-over in an automated vehicle. Transport. Res. Part F Traffic Psychol. Behav. 70, 135–148. https://www.sciencedirect.com/science/article/pii/S136984781930659X (2020)
Lorenz, L., Kerschbaum, P., Schumann, J.: Designing take over scenarios for automated driving: how does augmented reality support the driver to get back into the loop? Proceedings of the Human Factors and Ergonomics Society ... Annual Meeting Human Factors and Ergonomics Society. Meeting, vol. 58, issue 1, pp. 1681–1685 (2014). https://doi.org/10.1177/1541931214581351
Medenica, Z., Kun, A.L., Paek, T., Palinko, O.: Augmented reality versus street views: a driving simulator study comparing two emerging navigation aids. In: Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 265–274 (2011). https://doi.org/10.1145/2037373.2037414
Milgram, P., Takemura, H., Utsumi, A.: Augmented reality: aclass of displays on the reality-virtuality continuum. Telemanipulator and. http://wiki.commres.org/pds/Project_7eNrf2010/_5.pdf(1994)
Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipul Telepresence Technol. 2351, 282–292 (1995). https://doi.org/10.1117/12.197321
Ng-Thow-Hing, V., Bark, K., Beckwith, L., Tran, C., Bhandari, R., Sridhar, S.: User-centered perspectives for automotive augmented reality. In: IEEE International Symposium on Mixed and Augmented Reality, pp. 13–22. http://www.academia.edu/download/35225563/ismar13-Aug12-libre.pdf (2013)
Pärsch, N., Harnischmacher, C., Baumann, M., Engeln, A., Krauß, L.: Designing augmented reality navigation visualizations for the vehicle: a question of real world object coverage? In: HCI in Mobility, Transport, and Automotive Systems, pp. 161–175 (2019). https://doi.org/10.1007/978-3-030-22666-4_12
Sadeghian Borojeni, S., Boll, S.C.J., Heuten, W., Bülthoff, H.H., Chuang, L.: Feel the movement: real motion influences responses to take-over requests in highly automated vehicles. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 246:1–246:13 (2018). https://doi.org/10.1145/3173574.3173820
SAE International: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (No. J3016 JUN2018). SAE International (2018)
Schömig, N., Wiedemann, K., Naujoks, F., Neukum, A., Leuchtenberg, B., Vöhringer-Kuhnt, T.: An augmented reality display for conditionally automated driving. In: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 137–141 (2018). https://doi.org/10.1145/3239092.3265956
Schroeter, R., Gerber, M.A.: A low-cost vr-based automated driving simulator for rapid automotive UI prototyping. In: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 248–251 (2018). https://doi.org/10.1145/3239092.3267418
Schroeter, R., Steinberger, F.: Pokémon DRIVE: towards increased situational awareness in semi-automated driving. In: Proceedings of the 28th Australian Conference on Computer-Human Interaction, pp. 25–29 (2016). https://doi.org/10.1145/3010915.3010973
Sommerauer, P., Müller, O.: Augmented reality in informal learning environments: a field experiment in a mathematics exhibition. Comput. Educ. 79, 59–68 (2014). https://doi.org/10.1016/j.compedu.2014.07.013
Soro, A., Rakotonirainy, A., Schroeter, R., Wollstädter, S.: Using augmented video to test in-car user experiences of context analog HUDs. In: Adjunct Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 1–6 (2014). https://doi.org/10.1145/2667239.2667302
Spies, R., Ablaßmeier, M., Bubb, H., Hamberger, W.: Augmented interaction and visualization in the automotive domain. In: Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction, pp. 211–220 (2009). https://doi.org/10.1007/978-3-642-02580-8_23
Steinberger, F., Schroeter, R., Lindner, V., Fitz-Walter, Z., Hall, J., Johnson, D.M.: Zombies on the road: a holistic design approach to balancing gamification and safe driving. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 320–327. https://eprints.qut.edu.au/84799/ (2015)
Tönnis, M., Klinker, G., Plavšic, M.: Survey and classification of head-up display presentation principles. In: Proceedings of the International Ergonomics Association (IEA). http://campar.in.tum.de/pub/toennis2009ieaDimensions/toennis2009ieaDimensions.pdf (2009)
Tran, C., Bark, K., Ng-Thow-Hing, V.: A left-turn driving aid using projected oncoming vehicle paths with augmented reality. In: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 300–307 (2013). https://doi.org/10.1145/2516540.2516581
Tu, H., Li, Z., Sun, Y., Dai, X.: Driving behavior in case of haze: a high fidelity driving simulator study. CICTP 2014, 2408–2421 (2014). https://doi.org/10.1061/9780784413623.230
von Sawitzky, T., Wintersberger, P., Riener, A., Gabbard, J.L.: Increasing trust in fully automated driving: route indication on an augmented reality head-up display. In: Proceedings of the 8th ACM International Symposium on Pervasive Displays, p. 6 (2019). https://doi.org/10.1145/3321335.3324947
Wintersberger, P., Frison, A.-K., Riener, A., von Sawitzky, T.: Fostering user acceptance and trust in fully automated vehicles: evaluating the potential of augmented reality. PRESENCE: Virtual Augment. Reality 27(1), 46–62 (2019). https://doi.org/10.1162/PRES_a_00320
Wintersberger, P., van der Heiden, R., Borojeni, S. S., Gerber, M.A., Green, P.: First workshop on attentive and pervasive UI in automated vehicles. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings, pp. 63–70 (2019). https://doi.org/10.1145/3349263.3350764
Wintersberger, P., von Sawitzky, T., Frison, A.-K., Riener, A.: Traffic augmentation as a means to increase trust in automated driving systems. In: Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter, 1–7 (2017). https://doi.org/10.1145/3125571.3125600
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Gerber, M.A., Faramarzian, M., Schroeter, R. (2022). Inception of Perception—Augmented Reality in Virtual Reality: Prototyping Human–Machine Interfaces for Automated Driving. In: Riener, A., Jeon, M., Alvarez, I. (eds) User Experience Design in the Era of Automated Driving. Studies in Computational Intelligence, vol 980. Springer, Cham. https://doi.org/10.1007/978-3-030-77726-5_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-77726-5_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77725-8
Online ISBN: 978-3-030-77726-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)