Abstract
The simulation of a real-life environment in VR greatly reduces the time and cost to perform experiments. A useful application of Virtual Reality (VR) can be training employees and measuring their performances before their assignment in the real work environment. For this study, an experimental environment was created using VR to represent a machine shop in an industrial manufacturing facility. The VR provided with a safe environment for trainees to correctly identify hazards associated with each machine. A comparative study was conducted to evaluate two different ways a trainee can interact with the training system within the VR environment. Participants in the study were asked to perform training tasks with both user interfaces and complete user experience and usability questionnaires. The evaluation of interfaces played an important role in the design and selection of a useful mode of interaction within the VR environment.
Access provided by CONRICYT-eBooks. Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Advances in technology have greatly enhanced the quality of life by helping people perform tasks in large volumes and do calculations within seconds. Virtual Reality (VR) is one of the most important emerging technologies that has lead scientific research method to a new dimension. VR allows users to inhabit a simulated real-world environment using a head-mounted display (HMD). There are extended activities that can be performed within the VR environment through the use of controllers or haptic systems. This VR technology can be used to execute a broad range of real-life activities such as entertainment, education, training, medical procedures, physical activities, and so on.
There are many costs associated with performing practical experiments: experimental setup, replication of trials, change in scenarios, time and labor cost, surveillance control factors, etc. VR environment can eliminate most of these costs by providing a flexible mean for modification in scenarios and trials [5, 14, 15]. VR not only provides a three-dimensional (3D) environment but also the opportunity to interact with the objects to improve decision-making from both qualitative and quantitative perspectives [4]. Therefore, a useful application of Virtual Reality (VR) environment can be training employees to measure their performances before they are engaged in the real work environment [12, 17]. This implementation can be more crucial when the actual work environment involves several risks and hazards. However, designing a proper training interface in a virtual training environment is challenging considering trainees’ first-time exposure, experience level with the training module, and comfort with VR.
2 Related Work
Interface design in VR depends on users’ mental representation of the environment and realistic interaction with it. The challenge is to ensure that all VR components function similarly to the way the user would interact with the real world. Interaction modes should be selected based on the spatial movement requirements and task-types a user needs to perform in the VE. For risk assessment training using a VR environment, trainees require to select objects and enter data for hazard related surveys. Therefore, to make their interaction easy and realistic with virtual objects, the interface should be familiar, user-friendly, and easy to use. Trainees should feel comfortable enough to select an object, respond to the surveys based on their selection, make changes in their responses, and have the ability to skip, retry, go back, and exit the survey anytime they want.
Previous studies have considered different interfaces types to input and output information. In order to support symbolic interaction in a three-dimensional (3D) VR environment, input devices use voice commands, gestural commands, gaze, or menus floating in space while output devices use mostly visual, auditory, and haptic cues [18, 19]. Floating menu resembles a desktop screen or projector from the real world. In the virtual world, users require to use their fingers or some laser pointers, attached to the physical controllers of the VR system to interact with these floating menus. Schultheis et al. [16] reported that two-handed 3D interfaces to manipulate objects within the VR environment were considered natural and intuitive. In the real world, the user directly interacts with a physical object. Nevertheless, direct interaction is not possible in the virtual environment. In addition, the indirect interaction in VR takes much higher cognition levels and concentration to complete a task, which can eventually cause severe simulation sickness [7]. Having a VR interface similar to the real-world setting can reduce the cognition demands [7].
Literature shows different virtual windows types for 3D manipulation. Feiner et al. proposed 3 different windows types: surround-fixed windows (a fixed position interface in the VE that does not move with users’ movement), view-fixed window (windows move along with the user as they look around within the VE), object-fixed window (window is fixed, relative to a specific object in the VE. If the object moves, the window moves along with it) [8]. Without a physical support for manipulation, these windows provide limited user precision. Also, if the floating menu moves with the user, it blocks their view for the part of the environment. On the other hand, a menu, fixed at one position, would increase user head movements. To counter these problems, the researchers have introduced the ‘pen-and-tablet’ interfaces [2, 9]. With this approach, users can hold an object-fixed window in their non-dominant hand and interact with it using a finger from their dominant hand. These interfaces combine the use of a two-dimensional window interface with the necessary freedom provided by 3D manipulations. These hand-held windows are always within reach, move along with the user, and do not obstruct the user’s view.
In our experiment, we have built a VR environment representing a machine shop in an industrial manufacturing facility. The employees, especially the new hires, at the facility need to be trained and evaluated on their ability to identify different types of work hazards associated with the tools and machines. The purpose of the VR environment we have created is to provide a safe environment for the workers where they will be trained to identify the hazards associated with the tools and machines. In this particular study, we have built two different graphical user interfaces (GUI) as input and output modes for the purpose of interaction in the training system. These are: (1) a floating menu interface controlled by pointers triggered using controllers and (2) a tablet interface controlled by touch with the virtual index finger. The project’s scope was to compare the usability of these two user interfaces (UI) in a virtual reality (VR) environment. The participants of the study interacted separately with the two different user interfaces in the VR environment and filled out surveys afterward, reflecting on their experience with these user interfaces. Comparison and analysis have been performed on the collected feedback of the participants. The goal of this study was to analyze the survey results to understand how these interfaces can most effectively be designed to enhance user performance in training and evaluation.
3 Developed Virtual Reality Environment
3.1 VR Environment
We have developed a VR environment that represents a machine shop in an industrial manufacturing facility. Several machines and hand tools (drilling machine, toolbox, hammer, wrench, screwdriver etc.) were placed on the machine shop shelves, tables, and floor, within the environment.
3.2 Tools
Hardware. For this study, we used the Oculus Rift which comes with an Oculus Headset (HMD), left and right touch controllers, and two motion sensors. Using this VR hardware connected to a device which supports the VR application, i.e. a computer with enough memory and enhanced graphics cards, users can visualize and interact with the VR environment.
Software. We used Unity3D (version 2017.1.1f1), a very popular tool to build VR environments, to develop the virtual machine shop environment. The built-in models help to create a quick prototype of the intended environment. Unity 3D makes an asset store available that has thousands of free or purchasable assets that can be easily integrated with the environment that is being developed. Unity3D supports several virtual reality SDK packages that can be downloaded and used for various VR hardware types. Additionally, there are a lot of other tool-kits to build applications that can be run using any hardware. VRTK [1] is the toolkit we used for building our environment. It supports all major VR hardware and provides easy-to-use scripts. The VRTK toolkit also provides cross-platform open-source support for user interaction in VR.
4 Experimental Procedures
The Institutional Review Board of Mississippi State University reviewed and approved the protocol used in the study. The purpose of this proposed study was to investigate participants’ perceptions of two different user interfaces (UI) for interacting with a virtual environment. These user interfaces were displayed to the participants as menus with which they interacted. Participant’s interaction with the virtual environment was recorded using Camtasia. After completion of the interaction, participants filled out surveys. The survey results, as well as objective measures of the average interaction time required to complete the defined tasks, were used to evaluate the two interfaces in terms of usability and navigation measures.
4.1 Participants
Thirteen participants were recruited from the Starkville Mississippi area. Participants provided consent before participation. The participants, aged 18–60 years, were fluent English speakers, had normal vision, either naturally or with glasses or contact lenses, and had the physical ability to work in a machine shop environment. Participants who had limitations (such as tremors) that restrict their ability to hold and touch controllers to correctly point to objects, buttons, checkbox etc. were excluded from the study. For these people, it would be difficult and sometimes impossible to interact with the interface. Language efficiency and normal vision are necessary in order to interpret the interface contents and instructions correctly and perform tasks accordingly. The experiment took around 30 min to complete and participants were compensated with $10 for their time and participation. Table 1 presents an overview of the demographic information of the participants.
4.2 Experimental Setup
The experiment was performed in the Human Performance Lab at the Center for Advanced Vehicular Systems, Starkville, Mississippi (CAVS). This experiment did not require much movement from the participant around the VR environment; however, the lab had ample spaces to move freely around the environment after we set up the Oculus VR hardware. Two Oculus motion sensors were placed facing the participants’ standing position. The Oculus headset and the touch controller were placed on a table, along with the computer that ran the experimental VR environment.
4.3 Study Design
The study was divided into 3 parts. The first part was a ’practice’ session where the participants were asked to wear the headsets and use the touch controller to familiarize themselves with the VR environment and the general method for interacting with the components within the environment. The practice session took around 5 min to complete. In the second and third parts, they experienced two different interfaces: the floating menu and the tablet. The interfaces were randomly assigned; 7 participants used the floating menu as their first interface while the remaining 6 participants used the tablet as their first interface; afterwards, they switched. Each of these interactions lasted approximately 10 min (Fig. 1).
4.4 Task Description
Each participant was assigned a list of tasks for the risk assessment training within a VR machine shop environment using each interfaces. They needed to select an object from the machine shop, identify different hazard types associated with the object, determine each hazard’s severity, and select personal protective equipment/s to protect themselves from that specific hazard. Participants then repeated these steps for another object in the environment and continued for the 10-minute interaction time. The participants had to pick their responses from the survey containing multiple choice-based questions. Two proposed interfaces provided different ways to interact with the objects and respond to hazard-related surveys.
Floating Menu Interface. During their interaction with the floating menu, the participants pointed at and highlighted a particular object, then clicked on the touch controller’s trigger button to bring up a floating interface. This interface appeared in the participants’ perspective view displaying hazard-related information for the highlighted object. Then the participants used the controller’s pointer to interact with other components on the floating menu interface (i.e. Check-box, radio button, button, etc.). The participants browsed through several pages on the interface to provide responses related to the selected object, such as hazard type, severity, required personal protective equipment, etc. After completing the data entry, the participants exited VR and completed the survey for the floating interface.
Tablet Interface. For the tablet interface, a virtual tablet (hand-held electronic device) was attached to the participant’s touch controller. The participant performed the same steps to point, highlight and click on a particular object with the touch controller that had the tablet attached to it. The participants were able to view avatars of both of their hands attached to the controller. This helped them to visualize the different hand gestures performed on the virtual objects with the touch controllers such as touching, grabbing, trigger-clicking etc. Participants used their hands (index fingertip) to interact with the tablet interface through virtual touch and were able to take picture of an object using the camera button. These gestures are similar to the real-life tablet interactions. This session ended with the participants completing the data entry for this interface. After leaving VR, the participants completed a survey on the tablet interface.
4.5 Survey Instruments and Administration
The participants responded to a Simulation Sickness Questionnaire (SSQ, [11]) before their participation and after each time their exposure to one of the interfaces. The SSQ included 16 items, each having the ratings of “None = 1”, “Slight = 2”, “Moderate = 3” and “Severe = 4”. Any rating that showed an SSQ score> 5 or any item scored as “Severe” would trigger the discontinuation of the study. In our study, no participants were withdrawn due to the SSQ score. After interacting with each of the interfaces, participants also filled out surveys on the realism of the interfaces and System Usability Scale (SUS) rating [3]. Many characteristics of the visual interfaces within the VR were intended to successfully immerse participants, such as the screen size, image quality, text size, and the body movements required to accomplish tasks [6, 20]. The researchers asked the participants to rate proposed interfaces from “1” (Not realistic) to “5” (Realistic) for 11 characteristics in order to test the realism of the interfaces in the VR environment as compared to the real-world user interfaces. The SUS is a very popular tool to assess response to software, websites or any other digital interfaces. The SUS, modified for the interfaces used in the current study, contained 10 questions with a 5-point rating for each of them (from “1” denoting Strongly Disagree to “5” denoting Strongly Agree). At the end, the participants responded to the demographic questions (gender, age, education level, experience and frequency of daily usage of digital UIs) along with other surveys for the observed interface.
5 Results and Discussion
The survey results indicate the strengths and weaknesses of the two interface types. The evaluation of the interfaces will help in the design and selection of a useful mode of interaction within the VR. The following tables list the results collected from the survey responses of the 13 participants. Table 2 contains the means (M) and standard deviations (SD) of the ratings for different characteristics of the interfaces along with the pairwise comparison outcomes (t statistics and p values) for each of the items. Table 3 contains the same statistics for each of the interfaces’ SUS ratings.
5.1 Realism
Face validity was conducted by testing realism for the interfaces based on several constructs related to their designs and factors affecting interaction with each interface. Pairwise comparisons using t-tests showed significant differences between the two interfaces with respect to interface placement, screen size, graphics and text quality. For each of the constructs, the tablet interface was rated to be more realistic than the floating menu. On a 5-point scale for realism rating, the tablet received scores greater than 3.5 for all of the constructs and scores greater than 4 for interface setting, placement, and size and for head movements required for interaction. These higher scores confirm the realistic appearance of a tablet interface in the VR environment which provided participants with better placement, reasonable size, and quality graphics and text within the VR.
Consistent with previous studies using a tablet as an interaction interface within the VR environment [10, 13], this study recommends using a tablet interface for object selection and data entry with novice and inexperienced employees. Tablets have a similar interface to cell phones which is familiar to almost everyone. Therefore, it is clear that people will know the basic functionality required to accomplish a task using the tablet interface. In addition, the participants’ avatar hands may have made the interaction more realistic with a tablet. Watching themselves working on a tablet and getting the expected outcome could explain the results for this interface regarding realism.
5.2 System Usability Scale (SUS) Ratings
Pairwise comparisons and descriptive statistics for SUS scores revealed that participants only perceived significant differences between the two interfaces for two survey items. The results showed that participants found the floating interface to be a bit more cumbersome to use than the tablet. They responses indicated that significant technical support would be needed to use the floating menu as compared to the tablet interface. According to the standard definition of the SUS scaling factors, any interface with a SUS score higher than 68 can be considered to be good, and in both of the cases, the SUS scores are much higher than 68. In addition, no significant difference was found between the interfaces with respect to the overall SUS score. Therefore, it can be said that both interfaces were recognized to be useful for interacting within VR, but the tablet gives a more realistic interaction effect in terms of its placement, size, graphics and text quality.
5.3 Average Interaction Time
The average time to complete the tasks on one object (time between selecting an object and finishing the entering of hazard information) was found to be significantly different [t = −6.021, df = 12, p => 0.0001]. For the tablet interface, the average time was about 68 s with a standard deviation of 9 s, while for the floating menu, this was around 93 s with a standard deviation of 4 s. With the floating menu, the participants required a longer start-up time as well as more time to locate the floating screen and point the trigger to the correct place. The longer time may be the reason for unfamiliar settings with the VR controllers for multiple functions, such as pointing at the object, selecting the object, and filling out surveys on the floating menu. With the tablet interface, the familiar camera button and hand avatar helped the participants to use the interfaces properly.
6 Conclusion
The current study investigated employees’ preference for the virtual interface in risk assessment training and evaluation. The study allowed participants to experience a VR machine shop environment and interact with different objects using two proposed interfaces. Each interface had different methods of interaction, designed based on the task requirement and recommendations from previous studies in this area. The inclusion of 2D interfaces for 3D manipulation in risk assessment training and evaluation identified effective design criteria and suggestions for improvement.
The results of this study showed that the use of the standard metrics such as SUS, realism test with basic interface characteristics, and average task-completion time gave an overall idea of the usability of the interfaces. The quick and easy approach of the VR experiment brought out the important features of tablet interface (interface placement, screen size, and quality of graphics and text) that made it more appealing to the users. The study also revealed important aspects of the usability for a virtual UI. Users found the floating menu cumbersome to use and they needed more technical support to use the menu interface. However, further improvement of the interfaces will allow the user to respond more specifically which will make the comparison more robust.
References
Vrtk - virtual reality toolkit (2018). http://vrtoolkit.readme.io. Accessed 5 June 2018
Bowman, D.A., Wineman, J.D., Hodges, L.F.: Exploratory design of animal habitats within an immersive virtual environment. Technical report, Georgia Institute of Technology (1998)
Brooke, J., et al.: SUS-A quick and dirty usability scale. In: Usability Evaluation in Industry, vol. 189(194), pp. 4–7 (1996)
Chen, P.Q.: Virtual reality in design and manufacturing. Ph.D. thesis, Nanyang Technological University, School of Mechanical and Production Engineering (2002)
Deb, S., Carruth, D.W., Sween, R., Strawderman, L., Garrison, T.M.: Efficacy of virtual reality in pedestrian safety research. Appl. Ergon. 65, 449–460 (2017)
Ditton, T.B.: The unintentional blending of direct experience and mediated experience: the role of enhanced versus limited television presentations in inducing source-monitoring errors (1998)
Federoff, M.A.: Heuristics and usability guidelines for the creation and evaluation of fun in video games. Ph.D. thesis, Citeseer (2002)
Feiner, S., MacIntyre, B., Haupt, M., Solomon, E.: Windows on the world: 2D windows for 3D augmented reality. In: Proceedings of the 6th Annual ACM Symposium on User Interface Software and Technology, pp. 145–155. ACM (1993)
Fuhrmann, A., Loffelmann, H., Schmalstieg, D., Gervautz, M.: Collaborative visualization in augmented reality. IEEE Comput. Graph. Appl. 18(4), 54–59 (1998)
Grudin, J.: The case against user interface consistency. Commun. ACM 32(10), 1164–1173 (1989)
Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)
Mujber, T.S., Szecsi, T., Hashmi, M.S.: Virtual reality applications in manufacturing process simulation. J. Mater. Process. Technol. 155, 1834–1838 (2004)
Rekimoto, J.: Tilting operations for small screen interfaces. In: Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology, pp. 167–168. ACM (1996)
Rizzo, A.A., Buckwalter, J.G., Neumann, U., Kesselman, C., Thiébaux, M.: Basic issues in the application of virtual reality for the assessment and rehabilitation of cognitive impairments and functional disabilities. CyberPsychology Behav. 1(1), 59–78 (1998)
Rose, F., Attree, E.A., Brooks, B., Parslow, D., Penn, P.: Training in virtual environments: transfer to real world tasks and equivalence to real task training. Ergonomics 43(4), 494–511 (2000)
Schultheis, U., Jerald, J., Toledo, F., Yoganandan, A., Mlyniec, P.: Comparison of a two-handed interface to a wand interface and a mouse interface for fundamental 3D tasks. In: 2012 IEEE Symposium on 3D User Interfaces (3DUI), pp. 117–124. IEEE (2012)
Squelch, A.: Virtual reality for mine safety training in south africa. J. South. Afr. Inst. Min. Metall. 101(4), 209–216 (2001)
Stork, D.G., Hennecke, M.E.: Speechreading by Humans and Machines: Models, Systems, and Applications, vol. 150. Springer Science & Business Media, Heidelberg (2013)
Vo, M.T., Wood, C.: Building an application framework for speech and pen input integration in multimodal learning interfaces. In: 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP-1996, Conference Proceedings, vol. 6, pp. 3545–3548. IEEE (1996)
Wang, J., Lindeman, R.: Coordinated 3D interaction in tablet-and HMD-based hybrid virtual environments. In: Proceedings of the 2nd ACM Symposium on Spatial User Interaction, pp. 70–79. ACM (2014)
Acknowledgement
The authors wish to thank Toyota Motor Manufacturing, Mississippi for their assistance with risk assessment training module development.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Bushra, N., Carruth, D., Deb, S. (2018). A Comparative Study of Virtual UI for Risk Assessment and Evaluation. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2018. Lecture Notes in Computer Science(), vol 11241. Springer, Cham. https://doi.org/10.1007/978-3-030-03801-4_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-03801-4_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03800-7
Online ISBN: 978-3-030-03801-4
eBook Packages: Computer ScienceComputer Science (R0)