Keywords

1 Introduction

Advances in technology have greatly enhanced the quality of life by helping people perform tasks in large volumes and do calculations within seconds. Virtual Reality (VR) is one of the most important emerging technologies that has lead scientific research method to a new dimension. VR allows users to inhabit a simulated real-world environment using a head-mounted display (HMD). There are extended activities that can be performed within the VR environment through the use of controllers or haptic systems. This VR technology can be used to execute a broad range of real-life activities such as entertainment, education, training, medical procedures, physical activities, and so on.

There are many costs associated with performing practical experiments: experimental setup, replication of trials, change in scenarios, time and labor cost, surveillance control factors, etc. VR environment can eliminate most of these costs by providing a flexible mean for modification in scenarios and trials [5, 14, 15]. VR not only provides a three-dimensional (3D) environment but also the opportunity to interact with the objects to improve decision-making from both qualitative and quantitative perspectives [4]. Therefore, a useful application of Virtual Reality (VR) environment can be training employees to measure their performances before they are engaged in the real work environment [12, 17]. This implementation can be more crucial when the actual work environment involves several risks and hazards. However, designing a proper training interface in a virtual training environment is challenging considering trainees’ first-time exposure, experience level with the training module, and comfort with VR.

2 Related Work

Interface design in VR depends on users’ mental representation of the environment and realistic interaction with it. The challenge is to ensure that all VR components function similarly to the way the user would interact with the real world. Interaction modes should be selected based on the spatial movement requirements and task-types a user needs to perform in the VE. For risk assessment training using a VR environment, trainees require to select objects and enter data for hazard related surveys. Therefore, to make their interaction easy and realistic with virtual objects, the interface should be familiar, user-friendly, and easy to use. Trainees should feel comfortable enough to select an object, respond to the surveys based on their selection, make changes in their responses, and have the ability to skip, retry, go back, and exit the survey anytime they want.

Previous studies have considered different interfaces types to input and output information. In order to support symbolic interaction in a three-dimensional (3D) VR environment, input devices use voice commands, gestural commands, gaze, or menus floating in space while output devices use mostly visual, auditory, and haptic cues [18, 19]. Floating menu resembles a desktop screen or projector from the real world. In the virtual world, users require to use their fingers or some laser pointers, attached to the physical controllers of the VR system to interact with these floating menus. Schultheis et al. [16] reported that two-handed 3D interfaces to manipulate objects within the VR environment were considered natural and intuitive. In the real world, the user directly interacts with a physical object. Nevertheless, direct interaction is not possible in the virtual environment. In addition, the indirect interaction in VR takes much higher cognition levels and concentration to complete a task, which can eventually cause severe simulation sickness [7]. Having a VR interface similar to the real-world setting can reduce the cognition demands [7].

Literature shows different virtual windows types for 3D manipulation. Feiner et al. proposed 3 different windows types: surround-fixed windows (a fixed position interface in the VE that does not move with users’ movement), view-fixed window (windows move along with the user as they look around within the VE), object-fixed window (window is fixed, relative to a specific object in the VE. If the object moves, the window moves along with it) [8]. Without a physical support for manipulation, these windows provide limited user precision. Also, if the floating menu moves with the user, it blocks their view for the part of the environment. On the other hand, a menu, fixed at one position, would increase user head movements. To counter these problems, the researchers have introduced the ‘pen-and-tablet’ interfaces [2, 9]. With this approach, users can hold an object-fixed window in their non-dominant hand and interact with it using a finger from their dominant hand. These interfaces combine the use of a two-dimensional window interface with the necessary freedom provided by 3D manipulations. These hand-held windows are always within reach, move along with the user, and do not obstruct the user’s view.

In our experiment, we have built a VR environment representing a machine shop in an industrial manufacturing facility. The employees, especially the new hires, at the facility need to be trained and evaluated on their ability to identify different types of work hazards associated with the tools and machines. The purpose of the VR environment we have created is to provide a safe environment for the workers where they will be trained to identify the hazards associated with the tools and machines. In this particular study, we have built two different graphical user interfaces (GUI) as input and output modes for the purpose of interaction in the training system. These are: (1) a floating menu interface controlled by pointers triggered using controllers and (2) a tablet interface controlled by touch with the virtual index finger. The project’s scope was to compare the usability of these two user interfaces (UI) in a virtual reality (VR) environment. The participants of the study interacted separately with the two different user interfaces in the VR environment and filled out surveys afterward, reflecting on their experience with these user interfaces. Comparison and analysis have been performed on the collected feedback of the participants. The goal of this study was to analyze the survey results to understand how these interfaces can most effectively be designed to enhance user performance in training and evaluation.

3 Developed Virtual Reality Environment

3.1 VR Environment

We have developed a VR environment that represents a machine shop in an industrial manufacturing facility. Several machines and hand tools (drilling machine, toolbox, hammer, wrench, screwdriver etc.) were placed on the machine shop shelves, tables, and floor, within the environment.

3.2 Tools

Hardware. For this study, we used the Oculus Rift which comes with an Oculus Headset (HMD), left and right touch controllers, and two motion sensors. Using this VR hardware connected to a device which supports the VR application, i.e. a computer with enough memory and enhanced graphics cards, users can visualize and interact with the VR environment.

Software. We used Unity3D (version 2017.1.1f1), a very popular tool to build VR environments, to develop the virtual machine shop environment. The built-in models help to create a quick prototype of the intended environment. Unity 3D makes an asset store available that has thousands of free or purchasable assets that can be easily integrated with the environment that is being developed. Unity3D supports several virtual reality SDK packages that can be downloaded and used for various VR hardware types. Additionally, there are a lot of other tool-kits to build applications that can be run using any hardware. VRTK [1] is the toolkit we used for building our environment. It supports all major VR hardware and provides easy-to-use scripts. The VRTK toolkit also provides cross-platform open-source support for user interaction in VR.

4 Experimental Procedures

The Institutional Review Board of Mississippi State University reviewed and approved the protocol used in the study. The purpose of this proposed study was to investigate participants’ perceptions of two different user interfaces (UI) for interacting with a virtual environment. These user interfaces were displayed to the participants as menus with which they interacted. Participant’s interaction with the virtual environment was recorded using Camtasia. After completion of the interaction, participants filled out surveys. The survey results, as well as objective measures of the average interaction time required to complete the defined tasks, were used to evaluate the two interfaces in terms of usability and navigation measures.

4.1 Participants

Thirteen participants were recruited from the Starkville Mississippi area. Participants provided consent before participation. The participants, aged 18–60 years, were fluent English speakers, had normal vision, either naturally or with glasses or contact lenses, and had the physical ability to work in a machine shop environment. Participants who had limitations (such as tremors) that restrict their ability to hold and touch controllers to correctly point to objects, buttons, checkbox etc. were excluded from the study. For these people, it would be difficult and sometimes impossible to interact with the interface. Language efficiency and normal vision are necessary in order to interpret the interface contents and instructions correctly and perform tasks accordingly. The experiment took around 30 min to complete and participants were compensated with $10 for their time and participation. Table 1 presents an overview of the demographic information of the participants.

Table 1. Participant information for the VR interface comparison study.
Full size table

4.2 Experimental Setup

The experiment was performed in the Human Performance Lab at the Center for Advanced Vehicular Systems, Starkville, Mississippi (CAVS). This experiment did not require much movement from the participant around the VR environment; however, the lab had ample spaces to move freely around the environment after we set up the Oculus VR hardware. Two Oculus motion sensors were placed facing the participants’ standing position. The Oculus headset and the touch controller were placed on a table, along with the computer that ran the experimental VR environment.

4.3 Study Design

The study was divided into 3 parts. The first part was a ’practice’ session where the participants were asked to wear the headsets and use the touch controller to familiarize themselves with the VR environment and the general method for interacting with the components within the environment. The practice session took around 5 min to complete. In the second and third parts, they experienced two different interfaces: the floating menu and the tablet. The interfaces were randomly assigned; 7 participants used the floating menu as their first interface while the remaining 6 participants used the tablet as their first interface; afterwards, they switched. Each of these interactions lasted approximately 10 min (Fig. 1).

Fig. 1.
figure 1

Two interfaces tested in this study. (a) Floating menu and (b) Tablet

Full size image

4.4 Task Description

Each participant was assigned a list of tasks for the risk assessment training within a VR machine shop environment using each interfaces. They needed to select an object from the machine shop, identify different hazard types associated with the object, determine each hazard’s severity, and select personal protective equipment/s to protect themselves from that specific hazard. Participants then repeated these steps for another object in the environment and continued for the 10-minute interaction time. The participants had to pick their responses from the survey containing multiple choice-based questions. Two proposed interfaces provided different ways to interact with the objects and respond to hazard-related surveys.

Floating Menu Interface. During their interaction with the floating menu, the participants pointed at and highlighted a particular object, then clicked on the touch controller’s trigger button to bring up a floating interface. This interface appeared in the participants’ perspective view displaying hazard-related information for the highlighted object. Then the participants used the controller’s pointer to interact with other components on the floating menu interface (i.e. Check-box, radio button, button, etc.). The participants browsed through several pages on the interface to provide responses related to the selected object, such as hazard type, severity, required personal protective equipment, etc. After completing the data entry, the participants exited VR and completed the survey for the floating interface.

Tablet Interface. For the tablet interface, a virtual tablet (hand-held electronic device) was attached to the participant’s touch controller. The participant performed the same steps to point, highlight and click on a particular object with the touch controller that had the tablet attached to it. The participants were able to view avatars of both of their hands attached to the controller. This helped them to visualize the different hand gestures performed on the virtual objects with the touch controllers such as touching, grabbing, trigger-clicking etc. Participants used their hands (index fingertip) to interact with the tablet interface through virtual touch and were able to take picture of an object using the camera button. These gestures are similar to the real-life tablet interactions. This session ended with the participants completing the data entry for this interface. After leaving VR, the participants completed a survey on the tablet interface.

4.5 Survey Instruments and Administration

The participants responded to a Simulation Sickness Questionnaire (SSQ, [11]) before their participation and after each time their exposure to one of the interfaces. The SSQ included 16 items, each having the ratings of “None = 1”, “Slight = 2”, “Moderate = 3” and “Severe = 4”. Any rating that showed an SSQ score> 5 or any item scored as “Severe” would trigger the discontinuation of the study. In our study, no participants were withdrawn due to the SSQ score. After interacting with each of the interfaces, participants also filled out surveys on the realism of the interfaces and System Usability Scale (SUS) rating [3]. Many characteristics of the visual interfaces within the VR were intended to successfully immerse participants, such as the screen size, image quality, text size, and the body movements required to accomplish tasks [6, 20]. The researchers asked the participants to rate proposed interfaces from “1” (Not realistic) to “5” (Realistic) for 11 characteristics in order to test the realism of the interfaces in the VR environment as compared to the real-world user interfaces. The SUS is a very popular tool to assess response to software, websites or any other digital interfaces. The SUS, modified for the interfaces used in the current study, contained 10 questions with a 5-point rating for each of them (from “1” denoting Strongly Disagree to “5” denoting Strongly Agree). At the end, the participants responded to the demographic questions (gender, age, education level, experience and frequency of daily usage of digital UIs) along with other surveys for the observed interface.

5 Results and Discussion

The survey results indicate the strengths and weaknesses of the two interface types. The evaluation of the interfaces will help in the design and selection of a useful mode of interaction within the VR. The following tables list the results collected from the survey responses of the 13 participants. Table 2 contains the means (M) and standard deviations (SD) of the ratings for different characteristics of the interfaces along with the pairwise comparison outcomes (t statistics and p values) for each of the items. Table 3 contains the same statistics for each of the interfaces’ SUS ratings.

5.1 Realism

Face validity was conducted by testing realism for the interfaces based on several constructs related to their designs and factors affecting interaction with each interface. Pairwise comparisons using t-tests showed significant differences between the two interfaces with respect to interface placement, screen size, graphics and text quality. For each of the constructs, the tablet interface was rated to be more realistic than the floating menu. On a 5-point scale for realism rating, the tablet received scores greater than 3.5 for all of the constructs and scores greater than 4 for interface setting, placement, and size and for head movements required for interaction. These higher scores confirm the realistic appearance of a tablet interface in the VR environment which provided participants with better placement, reasonable size, and quality graphics and text within the VR.

Consistent with previous studies using a tablet as an interaction interface within the VR environment [10, 13], this study recommends using a tablet interface for object selection and data entry with novice and inexperienced employees. Tablets have a similar interface to cell phones which is familiar to almost everyone. Therefore, it is clear that people will know the basic functionality required to accomplish a task using the tablet interface. In addition, the participants’ avatar hands may have made the interaction more realistic with a tablet. Watching themselves working on a tablet and getting the expected outcome could explain the results for this interface regarding realism.

Table 2. Analysis of realism test results.
Full size table
Table 3. Analysis of system usability scale (SUS) ratings.
Full size table

5.2 System Usability Scale (SUS) Ratings

Pairwise comparisons and descriptive statistics for SUS scores revealed that participants only perceived significant differences between the two interfaces for two survey items. The results showed that participants found the floating interface to be a bit more cumbersome to use than the tablet. They responses indicated that significant technical support would be needed to use the floating menu as compared to the tablet interface. According to the standard definition of the SUS scaling factors, any interface with a SUS score higher than 68 can be considered to be good, and in both of the cases, the SUS scores are much higher than 68. In addition, no significant difference was found between the interfaces with respect to the overall SUS score. Therefore, it can be said that both interfaces were recognized to be useful for interacting within VR, but the tablet gives a more realistic interaction effect in terms of its placement, size, graphics and text quality.

5.3 Average Interaction Time

The average time to complete the tasks on one object (time between selecting an object and finishing the entering of hazard information) was found to be significantly different [t = −6.021, df = 12, p => 0.0001]. For the tablet interface, the average time was about 68 s with a standard deviation of 9 s, while for the floating menu, this was around 93 s with a standard deviation of 4 s. With the floating menu, the participants required a longer start-up time as well as more time to locate the floating screen and point the trigger to the correct place. The longer time may be the reason for unfamiliar settings with the VR controllers for multiple functions, such as pointing at the object, selecting the object, and filling out surveys on the floating menu. With the tablet interface, the familiar camera button and hand avatar helped the participants to use the interfaces properly.

6 Conclusion

The current study investigated employees’ preference for the virtual interface in risk assessment training and evaluation. The study allowed participants to experience a VR machine shop environment and interact with different objects using two proposed interfaces. Each interface had different methods of interaction, designed based on the task requirement and recommendations from previous studies in this area. The inclusion of 2D interfaces for 3D manipulation in risk assessment training and evaluation identified effective design criteria and suggestions for improvement.

The results of this study showed that the use of the standard metrics such as SUS, realism test with basic interface characteristics, and average task-completion time gave an overall idea of the usability of the interfaces. The quick and easy approach of the VR experiment brought out the important features of tablet interface (interface placement, screen size, and quality of graphics and text) that made it more appealing to the users. The study also revealed important aspects of the usability for a virtual UI. Users found the floating menu cumbersome to use and they needed more technical support to use the menu interface. However, further improvement of the interfaces will allow the user to respond more specifically which will make the comparison more robust.