Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Virtual Reality (VR) gaming platforms have become quite popular, with VR headsets such as the Oculus Rift [22], HTC Vive [15], PlayStation VR [25] and Samsung Gear VR [23] being released recently. In spite of their novelty, these devices are not part of the first wave of attempts to bring VR into the consumer market. Back in 1995, the Virtual Boy [8], released by Nintendo, stands out as one of the exemplars of the first wave of commercial virtual reality devices, which did not manage to establish a permanent market for the technology. Some of the reasons for the failure of earlier waves to establish permanent markets include a lack of usability, reduced user adoption, and the high cost of earlier VR technologies.

The most recent rising of commercially available Virtual Reality owes credit in part to Palmer Luckey [7] who founded Oculus back in 2012.

Fig. 1.
figure 1

Experimental setup for operation of virtual panels through hand gestures.

Full size image

His Oculus VR headsets set off a new wave of interest in Virtual Reality which continues until today. In the meantime, Apple [29], Samsung and HTC helped popularize the modern smartphone with high-resolution screens, accurate motion (and other) sensors and compact form factors that made them an appropriate fit to the technology required to make convincing VR.

As Google Cardboard [10] first illustrated in 2014 and later Samsung with the Gear VR in 2015, a low-cost, affordable and more casual form of VR using Smartphones is possible. By sliding a smartphone into a head-mounted display, users can get access into relatively simple virtual worlds. In addition to this, there was another significant development in the hardware that contributed to a more complete and affordable VR experience: portable, short-range depth sensing cameras [2]. Portable and short range depth sensors allow computing devices such as PCs, laptops and smartphones to capture its immediate environment as well as help identify a user’s hands and/or face in the proximity of the devices [13, 21, 24, 28].

In this work, we use the Leap Motion as the depth sensing camera and as a gesture recognition device. As shown in Fig. 1, the Leap Motion controller is originally designed to be placed on a physical desktop, facing upward, or it can also be mounted on a VR headset for first-person interaction [1]. The device uses three LED emitters to illuminate the surrounding space with IR light, which is reflected back from the nearby objects and captured by two IR cameras [18]. The device’s software installed on the connected PC will analyze the captured images in real-time, determine the position of objects, and perform the recognition of users’ hands and fingers. Its suitability for a variety of tasks has been the focus of recent research [1, 3, 12, 27].

The motivation of this research is to explore VR gesture control with a low-cost combination of equipment, including a smartphone, an infrared controller and a VR headset, to address implementation challenges and propose solutions to overcome the major barriers to being able to control, with hand motions and gestures, a simple VR game. In this research, we tested our software with four users, who tested the system under the 4 conditions described below (2 hardware setups X 2 applications) and provided unstructured feedback on their experience. Through the implementation of this project, we show how a low cost solution to develop VR applications using a laptop, Unity, a VR Headset, an Android phone and the Leap Motion controller can be used to address existing limitations and suggest solutions to usability issues in these platforms.

2 Implementation

2.1 Project Setup

This research is implemented in Unity 5.3 on OS X El Capitan system on a MacBook PRO. In addition, we have the Leap Motion SDK_Mac_2.3.1, Splashtop Streamer [26], as well as Leap Motion Core Asset 2_3_1 [19] package installed in the Unity engine. Meanwhile, the Samsung S5 phone is equipped with the Splashtop Streamer client and is inserted in the Freefly VR [9] headset.

The Leap Motion SDK connects the Leap Motion controller to the computer, allowing the captured data from the Leap motion to be delivered to the analyzing software on the computer in real-time (see Fig. 2). The Leap Motion Core Asset package, which is imported into the Unity engine, will function as the connector between the Unity engine and the Leap Motion analyzing software, which translates the input data from the analyzing software as a control signal and generate the corresponding hand model and gestures. The Splashtop Streamer mirrors the PC screen onto any mobile devices with the Splashstop client installed and in the same network. This novel setup allows us to avoid Android depolyment, as the program executes direclty from the laptop to the HMD. This makes development much faster than the process of deploying an executable for the HMD (an Android phone) each time the source code changes. A prototype in development can be tested using the actual HMD rather than on a software android simulator running on the laptop.

Fig. 2.
figure 2

Software setup flow chart.

Full size image

As for the Hardware part, the Leap Motion sensor is always connected to the PC through the USB port, whereas the Samsung phone is wirelessly connected, mirroring the PC screen. The Samsung Phone is slid into the Free Fly VR headset and calibrated for horizontal and vertical axis with the help of the Free Fly VR SDK (see Fig. 3). In this research, we tested two conditions with regards to the location of the Leap motion controller, either horizontally placed on the desk with the device looking up (referred to as “desk-mounted”) or mounted on top of the VR headset facing away from the user’s head (or “head-mounted”).

Fig. 3.
figure 3

Hardware setup.

Full size image

2.2 Applications

Dial Panels. To explore the experience of point touch accuracy using the Leap Motion gesture controller, we simulated the dial panel with 9 number buttons and two symbol buttons: # and * (see Fig. 4). The button’s color will change in response to users’ actions upon detection of a collision with the toggle buttons of the panel. A box collider component placed on each button allows buttons to react to a virtual collision with the hand model generated by the Leap Motion SDK. Multiple canvas layers on each button work under the button color changing script, allowing it to change colour when it is picked by the hand model to provide a clear visual feedback.

Shadow casting has been previously found useful to localize objects in virtual environments [6, 11, 16]. In our system, a shadow of the controlling hand is cast along with the hand model on a panel plane beneath the buttons layer, to provide a sense of the distance between the hand and the panel through visual feedback.

During the implementation process, we found that the hand model generated sometimes did not accurately reflect the real world’s hand pose or position. In addition, sometime multiple fingers would trigger simultaneous collisions on the panel. This was especially confusing for the user, as most users typically intended to push a button with just one of their fingers. For this reason, we modified the collision detection to be limited to the index finger, so only one finger can trigger the collision detection action.

Games. We also implemented the well-known Tetris game [20], which is displayed within the VR environment (see Fig. 8). The game is controlled by the user through a control panel which is placed besides the falling blocks. The user then moves his or her hand around and presses on the buttons that will control whether the falling block is displaced to the left or right, sped towards the bottom, or rotated around. For the implementation of the game control, we built a game control engine. The game control functions described above are triggered by detecting the collision between the buttons in the control panel and the hand model (see Fig. 5).

Fig. 4.
figure 4

Dial panel.

Full size image
Fig. 5.
figure 5

Tetris control panel.

Full size image

Comparison Between Head-Mounted and Desk-Mounted Leap Motion Placement. Both the implementation of the dial panel and the Tetris game are compared under head-mounted and desk-placed conditions. The leap motion is horizontally placed on the desk in the desk-placed condition, whereas it is mounted on to the top of the VR headset facing outside away from the user’s head in the head-mounted situation (see Fig. 6). The complete setup with the four experimental conditions is shown in the accompanying video available at our website.

Fig. 6.
figure 6

Leap motion positon comparison.

Full size image

3 Results and Evaluation

When testing the implementation of the dial panel we obtained a stable result where users can predict the placement of the hand using the shadow effect and easily choose the button using the output information provided by the visual feedback (see Fig. 7).

Fig. 7.
figure 7

Screen capture of dial panel simulation.

Full size image

As for the implementation of the Tetris game, the control panel can be difficult to operate during play. As a result of various evaluation sessions, we figured out several modifications on the gaming control panel regarding the size, transparency, shadow effects and the visual feedback on collision, with the goal of improving the user experience. Eventually, the game’s control panel became reachable and easier to control. However, the user’s attention needed to correctly operate the panel was still causing some distraction when trying to control the blocks. Overall, the control panel did not feel as comfortable as using a regular keyboard, from the point of view of the gaming experience (see Fig. 8). This is in line with Barret and Krueger [4], who found that typing performance suffers when applying keystrokes on planar surfaces when compared to keystrokes on actual physical keys that displace upon typing. We hypothesize that the lack of haptic feedback on a planar virtual keyboard further reduces the user’s performance.

Fig. 8.
figure 8

Screen capture of tetris game.

Full size image

With regards to the comparisons of the head-mounted versus desk-mounted conditions for the location of the Leap Motion controller, the head-mounted version more closely resembles the original setup of the Oculus Rift combined with the Leap Motion sensor, as used in most head-mounted setups [5, 14]. This eliminated the dependency between the user’s hand and the placement of the sensor on the table or desk. However, we noticed that the Leap Motion controller did not behave as effectively as when placed on the desk and so the hand tracking was not as stable as in the desk-placed configuration. We hypothesize this is mainly because the user’s hand orientation when pointing out was somewhat parallel to the line of sight of the Leap Motion controller, making it especially challenging for the internal Leap Motion algorithms to estimate the correct hand pose.

We found that the lack of haptic feedback in the prototype developed caused some sensory confusion. It was hard for users to figure out the depth of the panels containing the buttons to be clicked. Through the whole implementation of this research, we found out that achieving stable tracking and making sure the control panel was reachable and usable were the most challenging aspects of this work.

3.1 Motion Sensor Limitations

It has been reported that the Leap Motion controller does not perform in a way that can be reliably used as a professional tracking system [12] and that its performance as an input device for everyday generic computer pointing tasks is rather limited when compared with standard input devices [3]. In addition, we have found that the current version of the Leap Motion controller and the Leap Motion SDK provides at times unsatisfactory tracking results, in part due to the following technical limitations:

  • The LeapMotion SDK doesn’t produce an RGBD depth map, so it limits the general tracking of objects in 3D and the general usability of the controller when compared to other depth-sensing controllers that provide an RGBD image, such as Intel’s RealSense [17] and Occipital’s Structure sensor [21].

  • Other items within the neighborhood of the hand (such as a piece of paper behind the hand) can cause the system to loose track of the hand altogether.

  • The SDK’s ability to correctly estimate the hand pose seem to vary, depending on where the controller is placed (desk- vs head-mounted). In particular, the current version of the Leap Motion SDK produces reduced tracking quality in the head-mounted configuration, especially when the user is making a fist, with hand poses where the fingers are pointing towards the depth-direction and parallel to the device’s line of sight, and when the user has a single finger extended with the rest as a fist.

4 Insights on Interaction Improvement

After the implementation of the applications mentioned above, we came up with several suggestions for improving the interaction:

  • Providing visual feedback by means of projecting a shadow of the hand model on the control panels made a positive contribution for users trying to assess how far to move their hand to press on the buttons of the panels. This was even more relevant the first time users were interacting with the system, as it was the time when they were forming a mental map of the relationship between their own hand movements and the representation of the hand model shown within the VR environment.

  • A more natural choice for an application like the Tetris game would be the swipe gesture, where the blocks are displaced to the left or to the right by a physical waving of the hand within the VR scene.

  • The lack of haptic feedback was disconcerting for most users, since it was easy for the hand controller to pass through the button and past the back of the virtual panel, making it hard to know when the button was clicked. For this reason, an additional controller, such as a joystick, or the hand-held button clicker device provided by the Freefly VR, might be better able to indicate the button click operation rather than using a pressing motion of the fingers to indicate a button click.

  • Providing auditory feedback by having the users hear a sound when a button is clicked would help users to become certain that the button click operation was successfully registered.

  • Current technologies can already bring users to experience an immersive VR experience at a low cost, but more reliable gesture controllers and better depth-sensing SDK’s are still needed.

5 Conclusions

The built-in Leap Motion hand detection system presents some limitations in estimating the pose of a user’s hand in the real world, but it still constitutes a workable device for rapid prototyping of real world gaming environment conditions, particularly when the controller is placed on the desk. At times, the lack of stability of the controller prevented the users from concentrating on playing the game itself. The low-cost smartphone used as a VR display, with the support from a simple and lightweight VR headset and a desk-placed depth sensor constitutes a convenient and affordable method for VR development for people who want to try the technology at early stages of development. Any game or similarly suitable type of application can be directly mirrored from the laptop to the proposed configuration, allowing for fast development of prototypes for research and evaluation of new ideas. We plan to expand this work with a user study where we compare the performance outcomes of people using the Leap Motion in the desk-mounted setup versus the head-mounted configuration across a variety of tasks within the VR environment.