Evaluation of visual feedback techniques for virtual grasping with bare hands using Leap Motion and Oculus Rift

Abstract

Bare hand interaction (BHI) allows users to use their hands and fingers to interact with digital content without any attached devices or accessories. For BHI to realize widespread adoption, interaction techniques for fundamental operations, like grasp-and-release, need to be identified and optimized. This paper presents a controlled usability evaluation of four common visual feedback techniques in grasp-and-release tasks using bare hand interaction (BHI). The techniques are ‘object coloring,’ ‘connecting line,’ ‘shadow’ and ‘object halo.’ The usability was examined in terms of task time, accuracy, errors and user satisfaction. A software test bed was developed for two interface configurations: using the Leap Motion controller alone (desktop configuration) and using the Leap with Oculus Rift (virtual reality (VR) configuration). Participants (n 32) performed four trials × five feedback techniques × two UI (user interface) configurations, i.e., a total of 1280 trials. The results can be summarized into: (a) user performance is significantly better in the VR configuration compared to the desktop; (b) coloring techniques for visual feedback (‘object coloring’ and ‘object halo’) are more usable than ‘connecting line’ regardless of UI; (c) in the VR, coloring techniques remain more usable, while in the desktop interface the ‘shadow’ technique is also usable and preferred by users, (d) the ‘connecting line’ technique often distracts users from grasp-and-release tasks on static targets. (e) Some visual feedback is always preferred by users than none in both VR and desktop. We discuss these findings in terms of design recommendations for bare hands interactions that involve grasp-and-release tasks.

1 Introduction

Over the last few years, the release of affordable hand tracking sensors like Leap Motion controller and Myo armband has made it feasible to interact with the computer with bare hands. Bare hand interaction (BHI) (Von Hardenberg and Bérard 2001) allows users to use their hands and fingers, with no other devices or accessories, in order to interact with digital content typically with the exercise of particular gestures or postures. BHI focuses on the use of the hand, palm and fingers, and it is therefore different from the styles of whole-body interaction (England 2011), which refers to the use of body movements and postures, or midair interaction (e.g., Koutsabasis and Domouzis 2016), which emphasizes the use of (whole) hand gestures in the midair to interact with distant displays.

At the same time, we are witnessing the release of a new generation of affordable head-mounted displays (HMDs) for immersive VR, based on PCs or mobile devices, such as Oculus Rift, Google Cardboard, HTC Vive. It is anticipated that the engaging experiences produced by these devices will open up a market for several industries in fields such as entertainment, rehabilitation, teleconferencing, online shopping and social networking (Parkin 2016). Some of these displays can be effectively combined with hand tracking sensors to allow for BHI with the virtual content while immersed in the 3D environment.

The accessory-free nature of emerging hand tracking sensors has significant differences from existing VR technology that required users to wear data gloves, possibly with haptic or force feedback. In BHI users can exercise an unconstrained set of gestures (provided the hand remains in the field of view of the sensor), as well as switch between bare hand manipulations and the mouse. They can also use both hands to interact with the environment instead of one, which was the common helmet and glove VR configuration. However, no force or haptic feedback is available for bare hands. These differences from traditional VR technology make the design of effective interaction techniques for bare hands unclear (Nabiyouni et al. 2014), indicating a need for further exploration of possible interaction and feedback techniques.

Controlled usability evaluation studies of BHI can inform the identification and optimization of effective interaction techniques for the most basic operations, which are essential for its widespread adoption. Recently, a number of such studies have been conducted to inform the design of BHI applications. These follow a test bed approach (Bowman et al. 2001) to evaluate alternative interaction techniques for a particular task such as object selection or pointing (Seixas et al. 2015; Coelho and Verbeek 2014), manipulation (Song et al. 2012) and travel (Codd-Downey and Stuerzlinger 2014).

Until now, there are no studies of BHI visual feedback techniques for grasping tasks, although (a) there are already several applications in the Leap Motion app store (10 out of 97 free apps require holding a virtual object) as well as in the scientific literature (e.g., Vosinakis et al. 2016; Jayakumar et al. 2015) and (b) the issue has been investigated in glove-based VR configurations (e.g., Prachyabrued and Borst 2016). The problem with interactions such as selecting or grasping is that the absence of haptic feedback makes it difficult for users to understand whether their hand is in an appropriate position to interact with a virtual object. This issue becomes more intense if target objects are placed at different depths with respect to the users’ view, and the rendered scene does not include adequate depth cues. In such cases, it is possible that some form of additional visual feedback might improve the task by providing further cues.

This paper investigates the usability of four feedback techniques for virtual grasping with bare hands in two interface configurations; the techniques are: (1) object coloring, (2) connecting line, (3) shadows and (4) object halo; the configurations are: (a) the Leap Motion controller alone (desktop configuration) and (b) a combination of Leap Motion and Oculus Rift in a VR (virtual reality) configuration. We compare the overall usability of techniques between and within each configuration in terms of task time, errors, accuracy and user preference with a controlled usability test (n = 32). The results concern both usability of each interface configurations as well as each particular visual feedback technique and contribute to an improved understanding of bare hands interaction design issues.

2 Background and related work

2.1 Overview of the Leap Motion and Oculus Rift

The Leap Motion sensor can track the position, shape and motion of bare hands. As such, it can be used for natural interactions with a 3D environment within a restricted area. The device is using three infrared emitters and two infrared cameras to track the image of the hand, and it extracts information such as palm and fingers position and orientation. The extracted data are transmitted at high frequencies to the attached computer and, using the SDK provided, it can be used in real-time applications. The hand information can be utilized for recognizing gestures (e.g., controlling an application using designated gestures), providing continuous input (e.g., navigation in 3D by pointing to the desired direction), or introducing a respective ‘virtual hand’ in the application and letting it interact directly with the contents (e.g., pushing or grasping virtual objects). Given its low cost and its fast and accurate hand tracking, the Leap Motion controller has been used in a variety of applications, including cultural heritage (Vosinakis et al. 2016), education (Lin et al. 2015), rehabilitation (Khademi et al. 2014) and entertainment (a total of 228 apps are available in the Leap Motion app store¹).

Several studies have attempted to measure the quality of tracking of Leap Motion with generally positive results. In a study by Martin et al. (2014), it has shown an accuracy of over 80% in the recognition of hand gestures of the American Sign Language. In another study (Coelho and Verbeek 2014) that compared Leap Motion to a mouse for pointing tasks in 3D environments, Leap Motion had significantly better performance in a single pointing task. However, in a second task which involved successive pointing of two targets with different depth, the mouse outperformed Leap Motion. This inconsistency has been attributed to the additional degree of freedom and the lack of accuracy in finger tracking. In another comparative study between Leap Motion and the mouse (Apostolellis et al. 2014), the two devices have been used in a complex task of positioning, rotating and changing the intensity of a spotlight in a 3D scene. Leap was found to be more effective than the mouse for positioning and setting the desired intensity, and less effective for positioning and orienting the spotlight. Two problems of the Leap controller that have been reported are inaccurate finger tracking and user fatigue caused by continuous hand movements. Further studies on the tracking accuracy of Leap Motion (Weichert et al. 2013; Guna et al. 2014) revealed that it is highly accurate in static scenarios (less than a mm), while in dynamic scenarios its accuracy drops, especially when the hand’s distance from the controller increases. In another study (Bachmann et al. 2014), the error rate of Leap Motion controller as a pointing device was found to be three times higher than the standard mouse. However, comparable error rates have been reported with target widths of 40–20 mm and target distances up to 80 mm.

Oculus Rift is an affordable head-mounted display (HMD), which can be combined with Leap Motion to enhance the immersive experience with bare hand tracking. It needs to be connected to a high-end PC and offers high-quality stereoscopic rendering of 3D scenes. Furthermore, it allows for real-time orientation tracking using a variety of sensors (gyroscope, accelerometer and magnetometer) and provides positional tracking within a limited area through a dedicated IR camera. Recently the makers of Leap Motion started to offer special support for the combination of their controller with Oculus Rift. They provide a mount specifically designed to attach the controller to the front face of Oculus Rift, and they also extended their SDK to support improved hand tracking while attached to an HMD. With this configuration, the virtual hands of the user are introduced in the scene whenever her real hands are in the field of view of the leap controller, i.e., when the user’s head is oriented toward them, thus leading to more natural and immersive experiences. The integration of the two devices has been used for the implementation of applications that offer affordable VR experiences (Beattie et al. 2015).

2.2 Related studies and BHI applications

Leap Motion can theoretically support many bare hand interaction techniques, like those already implemented for data gloves in traditional, high-end VR systems. However, the absence of any haptics makes it difficult for users to get feedback on their actions, e.g., to know whether their virtual hand is touching an object or not. Especially in the desktop configuration, where the scene is possibly displayed at a smaller scale, it is difficult for the user to get a good sense of depth and to understand the relative location of her virtual hand with respect to other objects of the scene (Teather and Stuerzlinger 2007). Therefore, an additional form of feedback may improve the efficiency of user interactions. The most common feedback is the visual one, in which additional visual information is presented in some abstract or concrete form to guide the user in her actions. Common visual feedback techniques include changes in object coloring (Poupyrev et al. 1998), use of shadows to display relative distance (Hu et al. 2000), illumination of nearby collision areas (Sreng et al. 2006) and vibration of the virtual hand when touching an object (Prachyabrued and Borst 2016).

To identify current practices of visual feedback in virtual grasping tasks, we undertook an examination of free Leap Motion apps (97 apps are free from 229 in total). We found ten (10) apps that involve holding a virtual object, which we examined in more detail. From these ten apps, four apps make use of a tracked virtual hand representation, two apps make use of tracked hand points, and the remaining four use a pointer. Obviously, the hand representation is critical for the visual feedback technique that may be applied for a virtual grasping task. Regarding the main grasping technique, four out of five apps use the pinch gesture and only one adopts the hand grasp. Regarding visual feedback techniques, we found that four apps provide some feedback of virtual grasping, while the remaining six apps do not provide any feedback at all. The ‘robot chess app’² makes use of visual arrows or targets to help the user understand depth as well as the chessboard square under the virtual hand at any point of interaction. This app also supports coloring of chess pieces as the virtual hand approaches the piece. The ‘Autonomous’ game³ uses coloring when the virtual hand approaches a target object. The ‘Cyber science motion 3D’⁴ uses tooltips when the user hand is near the virtual object and when the user grasps the object it changes color. The ‘Playground’⁵ demo adopts the ‘connecting line’ to position an object onto a moving target.

A recent study by Prachyabrued and Borst (2016) examined various interpenetration cues for virtual object grasping with the use of a data glove and a VR head set. The techniques included changes in the color or transparency of the virtual hand or object, as well as a two-hand visualization, which showed both the actual shape of the user’s hand and an ‘ideal’ shape of the virtual hand grasping the object. The results indicate that the two-hand technique improves performance, although users’ subjective preference was on two coloring techniques used, i.e., coloring the target object or the virtual hand’s fingers. Prachyabrued and Borst (2016) identify the need for more focused studies of specific visual cues (our work addresses this dimension for BHI) and for choosing conditions in comparisons to other feedback modes, such as haptic, audio or multimodal.

Another recent study (Caggianese et al. 2016) investigated canonical manipulation of virtual objects which includes the selection, positioning and rotation of objects with BHI with the use of Leap Motion and the Oculus Rift. Two different approaches were tested that allowed a direct or constrained manipulation of the virtual object and the evaluation investigated the perceived usability with a standardized usability questionnaire only. Until now, there are no studies that examine both performance and satisfaction of alternative visual feedback techniques for virtual grasping with BHI in a desktop or immersive environment.

3 Techniques and test bed

3.1 Feedback techniques

Based on the commonly used techniques for grasping and releasing virtual objects in related research, we have selected and adapted four visual feedback techniques for the test bed environment: ‘object coloring,’ ‘connecting line,’ ‘object halo,’ and ‘shadow.’ The absence of any feedback (‘no feedback’) has been added as a fifth technique in the test bed. Figure 1 shows screenshots from the implementation of these techniques in the test bed environment.

Fig. 1

Screenshots of the test bed environment; from upper left to lower right: familiarization phase; no feedback; object coloring; connecting line; object halo; shadow

No feedback The environment provides no additional visual indication of the distance between the virtual hand and the target. In some VR applications, the stereoscopic vision and the natural cues provided by the environment (e.g., occlusion, shadows, relative size) may be adequate for proper hand placement without any additional feedback. However, in desktop BHI environments the lack of stereo vision might impede depth understanding, and some kind of feedback is necessary.

Object coloring The main idea of the coloring technique is to change the color of an object, when the user’s hand approaches it. A gradual change in coloring based on distance can provide an additional cue on the hand’s proximity to the object. This technique may be designed with variations, for example, to change the color of the user hand, or of the fingertips. According to the results of the study of Prachyabrued and Borst (2016), users prefer the coloring technique for grasping tasks (with data glove, VR configuration only).

Connecting line The main idea of the connecting line technique is to display a line that connects the user hand with an object that can be grasped. Two variations of this technique are to draw the distance to target dynamically onto the line, or to change the line color based on hand’s proximity to the target object. From our experience, the connecting line technique seems intuitive mainly at release tasks, especially when the release is to be made onto/inside a moving target.

Object halo The main idea of the halo technique is similar to coloring: When the target object is gradually approached by the user hand, a glowing halo appears around the object with its size increasing as the hand moves closer. The halo technique is commonly used in games and it does not change the look of the object itself (in contrast to the coloring technique described before). As such, it might be more appropriate in cases where it is significant that the objects’ appearance or color must not change at all.

Shadow The shadows of the virtual hand and object are visible on a flat horizontal surface in the environment to provide an extra cue of the distance between them. The shadow technique is more natural and intuitive, as it neither adds any additional abstract representations in the scene, nor distorts the realism of the environment. However, the scene arrangement and lighting should be carefully prepared, so that shadows are always visible and easily identifiable, to serve as an effective depth cue.

3.2 The test bed environment

The test bed application (Fig. 1) is a simple 3D world with geometric primitives, in which the user is asked to complete a series of grasp-and-release tasks using her virtual hands. The application can work in two configurations: the desktop configuration that requires the Leap Controller alone, placed on a flat surface in front of the user, and the VR configuration, in which users wear the Oculus Rift HMD and Leap Motion is mounted on it (Fig. 2).

Fig. 2

User interactions with the desktop (left) and VR (right) interface configurations

The test bed environment has been developed in Unity 3D using the Orion SDK provided by Leap Motion and has been compiled in two variations, one for each configuration. The only difference between the two executables was in the support of Oculus.

The test bed environment displays the virtual hands of the user, a sphere, a cube and a ground plane. The aim of the user is to grasp the sphere with his right hand and to place it inside the cube. Physics are disabled, so there is neither gravity nor collisions in the environment. The sphere diameter and the edge of the cube had a length of 7 cm.

The visual feedback techniques have been implemented as follows (Fig. 1):

• Object coloring: For the grasp phase, when the virtual hand approaches (reaches a threshold distance) the sphere, the color of the sphere changes instantly to red. If the hand continues to approach, the sphere color linearly changes from to red to green. When the color is green, the object can be grasped. For the release phase, the same technique is applied between the virtual sphere and the cube.

• Connecting line: For the grasp phase, when the target object is approached, a line connects the center of the palm of the virtual hand with the sphere. The distance in cm between the target object and the hand is displayed in text. The target object can be grasped at a distance lower than the radius of the sphere, i.e., 3.5 cm. For the release phase, the same technique is applied between the virtual sphere and the cube.

• Object halo: For the grasp phase, when the target object is approached (reaches a threshold distance), its outline is glowing. The halo is progressively enlarged and brightened as the user approaches the object. If the user hand passes through the sphere without grasping, the halo becomes smaller and less intense. For the release phase, the same technique is applied between the virtual sphere and the cube.

• Shadow: vertical shadows of the virtual hand, the sphere and the cube are displayed on the ground to provide a relative sense of distance.

The test bed environment also includes an initial familiarization phase, during which the user can freely interact with physical primitives using her bare hands to familiarize herself with the interface (Fig. 1, upper left).

Finally, a logging mechanism has been implemented to keep track of task-related events, errors and timestamps during the user tasks.

4 The experiment

4.1 Research questions and hypotheses

The experiment addresses the following research questions and hypotheses:

Q1:

Is there a difference in usability of grasp-and-release tasks with bare hands between the UI configurations (VR, desktop)?

Q2:

Is there a difference in usability between the feedback techniques (none, coloring, line, halo and shadow)?

Q3:

Is there a difference in usability within each condition of UI configuration × feedback technique (VR × feedback, desktop × feedback)?

Based on previous work and our experience, we have the following hypotheses with respect to Q1, Q2:

Hypothesis 1 (H1)

The VR configuration is more usable than desktop configuration.

Hypothesis 2 (H2)

The ‘no feedback’ would be less usable than any other feedback technique within each configuration.

Regarding Q3, since there is no previous evidence about the comparative usability of visual feedback techniques for BHI grasp-and-release tasks, we did not make any a priori assumptions about which might be more usable than another.

According to ISO 9241, usability is defined as ‘the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.’ Put simply, effectiveness refers to whether users can achieve their goals and perform tasks successfully with the system, efficiency refers to performance issues (e.g., time, without errors) and satisfaction refers to the subjective opinion of users about (perceived) usability.

4.2 Measures

There are many measures or metrics about usability (for a review see Albert and Tullis 2013) and the most common in controlled usability evaluations are task time, errors, accuracy and user satisfaction. These four measures are employed in this study.

The first three measures were automatically collected by the test bed application. The following event types were recorded by the test bed app: grasp, failed grasp, release and failed release. Each of the recorded events had an associated time from the beginning of the experiment. The release event was triggered when the user placed the sphere inside the cube at a distance between their centers less or equal than the radius of the sphere (3.5 cm). This would be approximately 50% or more overlap between the two primitives.

Failed grasp and failed release were treated as errors, while the release event marked the success of a trial.

At the successful release, the accuracy of placement (a) was calculated as follows:

$$a = \frac{R - d}{R},$$

where R is the radius of the sphere and d is the Euclidean distance between the sphere and the cube.

The fourth measure was user satisfaction. To obtain participant opinions, we asked them to fill in an online questionnaire after they finished with each user interface condition. The questionnaire consisted of four statements: (a) the technique was comfortable; (b) the technique was precise; (c) the technique was intuitive; (d) I liked the technique. Users provided their answers in a 5-point Likert scale (1 = strongly disagree; … 5 = strongly agree).

4.3 Experimental design and data analysis

The experiment was with a [2 × 5] within-subjects design. The first factor was the UI (user interface configuration) with 2 levels: VR and desktop. The second factor was the feedback technique with five levels: none, coloring, line, halo, shadow.

We analyzed the data by comparing means. The data analysis was performed in SPSS. All data were analyzed using 95% confidence intervals to provide the range that estimates the true population value for each measure. Confidence intervals are extremely valuable for any usability test (Albert and Tullis 2013) and according to Sauro (2012) ‘if the intervals do not overlap then you can be at least 95% confident there is a difference… the intervals can actually overlap by as much as 25% and still be statistically significant.’

Furthermore, we conducted a repeated measures factorial analysis of variance (ANOVA) to identify statistically significant differences for the three measures of task time, accuracy and errors. When such differences were observed, we conducted pairwise comparisons with Bonferroni corrections. A full factorial analysis provides the effect of each factor (UI, feedback) on the independent variables (usability measures: task time, accuracy, errors), as well as the effects of interactions between factors on the variables.

User satisfaction was collected with Likert scale data that is not normally distributed. Since that there is not a nonparametric equivalent for full factorial ANOVA, we conducted a Friedman’s test which is the nonparametric test for two-way ANOVA; however, this test does not include interactions (there is not a nonparametric equivalent of the full factorial ANOVA). We also conducted post hoc analysis using Wilcoxon signed-rank test for pairwise comparisons with Bonferroni corrections among conditions of UI × feedback.

4.4 Participants, tasks and procedure

Thirty-two (32) users participated in the evaluation on a voluntary basis: twenty (20) men; aged 19–44 years (average 25.1). Twenty-five (25) participants were students and seven academic research staff. Six participants had limited experience with the Leap Motion and three with the Oculus Rift. We excluded six participants (from originally recruited 38). Three of them have been excluded because they were left-handed and, given that the system has been implemented for the right hand only, they would not use their dominant hand in the tasks. The other three had low vision and reported discomfort when wearing the Oculus Rift headset.

During the experiment, participants had to complete four trials of grasping and releasing in each feedback technique, i.e., 20 trials for each UI configuration, 40 in total for each user, 1280 for all participants. For each trial, the user had to grasp a sphere with the right hand, move it toward a transparent cube and release it inside it. Each trial was therefore inherently split in:

• The grasp phase: the user had to successfully grasp the sphere by placing her virtual hand on it performing a grasp gesture. The user had to use all fingers to grasp the sphere and not just pinch it. For the software to identify the grasp gesture, some virtual hand penetration to the sphere had to be tracked.

• The release phase: the user had to bring the sphere to the semitransparent cube using her virtual hand and release it as close to its center as possible. While the user was moving the sphere, her hand had to remain in a grasping gesture (with some tolerance). If the user opened the palm, the sphere was released. If the released sphere was not placed inside the cube (over 50% degree of penetration of the sphere into the cube), the user would have to grasp the sphere again (from the point the sphere was dropped) and then retry the release phase.

We have treated each set of four trials as a single task, i.e., ‘successively place the sphere into the cube by using the A/B/C/D/E feedback technique.’ After the end of each trial, the user had to press the space bar to see the next placement of the sphere and the cube (i.e., users could take their time to move to the next trial). During all trials with a particular feedback technique, the participant would see its name on the top of the screen. After the user finished with the task, she moved to the next feedback technique. For each one of the four trials, the initial placement of the sphere and cube were different; these were in counterbalanced order for each participant to minimize the learning effect. The feedback techniques were also presented in counterbalanced order.

The procedure of the experiment was as follows:

• 1. Welcoming. Users were introduced to the purpose of the test.

• 2. First user interface condition (desktop or VR, counterbalanced). As they came along, users were appointed to one interface condition.

• 3. Familiarization. Users used their hands to interact with 3D objects (primitives) in order to get familiar with each configuration (Fig. 1, upper left).

• 4. Task performance. For each UI, for each feedback technique in counterbalanced order, users performed grasp-and-release tasks.

• 5. Questionnaire. Users filled in the questionnaire.

• 6. Steps 2–5 were repeated for the second user interface condition.

• 7. End. Wrap-up.

4.5 Apparatus

In the desktop configuration, the environment was running on a laptop with i7 2.50 GHz CPU, 16 GB Ram, nVidia GTX 850 M graphics card and 15.6″ display. The Leap Motion controller was positioned in front of the user at the center of the desk.

For the VR configuration, we used a laptop with i7 2.60 GHz CPU, 16 GB Ram and an nVidia GTX 970 M graphics card (this card is the minimum required for Oculus). The Oculus Rift DK2 was attached to the laptop, and the Leap Motion controller has been mounted on the front face of the HMD using the VR developer mount provided by Leap Motion. We have used a slightly higher CPU for the VR configuration to compensate for the extra computing requirements of the Oculus Rift. Prior tests ensured that there was no difference in observable user performance (both setups had mean refresh rate above 70 FPS).

In both configurations, users were seated during the test.

5 Results

5.1 Between user interfaces

5.1.1 Task time

UI has a significant effect on task time [F _4.17 = 81.143, p = 0.000 < 0.05, Table 1 (row 1)], i.e., users who perform grasp-and-release tasks with the VR configuration are faster than with the desktop. This is clear for any feedback technique applied (Fig. 3).

Table 1 Two-way repeated measures ANOVA test of within-subjects effects

Fig. 3

Mean task time (sec), with 95% confidence intervals

5.1.2 Accuracy

UI has significant effect on accuracy (F _4.17 = 30.562, p = 0.000 < 0.05, Table 2, row 1), i.e., users who perform grasp-and-release tasks with the VR configuration are more accurate than with the desktop. This is clear for any feedback technique (Fig. 4).

Table 2 Two-way repeated measures ANOVA tests of within-subjects effects

Fig. 4

Mean accuracy, with 95% confidence intervals

5.1.3 Errors

UI has significant effect on errors (F _4.17 = 10.147, p = 0.003 < 0.05, Table 3, row 1), i.e., users who perform grasp-and-release tasks with the VR configuration make fewer errors than with the desktop. This is clear for any feedback technique (Fig. 5).

Table 3 Two-way repeated measures ANOVA tests of within-subjects effects

Fig. 5

Mean errors, with 95% confidence intervals

5.1.4 User satisfaction

UI has significant effect on user satisfaction. A nonparametric Friedman test of differences among repeated measures was conducted and rendered a Chi-square value of 27.545 which was significant (Table 4), i.e., users who perform grasp-and-release tasks with the VR configuration are more satisfied than with the desktop, for any feedback technique (Fig. 6).

Table 4 Nonparametric Friedman test statistics

Fig. 6

User preference, with 95% confidence intervals

These results are confirmatory to the first hypothesis (H1) of the experiment, i.e., that VR configuration would be more usable than the desktop. This applies for all four measures of usability examined.

5.2 Among visual feedback techniques

5.2.1 Task time

Feedback (technique) has significant effect on task time [F _2.69 = 4.65, p = 0.002 < 0.05, Table 1 (row 2)], i.e., users who perform grasp-and-release tasks with at least one of the feedback techniques examined are faster to those who make use of at least one other technique.

Pairwise comparisons (Table 5) reveal statistically significant differences in means between:

• Coloring and line (p = 0.02 < 0.05; Table 5, lines 6, 10), i.e., users who perform grasp-and-release tasks with Coloring are faster than with Line.

• Halo and line (p = 0.016 < 0.05; Table 5, lines 11, 15), i.e., users who perform grasp-and-release tasks with Halo are faster than with Line.

Table 5 Two-way repeated measures ANOVA pairwise comparisons between conditions for feedback

5.2.2 Accuracy and errors

No other significant effects were found, i.e., there is not any feedback technique that has a significant effect on accuracy and errors.

These results do not confirm our second hypothesis (H2), i.e., that ‘no feedback’ would be less usable than any other feedback technique within each configuration.

5.3 Among conditions of UI × feedback

The interaction of UI × feedback has significant effect on task time [F _2.69 = 5.223, p = 0.001 < 0.05, Table 1 (row 3)]. To further identify these effects, we have performed pairwise comparisons for VR and desktop interfaces.

5.3.1 Task time

Pairwise comparisons of UI × feedback (Table 6) reveal some significant effect between:

• VR × None and VR × Coloring (p = 0.019 < 0.05, Table 6, rows 1, 5), i.e., users who perform grasp-and-release manipulations in the VR with coloring are faster than with no feedback.

• Desktop × line and desktop × halo (p = 0.016 < 0.05, Table 6, rows 31, 35), i.e., users who perform grasp-and-release manipulations in the desktop with halo visual feedback are faster than with line feedback.

• Desktop × line and desktop × shadow (p = 0.02 < 0.05, Table 6, rows 32, 39), i.e., users who perform grasp-and-release manipulations in the desktop with shadow visual feedback are faster than with line feedback.

Table 6 Two-way repeated measures ANOVA pairwise comparisons for interactions UI × feedback

5.3.2 Accuracy

No significant effects were found, i.e., there is not one condition of UI × feedback technique that has a significant effect on accuracy.

5.3.3 Errors

The interaction between UI × feedback significantly affects errors. Respective pairwise comparisons (Table 7) show a statistically significant difference in mean number of errors between:

• VR × no feedback and VR × coloring (p = 0.028 < 0.05, Table 7, rows 1, 5), i.e., users who perform grasp-and-release manipulations in the VR with coloring make fewer errors than with no feedback.

• VR × coloring and VR × halo (p = 0.008 < 0.05, Table 7, rows 8, 14), i.e., users who perform grasp-and-release manipulations in the VR with coloring make fewer errors than with halo feedback.

Table 7 Pairwise comparisons for interactions UI × feedback

5.3.4 User satisfaction

Pairwise comparisons of UI × feedback (Table 8) reveal some significant differences in user satisfaction between:

• Desktop × none and desktop × halo (p = 0.008 < 0.01, Table 8, row 3), i.e., users who perform grasp-and-release manipulations in the desktop with the halo feedback technique are more satisfied than with no feedback.

• Desktop × none and desktop × shadow (p = 0.006 < 0.01, Table 8, row 4), i.e., users who perform grasp-and-release manipulations in the desktop with the shadow feedback technique are more satisfied than with no feedback (Fig. 6).

• Desktop × halo and desktop × coloring (p = 0.003 < 0.01, Table 8, row 6), i.e., users who perform grasp-and-release manipulations in the desktop with the halo feedback technique are more satisfied than with the coloring technique.

Table 8 Pairwise comparisons for interactions UI × feedback (desktop)

Since that we had made no prior assumptions or hypotheses about conditions of UI × feedback, these results can inform the design of BHI in each particular condition.

6 Discussion

We can identify several design issues and recommendations based on the research questions and hypotheses, and we discuss them in terms of previous work and experience.

The first research question (Q1) of the experiment was if there is a difference in usability of grasp-and-release tasks with bare hands between the UI configurations (VR × desktop). We had assumed that there would be a significant difference (H1). We found that the VR interface outperforms the desktop in all aspects of usability regarding grasp-and-release tasks for any feedback technique. There are significant effects in all measures: task time, errors, accuracy and user satisfaction (Sect. 6.1). Therefore, applications that require grasp-and-release operations are more usable in the VR than in a desktop interface. This is true for any feedback technique, even when the desktop × ‘any feedback’ condition is compared to VR × none for almost all measures (but for a couple of comparisons about user satisfaction, see Figs. 3–6). Apparently, the affordances of VR like the stereoscopic view and increased presence contribute to this result.

This result is in alignment with Renner et al. (2013) who denote that ‘for shorter distances, missing binocular disparity impairs distance perception.’ In addition, we have seen in previous work (on qualitative think aloud studies of bare hand interactions in gamified applications of virtual sculpting, Vosinakis et al. 2016) that the desktop interface for BHI yields many usability issues, not only about visual (and audio) feedback but also about look and feel, hand tracking, gesturing, user orientation, visibility and control/reach. Therefore, there are various issues that impede the user experience in desktop configurations of BHI, and visual feedback techniques alone do not suffice for improving user performance.

The second research question (Q2) of the experiment was if there is a difference in usability between the feedback techniques (no feedback, object coloring, connecting line, object halo and shadow) regardless of UI condition. Based on our experience, we assumed that the situation of ‘no feedback’ would be less usable than any other feedback technique within each configuration (H2). We have found little evidence to support this hypothesis. Some feedback techniques are more usable than others regarding particular measures only, i.e., (a) the ‘object halo’ technique is preferred by users in comparison with ‘no feedback’ (user satisfaction) and (b) the ‘object coloring’ and ‘object halo’ techniques are faster (task time) than the ‘connecting line’ technique.

If we consider that the ‘object coloring’ and ‘object halo’ techniques are similar to the extent that they artificially augment the appearance of virtual objects with a strong visual indication of proximity, we can reach to a recommendation that both these proximity-based coloring techniques present particular advantages for usability. However, since that this recommendation is not fully backed up by this experiment, it is up to the designer to consider this in relation to other design issues related to application and context. Coloring techniques of visual feedback were also preferred by users in the study of Prachyabrued and Borst (2016), which examined interpenetration cues in grasping and release tasks with a data glove in VR.

We also found that the ‘connecting line’ technique seems to impede user performance (task time) and distract users. For most measures, this technique was worse in usability, even when compared to ‘no feedback.’ This technique is certainly the most intrusive for the design of the grasp-and-release interactions. Nevertheless, it is intuitive for moving targets (like for example in the game Playground of the Leap Motion app store).

The third research question (Q3) of the experiment was if there is difference in usability within each condition of UI configuration × feedback technique (VR × feedback, desktop × feedback). We did not make any a priori assumptions about which technique might be more usable than another. We have found few conditions (UI × technique) that are more usable than others for some measures (reported in Sect. 6.3).

For the VR interface, user performance is significantly improved with the ‘object coloring’ technique in comparison with the ‘no feedback’ in task time and errors. In the VR, ‘object coloring’ is also significantly different than ‘object halo’ in terms of user errors. These results are again in favor of the ‘object coloring’ technique for visual feedback particularly for VR interfaces. For the desktop interface, we have found that the ‘connecting line’ technique is significantly slower than ‘object halo’ and ‘shadow.’ In addition, users prefer the ‘halo’ and ‘shadow’ techniques to ‘no feedback’ as well as the ‘object halo’ technique to ‘object coloring.’

Based on these results, we identify the following recommendations:

• 1. If possible, prefer the VR interface over the desktop for BHI of grasp-and-release.

• 2. Prefer proximity-based coloring techniques (object coloring and halo) for feedback. Depending on the application goals, use ‘object coloring’ for better performance and ‘object halo’ for better user satisfaction.

• 3. Alternatively, prefer ‘shadow’ technique, especially in the desktop interface, as it is more natural and usable compared to ‘connecting line’ and no feedback.

• 4. Avoid using the ‘connecting line’ technique, unless there are important contextual factors involved, like for example if the release is to be made onto moving targets or if increased release precision is required.

• 5. Prefer using any form of visual feedback in both VR and desktop interfaces to no feedback at all.

This study has a number of limitations. First of all, it examined four selected feedback techniques. Although these techniques are quite popular in VR and game implementations, one may easily find alternative configurations or combinations of them that were not included in the study. Then, the equipment used in the two UI configurations was different. As reported in Sect. 4.5 (Apparatus), we have used a slightly higher CPU for the VR configuration to compensate for the extra computing requirements of the Oculus Rift. Also, there was a gender imbalance of the participants in this study (20 men and 12 women). Furthermore, we have not included additional measures for the quality of performance, such as trajectory and velocity shape. Finally, given that the study is a test bed evaluation, it does not take into account any external conditions that might affect task performance in specific applications, such as size and appearance of objects, moving targets, lighting and scene complexity.

Finally, we report on some issues that require attention about the technical performance of the interface configurations. A problem that appeared a few times with the VR configuration was that the users’ hands were standing between the HMD and the IR camera for positional tracking, causing some short but unexpected distortions in the view. Also, we noticed some tracking interferences of the Leap Motion Controller for users who wore metallic jewelry on their hands, wrist watches as well as leather jackets. In general, Leap Motion produced quite often tracking errors or misinterpretations of hand gestures, which were responsible for a significant percentage of the identified user errors.

7 Summary and conclusions

This paper presented a controlled usability evaluation of four common visual feedback techniques in grasp-and-release tasks using bare hands: object coloring, connecting line, shadows and object halo, in the interface configurations of the Leap Motion controller alone (desktop configuration) and a combination of Leap Motion and Oculus Rift (VR configuration). The VR configuration is more usable than the desktop for grasp-and-release tasks with bare hands. Coloring feedback techniques (with a color or halo) are more usable than shadowing and no feedback, while the line technique often distracted users rather than contributed to usability.

For BHI to realize widespread adoption, there is a need for identification and optimization of effective interaction techniques for fundamental operations. This study was sharply focused on a particular aspect of BHI design, i.e., visual feedback techniques for grasp-and-release tasks. It contributes to the growing corpus of usability studies in BHI and makes respective recommendations about visual feedback techniques. Interaction designers and developers should consider these recommendations in relation to the context and requirements of particular applications. Future work includes investigations of BHI interaction techniques not only in experimental setups but also in field settings to contribute to a better understanding of design issues and options.