Keywords

1 Introduction

Effective object manipulation, one of the basic interaction tasks in any virtual space, is important for the fluent usability, and as such, many interaction techniques and tools have been suggested for it [1, 2, 7]. However, most of them can be categorized as the “magic” techniques. That is, the “tools” are not reality-inspired, but purposely “designed” to achieve the task as effectively as possible. It is only natural to take advantage of the virtuality to free oneself from the bounds of the physical reality. After all, tools may be useful in the physical world but not necessarily in the virtual.

On the other hand, virtual reality (VR) also aims to provide a difficult-to-get “experiences”, if not in its entirety but at least the core. Take an example of providing the fun experience of carpentering and assembling a wooden desk using an assortment of hand tools without having to gather all the materials and set up a shop. Surely, one component of such a VR experience would be to employ an interaction method that is based on reality, e.g. sensing the hand/fingers movement as to pick up and apply the needed tool.

Instead, in this paper, we introduce “HandyTool” a method (and an interface) for virtual object manipulation based on a metaphorical/structural mapping of various everyday tools to our hands and fingers. The basic idea is to transform the hand/fingers into a proper tool (e.g. a fist becoming a hammer head) and gesturally apply it to and manipulate the target object (e.g. inserting a nail) directly. The intended objective of HandyTool is to enhance the experience of usage of the tool by one (or one’s body part) becoming the tool itself and thereby also improving the task performance. While the mapping is already intuitive and easily understood, it can be guided using a visual interface overlaying the control skeleton over the target tool (see Fig. 1). Once the mapping is established, the user can gesturally enact (and not indirectly control) the tool using one’s hand and fingers as if the tool was the one’s hand (applying fisted hand as if it was the hammer head to insert a nail).

Fig. 1.
figure 1

The concept of HandyTool (or using both the structural metaphor and finger tracking, right-most) and other tool-object manipulation methods (from the left, RB, MB, RF).

Full size image

This paper is organized as follows. We first shortly review related research. Then we present the detailed design of HandyTool and the usability experiment carried out to assess the projected merits. We also show the results of applying the method to partially controlling an avatar or virtual puppets as an educational tool for young children to train their hand skills, called HandyMan. Finally, we summarize and discuss our findings and conclude the paper.

2 Related Work

Various 3D interaction techniques, including those for object selection and manipulation, have been developed over the years for use in the virtual space [1, 2, 7]. Among many, we review those that are hand-based (or equally hand gesture based). The most prevalent form of object selection and manipulation is the “Direct Hand” method [2, 11]. Usually a tracker/button device attached to the user hand tracks and maps the hand into the virtual space to select an object by simple touch (i.e. collision with the object). Once selected, the target object is attached to the hand, and follows the moves of the hand (translate/rotate) to be manipulated. The button device or simple gestures can be used to simplify more complicated moves (e.g. twisting motion) or make logical commands (e.g. change color). Interaction controllers are a popular commercial realization of the tracker/button device today [4, 10, 14].

Secondary tools, especially those that are reality inspired, are rarely used in VR. For instance, to insert a nail, the direct hand itself can be used to accomplish the task either by physical movement (and simulation), gesture or button command. To truly emulate the usage of a tool, the user would have to somehow select/grab the handle portion of the tool, move it to “control” the tool and apply it to the target object. In this regard, such an approach requires more exact tracking of finger movements. Gloves [12, 13] and more recent advanced sensors (e.g. Microsoft Kinect [9], Leap Motion [6] can be used for this purpose, but more so for just 1:1 mapped animation or making logical gestural commands [8].

3 HandyTool

Humans have used tools to make everyday tasks easier (at least in the physical world). Tools are more efficient for the given task by design. But interestingly humans also make appearance hand gestures of tools to communicate as well (e.g. rock-scissor-paper play). A tool is grossly composed of the handle and the part which directly acts on the target object – called the “actor” (e.g. hammer head acting on the nail). HandyTool maps the hand structure to that of the actor part of the tool. Thus, HandyTool eliminates the indirectness of having to use the handle and possibly provides a more vivid/interesting experience and even improved performance of using the tool by the user becoming “actor” itself. The mapping is both structural and metaphorical.

One immediate issue in the design of HandyTool is how to establish the structural mapping between the tool and the hand. The hand/fingers are usually the more dexterous with higher degrees of freedom than the tool. Different mappings might be preferred by different users. We have conducted a survey (details of the survey omitted) asking users to designate the most preferred, intuitive and natural mapping for various tools (see Table 1 for few examples). Most tools we surveyed came up with one or two prevalent mapping forms. Note that in the actual usage, the user simply has to follow the visually guided mapping which is expected to be easily understood and accepted (see Fig. 1).

Table 1. Examples of survey for intuitive mapping
Full size table

4 Usability Experiment

A usability experiment was carried out to assess the projected merits, comparing HandyTool by two factors: (1) the use of metaphor and (2) the type of interaction device used for hand-based tool activation (the hand itself was tracked using a 3D tracker for all treatments). The experiment was therefore designed as a 2 factors × 2 levels within subject repeated measure. The four treatments are as follows (see Fig. 1):

  • RB: No metaphor (Real) + Button (Hand/fingers movement enacts the tool by a button press, and virtual hand/fingers is visually overlaid as if grabbing the handle).

  • MB: Metaphoric + Button (Hand/fingers movement enacts the tool by a button press, but virtual hand/fingers is visually overlaid and shown according to the structural metaphor).

  • RF: No metaphor (Real) + Finger tracking sensor (Hand/fingers movement enacts the tool by moving the handle grabbed by the hand/fingers).

  • MF: Metaphoric + Finger tracking sensor (Hand/fingers movement enacts the tool according to the structural metaphor, i.e. HandyTool).

The experimental task involved the subject to take the tool and carry out an associated task. Three tools/tasks were selected to be tested and evaluated: (1) hammer/striking in nails (2) tongs/picking and placings object, (3) pliers/rotating screws in (see Fig. 2). The quantitative dependent variables included the task completion time and accuracy (defined differently for different tasks). We also administered a subjective survey assessing general usability (NASA-TLX [3]), simulation sickness (SSQ, [5]), enjoyment/preference and presence/immersion level (modified and reduced PQ, [15]).

Fig. 2.
figure 2

The three experimental tasks: hammer: strike nails, plier: pick and rotate/place cubes, plier: twisting in screws.

Full size image

The testing platform was implemented with Unity3D and run on a desktop PC with the HTC VIVE head set. For MF and RF, finger movement was tracked by the Leap Motion sensor and likewise for the hand position. The virtual hand/fingers were visualized according to the motion data (scaled properly depending on the size of the hand and tool). As for MB and RB, the HTC VIVE controller was used for hand tracking and button press (no finger movement tracking). When the controller was used, a default hand/fingers pose (appropriate for the given tool) was visualized over the target tool (see Fig. 1). Further experiment procedural details are omitted due to space restriction.

5 Results and Discussion

A total of 17 subjects participated in the experiment (11 females and 6 males, average age of 23.4), who were given the 4 treatments in a balanced order. Our basic expected outcomes were that both quantitative and subjective performance will be significantly better with the use of HandyTool (MF). ANOVA/Tukey (or Kruskal-Wallis/Mann-Whitney) was applied with the Bonferroni’s adjustment to analyze the experimental data.

It was found that, overall, the task completion time and accuracy were significantly better with use of the button (MB-RB over MF/RF) when enacting the tool. However, the use of the metaphor was not helpful especially when finger tracking sensors was used. The similar trend was found for the subjective ratings, i.e. better usability, higher immersion/presence and enjoyment/presence were found with the use of the controller button, and the structural metaphor was found to be helpful, but not significantly (See Fig. 3).

Fig. 3.
figure 3

Experimental results – Quantitative (above) and subjective ratings (below)- \( * \) indicate p < 0.05.

Full size image

In summary, contrary to our conviction, the use of the metaphor did not bring about the projected merits. It was apparent that the instability of the finger tracking sensor much affected the general usability and other subject evaluation criteria. The only solace was that the use of metaphor was somewhat a factor when the stable button device was used (namely, MB > RB for the tongs and pliers, but not for the hammer), partially confirming our hypothesis that the direct tool enactment improved task performance. Given the interaction is stable, the subjective indicators were generally very high when the metaphor was used (MB). Perhaps, the effect of the metaphor could be different for different tools and tasks as well. In addition, subjects reported the clear preference for the use of controller (button device) through which the user is able to get tangible feedback of the tool (vs. the use of Leap motion sensor to track finger movement in the mid-air).

One observation was that it seemed that metaphoric control was not all that different from the real (no metaphor case): e.g. fist posture over the hammer head vs. grabbing the handle, or tweezing over the blade vs. over handle (too simplistic). The evaluation was also somewhat oriented toward task efficiency rather than in the experience itself. Considering this, we have applied the idea of HandyTool to controlling virtual puppets (e.g. mapping fingers to body joins) and deployed it for children’s education (e.g. dexterity development) and play (see Fig. 4).

Fig. 4.
figure 4

HandyMan for puppet avatar/control as applied to children’s education and play.

Full size image

6 Conclusion

In this paper, we introduced “HandyTool” a method (and an interface) for virtual object manipulation based on a metaphorical/structural mapping of various everyday tools to our hands and fingers. The basic idea is to transform the hand/fingers into a proper tool (e.g. a fist becoming a hammer head) and gesturally apply it to and manipulate the target object (e.g. inserting a nail) directly. Our experiment was able to partially show the benefit of the HandyTool approach when basic usability is established with stable tracking. Therefore, in the future, we would like to test the use of gloves as a more stable finger tracking device. Metaphors may also be useful depending on how much the metaphor reduces the complexity and the therefore the type of tool being used.