Keywords

1 Introduction

Humans are capable of experiencing the world through five senses: sight, smell, taste, hearing and touch. Machines which interact with humans can also experience or act upon the world via these senses as well, however, much of the electronic machines developed during the 20th century lent itself to only two of these senses: visual and audio interactions [1, 2]. It is only recently with the rise in virtual reality (VR) technology where the touch aspect (i.e. haptic feedback) has begun to be of interest to consumers [3]. Haptic feedback is the application of the sensation of touch on a human via a force feedback [4] and it is imperative for immersive telerobotic operations in virtual or physical reality [5]. The use of haptic feedback in systems can range from audio-visual editing, multimedia publishing, vehicular operations, control rooms etc. [6]. In terms of telerobotic operations within virtual reality (VR), the inclusion of tactile perception summed with the innate kinaesthetic perception of VR creates a more immersive environment, attributed to the multi-dimensional information relayed to users [4].

The use of vibrotactile feedback on finger tips have been investigated extensively for almost half a decade now [7]. Generally the investigation of the vibrations are to do with how we distinguish between vibrations and how we perceive them. Currently the controllers used to simulate human hands in VR do not lend themselves very well to immersion, as an example the controllers for an Oculus Rift simply have triggers on them for grabbing and releasing objects [8]. This does not fully represent the complexity and versatility of the human hand as it cannot vary the force at which we can grip objects nor does it allow the possibility of simulating the ‘feel’ of the object [9]. There are wearable data gloves out on the market which can interface with VR technology, however, these devices are quite expensive and they do not have readily available user studies so their reliability for immersion is uncertain [10]. Therefore this project will propose a cheaper alternative made from readily available commercial products and incorporate a user study on vibrotactile feedback.

2 State of the Art

Reports investigating the application of vibrotactile feedback on finger tips have existed for almost 50 years [7], however, as technology improves, the conclusions drawn from the experimentation setup of earlier reports may not reflect the trends found in more recent experiments. In addition to this the experimental setups are not set up for a more compact use and although they are generally quite similar [7, 11,12,13,14,15], they are not the same as there are varying factors between them which could affect results across various papers e.g. angle of the finger, proportion of fingertip placed on surface, age etc. [12]. Due to the large size of the technology it may be noted that the conclusions drawn from the experiments may not translate to a portable device. One way in which vibrotactile feedback can help in immersion is to replicate the sensation of touching an object. When using a finger to scan a surface distinctions can be made between textures of the surface as vibrations are produced on the surface of the skin and travel through mechanoreceptors [15].

The Meissner and Pacinian Corpuscles are the mechanoreceptors in the fingertips which are responsible for the ability to distinguish between vibrations. The Pacinian corpuscles are responsible for detecting high frequencies above 40 Hz [16] whereas the Meissner corpuscles can detect low frequencies with a maximum peak of 200 Hz [17]. Due to the overlap in frequency ranges between the two corpuscles it could suggest that there is an obtrusive relationship between the two which may affect a participant’s ability to distinguish between frequencies, furthermore it suggests that the coding of the two mechanoreceptors may be different [11].

3 Hardware/Software Architecture

3.1 Dataglove System Design

The data glove is made using the LilyPad system [18], a set of Arduino-type microcontrollers used in development of interactive wearable systems. Vibrotactile feedback is provided to the fingertips via a set of six vibe boards (max frequency of 250 Hz) sewn to a Simblee BLE board [19] with conductive thread and a coin cell battery holder to power the board if used wirelessly. Since the LiyPad has a Bluetooth chip, an attempt was made to enable wireless communications between the data glove and computer connected to another Bluetooth LilyPad along the serial port. It was discovered that Arduino-Arduino Bluetooth communication was not useful as the range to connect was very short and communication was not always stable (Fig. 1).

Fig. 1.
figure 1

The completed data glove, with five vibe-motors on the fingertips (left) and one on the side of the palm (bottom center), Simblee board (bottom right) and battery holder (top right).

Full size image

3.2 Virtual Reality – Object Stiffness Design

The data glove is tracked in real time in a 3D space via a Leap Motion Controller [20] and displayed in virtual reality on a computer. A virtual object is simulated as a spring between the index finger and the thumb. By definition of Hook’s law if the displacement between thumb and index finger were to decrease, the force, or in this case the frequency, would increase. Figure 2 displays the human-machine interaction system. A computational node based on the ROS architecture is used to determine the distance between the index finger and thumb and send that data to the data glove. There are 5 distances/ranges which will cause the vibes to change frequency: greater than 75 mm, 75–60 mm, 60–45 mm, 45–30 mm and less than 30 mm; frequencies are mapped to these ranges which increase as the distance gets smaller.

Fig. 2.
figure 2

Human-machine interaction system design

Full size image

4 Experimental Design

In order to test the effectiveness of the dataglove two usability tests (user studies) were conducted. The first test would be to determine how users perceive stimulus on different fingers. The second test would be to determine the JND of virtual objects with different “stiffness”. Before the experiment were conducted we obtained official approval (reference code #1833) from the Queen Mary Research Ethics Committee (QMREC), because of human participants being involved.

4.1 Simultaneous Stimulation on Different Fingers

This test was to determine how users perceive vibrational feedback and how the number of active vibrating motors would influence the user’s ability to locate the position of the feedback. There were 10 participants in total, 6 female and 4 male, ages 23–29 and were all students of Queen Mary University of London.

There were two key factors in this test: the level of frequency and the number of simultaneous stimuli (i.e. quantity of stimuli) provided to the user. Each factor was tested at several different levels. The vibe motors a frequency range of 130 Hz to 230 Hz at minimum and maximum voltages respectively (1.8 V and 3.6 V) according to the vibe motor data sheet. As such four different levels of frequency with an increment of 30 were set. For each level of frequency, different number of fingers were given feedback in a random combination and order. The configuration of frequency is displayed in Table 1. Once participants agreed to take part in the experiment they were asked to wear the dataglove appropriately and rest their right arm and hand comfortably on the desk. Noise cancelling headphones were provided to eliminate the sounds from the motor.

Table 1. Configuration of frequency and amount of stimulus
Full size table

Vibrational feedback of the motors was controlled via a separate mobile application developed for the Simblee BLE board. Participants were asked to be seated opposite the examiner and should not see the mobile phone interface. The participants were then allowed a pre-experiment section which allowed them to adapt to the glove. They would feel the different frequencies on each finger. The formal test then begins. At each frequency level, different number of stimuli were provided to the participant in incremental order. First, only one motor would be active at a time and the five motors were turned on once each in a random order. Second, two motors were turned on together, according to Table 1, there are 10 possible combinations in this situation. To prevent the user from being tired of a tedious experimental process, only 5 random combinations were tested. Similarly for the combinations of three and four motors only 5 random combinations were chosen. Finally, all the motors corresponding to the five fingers were turned on together at once. Whenever there was a new active vibrating motor, participants would tell the experimenter which finger or fingers had an active motor. Participants were allowed to take a one-minute break before each frequency test. The whole procedure lasted about 20 min.

4.2 Simulation of the Stiffness of a Virtual Object

Another user study was designed to determine what were the minimum changes to the “stiffness” of virtual objects i.e. the frequencies mapped to the ranges, such that the user can distinguish between two virtual objects (i.e. which one is harder/softer?).

A just noticeable difference (JND) experiment would therefore be appropriate to determine the minimum amount of changes required for it to be noticeable. The following table is a list of suggested mappings of frequencies for the object based upon 5 distinguishable frequencies (0, 140, 170, 200, 230).

With each mapping from the table, one of the ranges in the frequency scale increments by one discrete level of frequency. As an example, going from mapping 1 to mapping 2 means an increment of 1 frequency level as the range of 45–30 is increased to 170 in mapping 2 and going from mapping 1 to mapping 4 means an increase of 3 frequency levels as 60–45, 45–50 and 30 > is incremented by 1 level each. For ease of understanding, the increase in frequency levels will simply be referred to as positional differences (Table 2).

Table 2. Table of suggested mapping of frequencies
Full size table

The proposed setup for the JND experiment was for users to compare 20 randomly chosen pairs of mappings from the table and asked which object they felt was more ‘stiff’. The stiffer object in this case being the one with the higher mapping. Since this experiment required a right hand to conduct it was necessary for participants to have control over their right hand. There were 10 participants in total, students and lecturers from the university aged 20–36, were asked to read the information sheet and given a quick reiteration of the experiment by the investigator. Once the participants confirmed they understood what the experiment would be and how the data would be analysed, they were then given a consent form to sign. After this procedure had concluded the experiment began. The experiment was conducted on a laptop PC as this experiment could be conducted anywhere which suited the schedule of participants. Manual switching of the objects was required by the investigator, however, participants were free to move onto the next object or pair whenever they wished, in addition to this they could switch between either object in a pair if they wished to feel the first one again. No time limit was set for this experiment in order for participants to proceed at their own pace although on average the experiment would take anywhere between 20–35 min. If the participant wished to take a small break they were free to do so and resume the experiment after. Participants were given the option to view the hand in 3D space using the visualiser and the distance between the fingers was also given to them. Initially the experiment was to be conducted using the index finger and thumb, however, the participants made note that the index finger was not particularly consistent at vibrating. The index finger was then switched to the middle finger which made the vibrations more consistent as well as making pinching the object feel “more natural” according to the participants.

5 Results and Discussion

5.1 Detection of Feedback on Multiple Fingers

In Experiment 1, participants need to judge which finger or fingers were receiving the vibrational feedback at different vibration frequency levels. To evaluate if the two factors, vibration frequency and the amount of vibration feedback stimulus, had an influence on the perception of the feedback independently, we calculated the judgement accuracy of participants overall on different frequency and stimulus numbers separately. The results of groups and the mean value of the judgement accuracy of each group are shown in Figs. 4 and 5. A one-way analysis of variance (ANOVA) method was used to determine whether there are statistically significant differences among those groups. This process was conducted twice as any significant difference in either factors should be determined (Fig. 3).

Fig. 3.
figure 3

Participant during the second test

Full size image

As it can be seen in Fig. 4, the average frequencies show a slight declining tendency, which means as the frequency decreases, the judgement accuracy slowly falls down. However, this difference turned out not to be significant, F = 0.212, p = 0.886 (5% significance level was used in this case). Thus, the level of frequency does not affect participants’ perception of positions of stimulus on the dataglove. The standard deviation value at all five frequency levels are relatively stable but quite large, which indicates that the accuracies under different stimulus quantities differ greatly from each other.

Fig. 4.
figure 4

Judgement accuracies at different frequency levels

Full size image

Figure 5 illustrates the perception accuracy when subjects received the vibration feedback on different fingers at the same time. As shown in the figure, perception accuracy fluctuates among different groups widely. When there was only one stimulus provided to subjects, the perception accuracies at all frequency levels were showed to be as high as nearly 1, which means almost all participants could tell which finger was feeling the feedback correctly. When there were two stimuli provided to subjects, the average accuracy shows a minor decrease to be 0.905. However, the accuracy sharply goes down to 0.5 or less when the stimulus is provided to 3 fingers or more.

Fig. 5.
figure 5

Judgement accuracies with different number of simultaneous stimuli

Full size image

To compare them more precisely, a series of pairwise comparisons were conducted between these groups. The result shows that there was no significant difference between one-stimulus group and two-stimuli group, yet these two groups differ from the other three groups significantly. No significant difference was showed between pairwise comparison among group 3, 4 and 5. Thus, the quantity of stimulus has an influence on the perception of the positions of stimulus. And when the quantity of stimulus is less than or equal to 2, there is large probability that the user could perceive which finger is feeling the stimulus correctly. However, when there are more than two vibration stimulus provided to the wearer simultaneously, it’s hard for them to tell which fingers are feeling the vibration

5.2 Detection of Different Stiffnesses of Virtual Objects

The results of the experiment can be seen in Fig. 6. The results show a noticeable trend as the positional difference increases the accuracy increases as well. It was surprising to note that even at the minimum difference of 1 positional difference, the average was still almost 0.5 or 50% (the chance level) which means that differences were most likely not being perceived very well. However, this is the lowest accuracy of the positional differences and the increased positional differences make a bigger impact. The minimum JND is 0.8 which means that the minimum positional difference for a user to notice is 3 positional difference. This could be attributed to the fact that there are 5 ranges where the frequency can change, if the majority (3/5) of the ranges were different from the previous object then it was a lot easier to notice. Therefore it could be concluded that if there is a majority difference in ranges when comparing two objects then it is easier to determine which object was stiffer than the other. From the testing it was found that the synchronisation of the vibes on the middle finger and the thumb were not always consistent which made the testing of the experiment difficult for some of the participants. This also means that the experiment was not without errors. Other errors found in the experiment could be the inaccuracy of the leap motion in response to the amount of IR light in a room leading to incorrect ranges given to users.

Fig. 6.
figure 6

JND results of the experiment

Full size image

Other observations include the conductive thread coming loose after vibes had been running for long periods of time. If the user moved their fingers together too quickly, they would skip ranges and thus skip frequency ranges.

Advantages of this feature include the fact that it is also very easy to implement. The majority of the code used to run the feature e.g. the leap motion was already readily available as a package on the site. Although the simulation was not always synchronised the participants made note that it did mimic resistance well.

Improvements to this include rewiring the conductive thread to the vibes and ensure that the thread will not come loose. The synchronisation issue could be fixed by using timestamps and delays to ensure that the changes are processed and not skipped. It is also necessary to repeat the experiment in case the synchronisation issue was a significant factor in affecting the results. In addition to this the experiment should be conducted only in rooms with low levels of IR light to limit the amount affecting the leap motion. In future experiments it might also be of significance to the user if there was a 3D model of the object for them to visualise, it is possible that the 3D visualiser of the hand and the numerical value of the distance between the fingers was not enough to help the user determine the stiffness.

6 Conclusions

Our tests have highlighted some of the features and limitations of the proposed system. Our results show that when the number of simultaneous stimuli on the fingertips was less than 3 most participants could correctly perceive which finger was receiving the feedback; however, with 3 or more stimuli, the perception accuracy decreases sharply. Interestingly, there was no significant difference among groups with different vibration frequency levels. The simulation of stiffness of virtual objects was implemented with success; we verified that a minimum positional difference of 3 in the frequency was necessary for a user to just notice the difference i.e. the majority of the frequencies when comparing objects of varying stiffness should be different if any differences are to be observed.

However, there were small issues with the components we have chosen to implement our system; the Simblee LilyPad is cheap and excellent for broadcasting to multiple users, however, the quality is not always guaranteed and thus consumers may find that the features do not meet the expected level, especially in terms of data synchronisation on the board and data transmission with the BLE protocol.

The approximate cost of the complete system is less than 250 GBP: the Leap Motion is commercially available for about 80 GBP, and the cost of all components needed to assemble the data glove is about 150 GBP. This is much cheaper than the cheapest available systems available on the market for fingers tracking with haptic feedback. Although we have not yet compared our solution to any existing product, the features we implemented work sufficiently well to provide some useful haptic feedback to users, and the preliminary findings of our user tests can be used in the future to compare this technology with more expensive systems, and to drive development in this area. In the next future we aim to test our system in telerobotics applications, and in particular for robot learning by human demonstration. A robotic hand equipped with tactile/force sensors on the fingertips (such as [21] or [22]) can be remotely controlled by the human user to haptically “explore” an object, i.e. to touch different parts of the object while receiving the vibrotactile feedback when contacts are experienced, with the final objective of finding good grasp configurations for the robotic hand; interestingly, the human demonstrations (or, “suggestions”) can complement automatic search algorithms, e.g. they can be used as priors for a Bayesian Optimization search [23, 24].