Ultrasound Mid-Air Haptic Feedback at the Fingertip

Abstract

Ultrasound haptic feedback is typically used to augment the multi-sensory experience with spatiotemporal patterns projected on the users’ hands. Many studies have considered the usability of such techniques on the users’ palms as it is more sensitive to ultrasound stimuli. Studies exploring the ultrasound feedback on the users’ fingertips have utilized large ultrasound phased arrays to project perceptible haptic stimuli. Spatiotemporal patterns at the fingertips using smaller phased arrays have been largely unexplored due to their weaker sensations. In this chapter, we first present a survey of ultrasound stimuli patterns that have considered the users’ fingers for haptic feedback. Then, a set of spatiotemporal stimuli for ultrasound feedback on the finger is presented along with results from a user study and associated examples of mid-air gestures. In the end, the prospect of ultrasound haptic sensations at the fingertip is summarized from a survey.

1 Introduction

Mid-air gestures can be natural and intuitive with haptic feedback, adding realism to virtual interactions (Culbertson et al. 2018; Grandhi et al. 2011). Freehand mid-air interactions with ultrasound haptic feedback are less disruptive than wearable haptic devices and can aid the feeling of immersion and presence (Pacchierotti et al. 2017; Rakkolainen et al. 2021). Current research in mid-air haptics has focused on creating virtual haptic shapes and patterns in mid-air for freehand direct exploration and localization. It has applications of mixed reality and haptic-augmented interfaces in automotive, digital advertising, and sterile medical interfaces (Rakkolainen et al. 2021). Mid-air haptics could enrich the user experience of applications of mid-air interactions, i.e., distant displays, ubiquitous environment, therapeutic assistance, accessibility, cultural heritage, text entry, sharing among devices and others (Koutsabasis and Vogiatzidakis 2019).

Most studies in the literature have explored ultrasound haptic feedback on the users’ palm, as it is more sensitive to ultrasound stimuli (Rakkolainen et al. 2021; Sun et al. 2019), but also because it provides a larger canvas for tactile stimulation. Most studies exploring the ultrasound feedback on the users’ fingertips have utilized large ultrasound phased arrays to project perceptible haptic stimuli (Matsubayashi et al. 2019). In contrast, spatiotemporal patterns at the fingertips using smaller phased arrays have been largely unexplored due to their limitation of creating stronger forces (Hoshi et al. 2010).

Carter et al (2013) initially proposed ultrasound mid-air haptic feedback at the fingertip with the pinching gesture. They created multiple focal points in mid-air and modulated its amplitude at different frequencies, e.g., 50  Hz and 200 Hz, to present different haptic cues. Since then, many modulation techniques have been developed to improve the ultrasound haptic feedback stimulation force that the users tested by exploring various 3D shapes like points, lines, circle, sphere, and pyramid patterns with their palms (Chilles et al. 2019; Hasegawa and Shinoda 2013; Kappus and Long 2018; Korres and Eid 2016; Long et al. 2014). Other research on ultrasound mid-air haptic feedback has focused on developing stimulation patterns to improve the perception of 3D shapes in mid-air, which users mostly tested with their palms. For example, Wilson et al. (2014) evaluated the localization and apparent motion of focused ultrasound on the user’s palm; Frier et al. (2019) explored the effect of spatial sampling strategy on perceived strength of a pattern, e.g., a circle; and Hajas et al. (2020) found that the accuracy and confidence in identifying geometric shapes, e.g., circle, square, and triangle on the palm improved significantly when the moving focal point or haptic pointer slowed down at corners.

Recently, researchers have started exploring ultrasound mid-air haptic feedback with the users’ fingers. For example, Howard et al. (2019) reported lower detection and identification thresholds for 15 cm long line patterns extending from the palm to the fingertips than the detection threshold for a single point on the palm. Matsubayashi et al. (2019) demonstrated ultrasound mid-air haptic feedback at the fingertip by stimulating the finger using a large phased array. Their system presented haptic cues when users touched and manipulated 3D virtual objects without wearing any device (Matsubayashi et al. 2019). Previously, researchers have tried ultrasound feedback at the fingertip using smaller phased arrays with limited success. For example, Sand et al. (2015) proposed a head-mounted display with ultrasound feedback for the finger and palm using a smaller phased array, but the stimulation was not strong enough for the fingertip. Palovuori et al. (2014) proposed an immaterial fog-screen display with mid-air ultrasound feedback, but with tileable small phased arrays to create focal point with higher intensity.

In this chapter, we present the prospect of ultrasound haptic feedback at the fingertip using a smaller phased array. To this end, we present the Ghostrokes technique, which evokes a stroking sensation on the fingers using a smaller phased array with well-perceived tactile sensation (Fig. 1).

Fig. 1

Ghostrokes: Ultrasound mid-air haptic feedback at the fingertip. a A user is swiping their hand in mid-air to scroll up/down and left/right. A stroking stimulation using ultrasound mid-air haptics is applied to the fingers in congruence with the finger movement, up/down and left/right. b The user could imagine moving their fingers on a trackball to scroll up/down and left/right. The stroking sensation from the ultrasound haptics is meant to elicit the rolling friction between the fingers and the trackball

2 Ghostrokes

Ghostrokes is a new touchless technique for haptic feedback on the fingers. It differs from other ultrasound mid-air haptic feedback techniques as it provides a stroking sensation to the fingers only. We implemented it using the Ultrahaptics evaluation kit device (UltraLeap Ltd.) which is based on a phased array with considerably smaller form factor than the system from (Matsubayashi et al. 2019).

Fig. 2

The Ultrahaptics evaluation kit with a 16 $\times$ 16 ultrasound phased array and a leap motion computer-vision sensor

We followed a laboratory-based participatory design approach to evaluate as well as explore the design space and applications of Ghostrokes. This device could uniquely enable Ghostrokes with its ultrasound phased array (see Fig. 2) that can provide $\approx 8.6$mm size haptic stimulation on the fingers with amplitude and spatiotemporal modulation of a focal point. The integrated hand and finger tracking sensor (leap motion controller) would allow for closed-loop/active haptic feedback. The working volume of this system is approximately the size of an inverted cone that extends 50 cm above (out) of the device center and with an opening angle of 45$^\circ$.

Fig. 3

The four Ghostrokes patterns on the index and middle fingers: (1) stroking from the base to the tip of the fingers, (2) stroking from the tip to the base of the fingers, (3) stroking from the tip to the base of the index finger and stroking from the base to the tip of the middle finger, and (4) stroking from the base to the tip of the index finger and stroking from the tip to the base of the middle finger

We explored the design space of Ghostrokes with three participants who did not take part in the controlled experiment. We found that ultrasound haptic feedback for Ghostrokes could be designed considering the following.

2.1 Ultrasound Modulation Technique

A range of ultrasound haptic modulations can be created using amplitude modulation (AM), spatiotemporal modulation (STM) (Frier et al. 2019), or lateral modulation (LM) techniques (Takahashi et al. 2018, 2020). All of these techniques apply vibrotactile stimuli to the skin and could thus be used to implement the Ghostrokes sensations. STM relies on the rapid movement of the focal point to create haptic sensation along a trajectory which could be tailored to create a stroking sensation. LM relies on small movement of the focal point to create a point haptic sensation which could be tailored to create a stroking sensation. AM does not require movement of the focal point to create a point vibrotactile stimulation. However, it could be readily used to create a stroking sensation, e.g., by moving the AM point along the stroking path in a similar way to Hajas et al. (2020).

2.2 Stroking Patterns

A range of stroking patterns could be designed with continuous and discontinuous movements of the focal point. Figure 3 shows four stroking patterns applied to the index and middle fingers [(see Zhang et al. 2020]. To create these patterns, a focal point rapidly jumps between the fingers in a zigzag motion at every step while it moves along the fingers at a slower speed. It creates an illusion of two focal points (or focused ultrasound tactors) moving along the two fingers. More complex patterns could be designed using this technique.

3 Experimental Evaluation

After the preliminary experiments with the three participants mentioned before, the four stroking patterns shown in Fig. 3 were chosen, and a within group lab experiment was performed with the four stimuli presented in random order. The group consisted of 18 participants (8 females and 10 males) aged between 22 and 40 (mean = 27.7 and s.d. = 5.4) and all right handed. The study session including the interview lasted between 40 and 50 min, and each participant was compensated for their time with a gift voucher.

3.1 Procedure

We began the experiment by measuring the lengths of the index and middle fingers of the participant’s right hand and then conducted a preliminary sensitivity assessment of their fingers with a two-point discriminator tool (Brand: Touch Test) with 4 mm gap and light pressure applied by the researcher (Lundborg and Rosén 2004). We then continued with a pre-study questionnaire to gather basic demographics before proceeding to carry out the study. An information sheet was given to the participants prior to the experiment. We also explained the information on the sheet to familiarize them with the experimental setup and procedure.

Fig. 4

Experimental setup. It was used during both the practice and test run of the patterns. The large slit allowed the participants to position and rest their hand while exposing their index and middle fingers to the tactile stimulation patterns delivered by the Ultrahaptics device

Fig. 5

The graphical user interface for the lab study is shown. It was used to practice and test the patterns. The participants could see it in front of them on screen

During the study, the participants were asked to sit comfortably on a chair and rest their hand on a support box which housed the Ultrahaptics evaluation kit. They adjusted the height of the chair and the orientation of the box according to their preference. The support box has a 5$\times$11 cm hole on top to expose the fingers to ultrasound stimuli. The participants could align their finger using a guide (see Fig. 4) on top of the Ultrahaptics device where the stimuli would be applied. We used an AM stimuli 200 Hz frequency and full amplitude range (0–1). The hand was placed 15 cm above the Ultrahaptics. The stroking period was set at 3.5 s, which the three previous participants found as a comfortable and natural sensation. We also developed a graphical user interface (see Fig. 5) to assist in the lab study. The room temperature was controlled to maintain the skin temperature and sensitivity constant. Finally, the participants wore studio headphones and an ambient white noise was played to prevent any audible clues from the device.

First, the participants were given one trial of the four patterns in a random order to practice them (using the ‘test’ button in GUI). During each trial, the stimulation of a pattern was given three times. The participants were then offered to practice any pattern of they wanted which was given to them. There was no limit on how many times they could practice. Then, each participant was given 40 trials (10 $\times$ 4 patterns) in random orders. They could see the pictures of the four patterns on a GUI in front of them. After each trial, they reported the pattern they felt. The researcher then recorded it using the GUI which was stored on anonymous files. After 20 trials, the participants reported the area of the finger where they could feel the stimuli. They were then given a 5 min break to rest. Then, the previous steps were repeated. At the end of the study, we asked the participants to fill in a questionnaire to feedback the ‘mental demand’, ‘temporal demand’, ‘performance’, and ‘frustration’ felt during the study. Lastly, we conducted an interview to gather feedback about their thoughts and suggestions on the Ghostrokes technique.

Fig. 6

a The number of mistakes made by the 18 participants is shown. b The confusion matrix of the errors made in recognizing the patterns is shown. A high score implies a large confusion

3.2 Results

We first analyzed the data for errors. The number of errors committed by the participants and the confusion matrix of errors committed for each pattern are shown in Fig. 6. Five participants correctly recognized all the patterns without making any mistake, while participants 4 and 17 made most errors having passed the two-point discrimination touch test with $\approx 75\%$ success rate. The participants made the least error in recognizing pattern 3 and most errors in recognizing pattern 4. Pattern 2 was only confused with pattern 3, but pattern 3 was only confused with pattern 2 only once in 180 trials. Pattern 1 and 4 were sometime confused with one another. This is interesting as all the participants who wanted to practice further had requested to practice patterns 3 and 4 a second time. Their first impression was that these patterns are more difficult than patterns 1 and 2. But none but one of the participants (with shaky hands) requested to try the stimuli for a third time. However, many participants had proceeded to the next stage of testing practicing the initial one trials. We did not record the practice session. In the interview, all participants said that they were confident recognizing patterns 1 and 2 during the first of the three stimuli given during a trial. But they sometimes waited for the second or rarely for the third stimuli to conform when they thought it could be a pattern 3 or 4. The ‘I don’t know’ option was not offered but the participants always guessed a pattern and never said they could not detect a stroking sensation or recognize a pattern. The stimulation areas of the fingers are shown in Fig. 7a. The participants felt the stimuli up to the fingertip which would not have contributed to the errors. The total number of errors was 29 in 720 trials, i.e., an overall accuracy of $96\%$.

Fig. 7

a The stimulation areas of the fingers are shown. b The confusion matrix of the errors made by the patterns is shown

The Shapiro-Wilk test for normality on the ‘given’ and ‘guessed’ data gave a $p<0.01$ for both which means they are not normally distributed. Consequently, we conducted the Kruskal Wallis test and found a $p=0.0609$. Because $p>0.05$, there is no significance of the pattern of the stimuli on the error rate. However, this is a significant result for Ghostrokes as all the patterns are statistically similar for the user’s to recognize.

In the interview, the participants described the haptic sensation as a stream of air and a soft brush stroking on the fingers and a finger drawing on the finger. They did not feel a hot or cold sensation. We had adjusted the length of the stimuli according to the lengths of the fingers. However, the participants did not perceive any difference of stroking speed. The preliminary workload assessment is shown in Fig. 7b. It suggests that the ultrasound haptic feedback might need attention, and the users might need stronger stimuli to boost their confidence; however, the stimuli is less frustrating.

3.3 Discussion

Rutten et al. (2019) argued that age can be strongly related to a decline in performance in identifying different mid-air haptic shapes. The young age of our participants might have contributed to higher accuracy. They also found that significant differences in accuracy rates for different types of mid-air haptic shapes. It is possible that the haptic patterns chosen in this work could have contributed to higher accuracy.

The lab study was conducted in the passive haptic feedback condition, i.e., the participants kept their hand still throughout the trials. In a deployment scenario, the positions of the fingers might need to be tracked. The stroking stimuli will need to be adapted considering the movement of the fingers to effectively elicit the sensation of the sliding of a tool or stroking a virtual object.

4 Applications

Following the controlled user study in our laboratory, we interviewed the participants for applications of Ghostrokes. The participants were mostly computer science and engineering graduate students working on the university campus under COVID-19 restrictions. We found the following applications of Ghostrokes innovative and relevant. They either represent an active stroking feedback while using a virtual tool or passive stroking feedback by a virtual object.

4.1 Active Stroking Feedback with Virtual Tools

All participants initially mentioned at least one touchless interaction using a virtual tool. Many were public interfaces which might be influenced by the ongoing COVID-19 pandemic. For example, automatic doors in public buildings sometimes open and close unnecessarily when a person is sensed near it, even if they don’t want to enter and exit through the said doors. This could lead to inconvenience, waste of energy, and sometimes minor accidents. One suggestion was to rotate an imaginary door handle to open or close the motorized doors. Ghostrokes could mimic the stroking sensation felt while the fingers slide on the door handle while rotating it. A similar suggestion was sliding imaginary doors to open motorized sliding doors, with Ghostrokes mimicking the stroking sensation of the edge of the doors sliding on the fingers. These ideas could be extended to use virtual tools to operate various motorized doors and windows in public buildings, with Ghostrokes simulating the natural stroking sensations on the fingers like using those tools physically.

Participants also suggested applications where the stroking sensations of Ghostrokes could be learned to interact with public interfaces. One suggestion was for active haptic feedback for touchless keypads for ATMs, public phones, cash machines, and elevators. The users can imagine pressing virtual keys/buttons in mid-air and the stroking sensation can guide and notify during the operation. The virtual buttons could be imagined, for example, 15–20 cm above the Ultrahaptics device where the user’s eyes meet the physical buttons and the position of the virtual button could be sensed with a depth camera. A stroking stimulation traveling toward the fingertip can notify the proximity of a button, and a stroking stimulation traveling away from the fingertip can notify the pressing of a button. Many participants suggested Ghostrokes as an assistive technology. It could help visually impaired users with touchless information input and output with the above-mentioned public interfaces.

All participants suggested use of Ghostrokes in playing games in virtual or mixed reality (VR/XR) environments. Players could select icons by dragging or pulling them and feel a corresponding stroking sensation. They could drag their hands on surfaces or accessories like the face of a horse or the hilt of a sword to feel them. They could use virtual pointing devices or accessories like an arrow or gun with mid-air gestures and their movement communicated with different stroking patterns on their fingers. They could also feel the pull of the string or trigger with a stroking stimulation. All the participants agreed that when a physical trackpad or mouse is not available, like for touchless interaction with a public display or in a VR environment, a virtual keypad, or trackball mouse could be useful. One participant suggested that Ghostrokes could be useful for extended reality where a small movement of fingers translate to large movement in the virtual world and the stroking feedback can elicit the extent of scaling.

4.2 Passive Stroking Feedback from Virtual Objects

Many participants suggested freehand stroking feedback is suitable for discrete, private, and secure communication in the public or part of a group. One suggestion was to receiving notification, assistance, or instructions as if someone is pulling or a virtual pointer is stroking their fingers to attract their attention to certain task or direction. For example, while exploring a map of a city in the public (for free attractions) or navigating a list of stores or restaurants in a mall (for deals) or wards in a hospital (for an available medical specialist), Ghostrokes can guide the attention to the point of interest on the hovering fingers which others cannot see, hear, or feel.

Other suggested applications of Ghostrokes relied on passive feedback which could be from real or virtual objects. Direction cues of Ghostrokes could assist visually impaired persons with navigation inside buildings. An interesting suggestion was to warn people of potentially dangerous situations like infected surfaces, sharp edges of objects or unseen obstacles. Ghostrokes could be used for nonverbal communication for user’s comfort or during stressful situations. For example, stroking feedback could give direction cues on the fingers during a lesson, like learning to drive or playing an instrument. Another notable idea was to receive (passive) stroking feedback for music or musical instrument with vibration traveling through the fingers like while playing drums. Participants also suggested stroking touch by virtual fingers using connected haptic devices for family members (romantic partners) to address isolation, boredom, and loneliness as being stroked is more pleasant than stroking and it decelerates heart rate (Triscoli et al. 2017).

From these suggested applications, we conjecture that the participants considered their mental models of real-life tools and haptics experiences.

5 Conclusion

Ghostrokes is a new ultrasound haptic feedback technique for the fingertips. It can provide easily perceivable haptic cues while implemented on the smaller commercial phased arrays. In this chapter, we have described its design space and the stroking patterns explored and designed with users. We reported the efficacy of recognizing the haptic patterns in a controlled user study and found an overall accuracy of $96\%$. We also presented a broad range of innovative applications suggested by the study participants following the lab-based user study. We envision Ghostrokes will pave a new way to deploy virtual tools for freehand interactions in real-life use case scenarios like public buildings and personal use case scenario like entertainment and gaming and consider the users’ real-life mental models to develop future applications.