Airborne Ultrasound Tactile Display

Abstract

The authors and colleagues invented an ultrasound-based noncontact tactile display in 2008 and have been developing this technology since then. It is suitable for gesture input systems and aerial imaging systems because no physical contact is required to provide haptic feedback. An ultrasonic phased array generates a focal point of airborne ultrasound to press the skin surface. The amplitude modulation of ultrasound provides vibrotactile stimulation covering the entire frequency range of human tactile perception. The position of the focal point is computationally controlled to follow users’ hands and/or provide a trajectory of stimulation. While this technology was originally invented as a tactile display, a wide variety of other applications has been recently reported that exploit noncontact force generated at a distance. Examples include noncontact measurement by pressing or vibrating objects, levitation and manipulation of small objects, and actuation of fragile or soft materials. The present chapter describes the background, principles, systems, and applications of this ultrasonic technology.

1 Introduction

Aerial touch panels are being tested as next-generation touch-panel interfaces. When users reach toward floating images in front of them, contact between the users’ hands and the images is detected, and the images react accordingly. Such interactive floating images [1–3] provide clean, dirt-free touch panels that can be used in medical operation rooms, hospital waiting rooms, food processing plants, restaurants, entertainment, games, exhibitions, and digital signage.

In light of the foregoing, demands for noncontact and mid-air haptic feedback are emerging. People who experience aerial touch panels feel a strong need for haptic feedback. Human vision, despite its high lateral resolution, is poor at determining the distance to an object. Therefore, haptic feedback is helpful for a user to perceive and confirm that his/her finger contacts the aerial touch panel. The absence of haptic feedback yields uncertainty regarding the contact instant and input operation. Moreover, haptic feedback is indispensable in manipulating three-dimensional (3D) volumetric images.

There are two known methods for realizing haptic feedback in air. The first is to wear wearable devices such as a glove with small vibrators [4], motor-driven belts [5], or pin arrays [6]. One problem of this method is that users must wear the devices in advance. The second method involves transmitting tactile stimulation via air. For example, air jets [7] and air vortices [8, 9] were used for providing noncontact haptic feedback to bare hands.

Our research group first demonstrated that focused ultrasound produces perceivable force on a bare hand in air [10]. Our ultrasonic device (airborne ultrasound tactile display, AUTD) has hundreds of ultrasonic transducers and creates an ultrasonic focal point using phase control. High-amplitude ultrasound within the focal point presses objects in the propagation direction. This phenomenon is known as acoustic radiation pressure. The output force is sufficient to stimulate human tactile sensation. The focal point is generated at an arbitrary position in a 3D space. One of the greatest strengths of the AUTD is that both the spatial and temporal resolutions are high, and hence, various tactile patterns can be reproduced. It can produce not only active-touch sensation for aerial touch panels but also passive-touch sensation for alerts, communication, guidance, and other purposes [11]. Additionally, the AUTD offers precise movement of the focal point, which can be used to reproduce trajectories of hand-written letters and/or illustrations [12].

Other researchers have tried to use the AUTD for their own purposes. Ciglar proposes the AUTD as a musical instrument with tactile feedback [13]. Researchers at Bristol Interaction and Graphics combined the AUTD with a mobile TV [14], demonstrated an interactive table on which tangible objects were freely moved [15], developed an aerial interaction system comprising the AUTD and an acoustically transparent screen [16], and reported the simultaneous generation of multiple focal points [17].

In this chapter, first, the theoretical background of AUTD is explained. There are two main principles: acoustic radiation pressure and phased array focusing. Second, an example device is introduced. Last, developed systems based on the AUTD are shown. The latest application trials other than tactile displays are also introduced.

2 Ultrasound-Based Tactile Stimulation

Production of tactile stimulation by ultrasound has been attempted underwater [18–21], and there are two methods for this. The first involves direct stimulation of nerve fibers underneath the skin by ultrasound [18, 19]. In this method, ultrasonic waves penetrate the skin because their characteristic acoustic impedances are comparable. The second method is to press the skin surface by acoustic radiation pressure [20, 21]. In this method, a reflective membrane or plate is used to reflect ultrasonic waves.

In contrast, the proposed method is conducted in air and requires no reflective material. Users can receive tactile stimulation with their bare hands because total reflection occurs at the skin surface owing to the large difference in acoustic impedance between air and skin. This feature reduces the complexity of the AUTD.

In this section, we review the theoretical background of the AUTD. There are two main principles: acoustic radiation pressure and phased array focusing. The control methods and characteristics of ultrasound are also discussed.

2.1 Acoustic Radiation Pressure

The acoustic radiation pressure [22, 23] is a nonlinear phenomenon of ultrasound. When an ultrasound beam is reflected vertically at an object’s surface, the surface is subjected to a constant vertical force in the direction of the incident beam (Fig. 8.1). Assuming a plane wave, the acoustic radiation pressure P [Pa] acting on the object’s surface is written as

$$ P=\alpha E=\alpha \frac{I}{c}=\alpha \frac{p^2}{\rho {c}^2} $$

(8.1)

where E [J/m³] is the energy density of the incident wave, I [W/m²] is the sound intensity, c [m/s] is the sound speed, p [Pa] is the RMS sound pressure of ultrasound, ρ [kg/m³] is the density of the medium. α is a constant depending on the amplitude reflection coefficient R and the amplitude transmittance coefficient T at the object’s surface (α = 1 + R ² – T ²). In the case of air and human skin, α is 2 because of total reflection: R = 1 and T = 0.

Fig. 8.1

Radiation pressure on object’s surface

Note that p ² is typically assumed to be negligible (approximately zero) in the wave theory for linearization. The radiation pressure P becomes noticeable when the sound pressure p is relatively large such that p ² is not negligible. Additionally, (8.1) suggests that the spatial distribution of the radiation pressure P can be controlled by synthesizing the spatial distribution of the sound pressure p.

2.2 Phased Array Focusing

The phased array focusing technique is used to achieve sufficiently high-amplitude ultrasound. An ultrasonic focal point is generated by setting adequate phases. The phase θ _i [rad] for the i ^th transducer is written as

$$ {\theta}_i=2\pi \frac{d_i-{d}_0}{\lambda } $$

(8.2)

where d _i [m] and d ₀ [m] are the distances from the focal point to the ith and 0th transducers, respectively (Fig. 8.2). λ [m] is the wavelength. Theoretically, the peak value of the sound pressure in the center of the focal point is proportional to the number of ultrasonic transducers. The focal point can be generated at an arbitrary position by tuning the phase delays.

Fig. 8.2

Difference in distance from focal point between ith and 0th transducers

There is a trade-off between the spatial resolution (size of focal point) and array size. It is theoretically derived that the spatial distribution of ultrasound generated from a square transducer array is approximately sinc-shaped [24]. The width of the main lobe (w [m]) parallel to a side of the square is written as

$$ w=2\lambda \frac{L}{\;D} $$

(8.3)

where L [m] is the focal length, and D [m] is the side length of the square (Fig. 8.3).

Fig. 8.3

Device size, focal length, and size of focal point

Note that (8.3) is obtained using an approximation of the distance. It is no longer valid when the focal length L is far shorter than the array size D. Additionally, the directivity of ultrasonic transducers become noticeable at such a distance.

2.3 Vibrotactile Stimulation

Vibrotactile stimulation can be provided by the amplitude modulation (AM) of ultrasound. The acoustic radiation pressure is controlled between zero and the maximum value. The simplest method of control is turning the ultrasound ON/OFF at a constant frequency (Fig. 8.4). Although conventional vibrators have the possibility of loss of contact because of the viscosity of the skin, the AUTD can apply the intended vibration to the skin because it requires no physical contact.

Fig. 8.4

Overview of driving signals

Regarding human tactile properties, a bandwidth of 1 kHz is sufficient for tactile displays [25]. The AUTD can produce various tactile feelings because the carrier frequency, 40 kHz, is far higher than the required modulation bandwidth, 1 kHz. The frequency characteristics of the transducer are important here.

2.4 Intensity Control

Pulse-width modulation (PWM) can be used to easily solve electrical circuits, rather than modulating the amplitude of driving signals. The driving signal V(t) [V] is a rectangular wave written as

$$ V(t)=\kern0.5em \left\{\begin{array}{l}{V}_0 \kern0.6em \left(nT\le t<nT+W\right)\hfill \\ {}\kern0.62em 0\kern0.5em \left(nT+W\le t<nT+T\right)\hfill \end{array}\right. $$

(8.4)

where W [s] and T [s] are the pulse width and cycle time of ultrasound, respectively (Fig. 8.4), and n is an integer. We control the pulse width W. The amplitude of the fundamental frequency component a ₁ [V] of V(t) is written as

$$ {a}_1=\frac{2}{\pi }{V}_0\left|\; \sin \pi \frac{W}{T}\;\right| $$

(8.5)

According to (8.5), a 50 % duty ratio (i.e. W = T/2) yields the maximum carrier amplitude. The output sound pressure is proportional to a ₁ because the driven device is a resonator, and the resulting acoustic radiation pressure is accordingly proportional to |sin(πW/T)|².

2.5 Absorption in Air

Air is a lossy medium, and its attenuation coefficient β [Np/m] for a plane sound wave varies according to the wave frequency. The energy density E at the distance z [m] is written as

$$ E={E}_0{e}^{-2\beta \kern0.1em z}, $$

(8.6)

where E ₀ is the energy density at the transducer surface (z = 0 mm). According to [26], the attenuation coefficient β at 40 kHz is 1.15 × 10⁻¹ Np/m (i.e., 1 dB/m), and β is proportional to the square of the frequency. Figure 8.5 shows the relationship between the frequency and the energy loss rate at z = 200 mm. The energy loss is 4 % when the frequency is 40 kHz but over 50 % when the frequency is 200 kHz. We use 40-kHz ultrasound because the attenuation is relatively small and 40-kHz ultrasound transducers are commercially available. If we limit the range to 400 mm, the energy loss is less than 10 % everywhere in the workspace.

Fig. 8.5

Energy loss rate with respect to frequency

As shown the previous section, the wavelength also determines the spatial resolution. Thus, there is a trade-off between the spatial resolution and effective distance.

2.6 Safety Limits

Regarding human safety, there are two ultrasound-intensity limits. The first concerns heat damage to tissue under the skin; therefore, we estimate the reflection coefficient of the skin surface. The characteristic acoustic impedance of skin (soft tissue) Z _s and that of air Z _a are 1.63 × 10⁶ N∙s/m³ and 0.0004 × 10⁶ N∙s/m³, respectively [27]. The reflection coefficient R is determined as

$$ R=\left|\frac{Z_{\mathrm{s}}-{Z}_{\mathrm{a}}}{Z_{\mathrm{s}}+{Z}_{\mathrm{a}}}\right|\approx 0.9995 $$

(8.7)

In this case, 99.9 % (= R ² × 100) of the incident acoustic energy is reflected at the skin surface, i.e., an approximately total reflection. In the field of ultrasonography, a safe level of tissue exposure to ultrasound is defined as 100 mW/cm² [26]. Because 0.1 % of ultrasound power is absorbed by skin, ultrasound up to 100 W/cm² can be applied to the skin surface. The sound intensity I gives the acoustic radiation pressure: P = 2I/c = 5882 Pa.

The second safety limit concerns the impingement on the human ear. The limit recommended in [28] is 110 dB SPL. The sound pressure at a distance of 300 mm from a single transducer is 121.5 dB SPL, which is larger than the recommended limit. Furthermore, hundreds of transducers are used in the AUTD. Until the safety of the AUTD is confirmed, users are advised to wear headphones or earmuffs to protect their ears.

3 Implementation

Various AUTD prototypes and systems were developed. Their performance differs because of the number and arrangement of ultrasonic transducers and other design parameters. In this section, one of the prototypes [29] (Fig. 8.6) is considered as a typical example. It has 285 ultrasonic transducers [30] (Nippon Ceramic, T4010A1, 40 kHz, 10 mm in diameter) arranged within a 170 × 170 mm² square area. The power-supply voltage is 24 V. The resolution of the position control of the focal point is 0.5 mm, which is determined by the digitization of the phases in the control circuit.

Fig. 8.6

Focused ultrasound flips up paper strips

Firstly, the spatial distribution of the sound pressure of a 40-kHz ultrasound was measured by a microphone mounted on an XYZ stage. The array size D was 170 mm, the focal length L was 200 mm, and the wavelength λ was 8.5 mm. Then, the diameter of the focal point w was estimated as 20 mm. The measurement results on the focal plane show that the generated focal point has an equal diameter (Fig. 8.7). The measurement results along the acoustic axis are shown in Fig. 8.8. The peak value was 2,585 Pa RMS (162 dB SPL), which is 100 times larger than that for a single transducer, although 285 transducers were used. This decrease may originate from the directivity (100° half-amplitude full angle), the incident angle of each ultrasonic wave into the microphone, and/or other problems arising in practice.

Fig. 8.7

Spatial distribution of sound pressure on focal plane (x-y)

Fig. 8.8

Spatial distribution of sound pressure along acoustic axis (z)

Secondly, the time-response characteristics were tested. The focal length was 200 mm, and the microphone was fixed at the focal point. The sound pressure and radiation pressure are shown in Fig. 8.9. The 40-kHz driving signal was modulated by a 100-Hz rectangular wave. CH1 is the sound pressure that is the normalized waveform of the raw data, and CH2 is the radiation pressure that is the squared and normalized waveform of the waveform extracted by a lock-in amplifier. The rising time is shorter than 1 ms, indicating that the refresh rate is higher than 1 kHz. Figure 8.10 shows the frequency characteristics of the radiation pressure from 1 to 1,000 Hz. The radiation pressure at 1,000 Hz is nearly 5 dB less than that at 400 Hz.

Fig. 8.9

Normalized waveforms of ultrasound (CH1) and radiation pressure (CH2). (a) 100-Hz modulated waveforms. (b) Close-up of (a)

Fig. 8.10

Frequency characteristics of radiation pressure

Thirdly, the output force was measured by a digital scale. The prototype vertically pressed down the center of the scale as a result of focused ultrasound from a 200-mm height. The maximum measured force was 16 mN, which is sufficient to produce vibrotactile stimulation on a palm.

Lastly, the PWM-based intensity control was confirmed. Figure 8.11 shows the normalized force, which was measured by a digital scale. This force is a squared sine function of the duty ratio, as theoretically predicted.

Fig. 8.11

Normalized output force vs. duty ratio

4 Applications

We developed application systems using AUTDs, as discussed in this section. Applications other than a tactile display are also introduced.

4.1 Touchable Holography

A system that adds tactile feedback to floating images was developed [31] (Fig. 8.12). The images are projected by HoloVision (Holo17T, Provision Interactive Technologies, Inc.), which provides floating images from a liquid-crystal display (LCD) by utilizing a concave mirror. The projected images float 300 mm away from the HoloVision. The position of a user’s hand is estimated by triangulation using two infrared (IR) cameras (Wii Remote, Nintendo Co., Ltd.). A retroreflective marker is attached on the user’s finger and illuminated by IR LEDs. According to interactions between the user’s hand and floating images, noncontact tactile feedback is adequately provided by the AUTD.

Fig. 8.12

Touchable holography; (top) system overview; (bottom-left) raindrops; (bottom-right) small animal wandering on a palm

4.2 Tactile Projector

A noticeable feature of the AUTD is the freedom of stimulation-point location. The AUTD can generate passive haptic sensations at various parts of the human skin for alert and guidance, e.g., in a driver’s seat. To demonstrate this application, the large-aperture AUTD shown in the top photos in Fig. 8.13 was developed [11]. It comprises 3 × 3 units that are designed to be connected to each other and collaborate together. By connecting more units, the workspace can be extended to realize a larger-aperture AUTD, under which people can receive haptic information even while walking or working.

Fig. 8.13

Haptic projector; (top-left) large-aperture AUTD using 3 × 3 units; (top-right) system overview; (bottom) combination with visual projection

An example of a combination of the AUTD with visual projection is shown in the bottom photos of Fig. 8.13. A moving image projected on skin, together with its haptic stimulation, evokes the realistic sensation of a small animal or bug. This system can also be used for communication. Suppose that you are under a large-aperture AUTD and observe a visual image of another person. If tactile stimulation on your skin is synchronized with the person’s image touching you, you feel as if touched by the person.

4.3 HaptoMime

This system demonstrates an aerial touch panel with haptic feedback [32]. As shown in the upper-right image of Fig. 8.14, an aerial-imaging (AI) plate manufactured by Asukanet Co. Ltd. produces an aerial image as a copy of the LCD image [3]. The ultrasonic wave radiated from the AUTD is reflected by the AI-plate surface and forms a focus at the finger position detected by an IR touch sensor. Because the visual response and haptic feedback are well synthesized, users can operate the aerial touch panel comfortably. A drag operation for aerial icons is shown in the top-left photo of Fig. 8.14. A numeric pad, a keyboard, and free drawing are shown in the bottom photos. For the number input, the finger is stimulated in a short time around the touch moment. In the drag operation and free drawing, the tactile stimulation is produced while the finger is detected in the virtual plane.

Fig. 8.14

HaptoMime; (top-left) system appearance; (top-right) system structure using reflected ultrasound; (bottom) floating-touch-panel demos of numeric pad, keyboard, and free drawing with haptic feedback

4.4 HORN (3D Haptic Hologram)

The aforementioned examples involve traveling ultrasonic waves synchronized with finger motion. The HORN (hapt-optic reconstruction) system produces a floating image with haptic feedback using a stationary ultrasonic wave [33]. The energy density of the stationary wave is spatially modified to be consistent with the visual image; thus, multiple users can touch the image at different positions simultaneously. For a stationary virtual object, people feel tactile sensation even without temporal amplitude modulation; hence, silent (noiseless) tactile feedback is realized. The top figures of Fig. 8.15 show HORN, which creates haptic holograms in a space surrounded by an octagonal layout of 16 AUTD units with visual floating images. The bottom-left photo shows stationary objects (letters) with rough textures. This photo does not suffer from delay or error due to a motion sensor, because the ultrasonic wave is stationary and the haptic sensation is produced by the user’s hand motion. The bottom-right photo shows a dynamically deformable object according to the finger motion sensed by a Leap Motion Controller (Leap Motion, Inc.).

Fig. 8.15

HORN; (top-left) system appearance; (top-right) dimensions of system; (bottom) demos of 3D volumetric image with haptic feedback

4.5 Other Applications

Applications other than a noncontact tactile display have been explored. The advantages of AUTDs are the noncontact feature and spatial control. On the other hand, the drawback is that the output force is weak and in only one direction. Our investigation revealed that many applications benefit from the advantages of AUTDs in spite of the drawbacks.

The following are the authors’ ongoing studies. Here, we call our device an airborne ultrasound focusing device (AUFD) instead of the AUTD because it is not restricted to a tactile display.

• (a) The surface-compliance distribution of a soft material is measured [34]. The surface is locally pressed by an AUFD and observed using a laser ranger. The compliance is estimated according to the surface deformation. The distribution is obtained by scanning the measured point.

• (b) The static-electricity distribution on a thin film of polyimide or other material is measured [35]. A low-frequency radio wave is radiated while the film is locally vibrated by an AUFD. The static electricity is estimated according to the amplitude of the radio wave detected by an antenna. Scanning yields a distribution result.

• (c) A medical training system is developed using an AUFD [36]. A thin membrane representing skin is deformed by an AUFD to reproduce the pulse/thrill sensation for palpation training. The softness of the membrane is not lost, because there is no hard structure.

• (d) Artwork is created that exhibits creature-like motion by standing-wave acoustic levitation [37]. An AUFD generates a movable acoustic standing wave. A small, black particle representing a fruit fly is levitated in the standing wave and moved above a plate containing leftovers.

• (e) The reflection property of a soap film is controlled [38]. The reflection state of the film vibrated by AUFD is diffuse, whereas the original reflection state is mirror. By altering these two states at a high speed, the reflectance of various materials is represented owing to the persistence of vision.

• (f) Three-dimensional noncontact manipulation of small particles is demonstrated using multiple AUFDs [39]. This technology is further extended to real-world computer graphics [40], in which the levitated objects play the role of physical pixels.

• (g) A fur carpet turns into a computer display when graphics are drawn on it and erased [41]. Two AUFDs control the distribution of flattened and raised fibers in a noncontact manner.

• (h) Artificial pollination is conducted in plant factories [42]. Flowers are vibrated by an AUFD mounted on a monitoring robot moving in the room. Thus, pollination is automatically achieved, without bees and humans.

5 Conclusion

The airborne ultrasound tactile display (AUTD) produces noncontact tactile stimulation. Its theoretical background, including acoustic radiation pressure, phased array focusing, vibrotactile stimulation, intensity control, absorption in air, and safety limits, was discussed. Next, a prototype device was introduced as an implementation example. The focal point was ~20 mm in diameter, the maximum output force was 16 mN, and the refresh rate was 1 kHz. Applications of the AUTD were demonstrated: floating images with haptic feedback, a tactile projector, a floating touch display, and all-directional tactile feedback. Additionally, applications other than a noncontact tactile display were introduced.