Keywords

1 Introduction

Recently, two examples of interaction systems through floating images with midair haptic feedback have been demonstrated. One is HaptoMime [1] that enables interaction with a floating 2D image and produces noncontact tactile feedback by using air born tactile display (AUTD [2]). The other example is HaptoClone [3] that produces haptic and optical clone image. In HaptoMime, the interaction was limited in 2D and a midair touch panel was presented. HaptoClone realized 3D interaction through 3D images, but the image was optically produced from a real object. Even before the two examples, we can find systems that enable interaction between user and programmable 3D image with tactile feedback, as RePro3D [4]. However, it needs wearing a special device to obtain haptic feedback. In contrast to these, this research produces programmable 3D image with noncontact tactile feedback. Therefore, the proposed system in this paper produces programmable 3D image with noncontact haptic feedback, which possesses all the three features that it is programmable, 3D, and noncontact. Figure 1 shows schematic of the equipment.

Fig. 1
figure 1

System overview

Full size image

2 System

Figure 2 shows appearance of the system. The system is composed of three elements, a light field display device (3D part), depth sensor (sensor part), and tactile feedback part (AUTD part). First, the 3D part is divided into rotating LED arrays and a micro-mirror array plate named AIP (Aerial Imaging Plate by ASUKANET, Co., Ltd.). A 3D image is drawn by flashing each LED at an appropriate timing. By reflecting this 3D image to the user’s side using AIP, it produces a touchable 3D optical image. In the current specification, the 3D image is created with 36864 dots in a cylindrical region of 48 mm height and 146 mm diameter. The refresh rate of the 3D image is 18 Hz. Second, the sensor part has a role to get the position and posture of the user’s hands by a depth sensor. The obtained data is transformed to a simulation model in a game engine, Unity. The model interacts with a virtual object prepared in Unity. By synchronizing this interaction in virtual and real spaces the contact between a user and optical 3D image is produced. Third, the AUTD part creates haptic feedback on the hand by the ultrasonic radiation pressure. The refresh rate of the haptic stimulation is also 18 Hz in the current system.

Fig. 2
figure 2

Photograph of the device

Full size image

3 Application and Purpose of the Research

Using this device, a user can handle a computer-created 3D object with touch sensation in real world (Fig. 3). In principle, it will support the 3D design in computer, where the designers can create 3D models intuitively as if they are shaping clays. However, the current device is the first prototype, and there still is a distance to such a practical application. The most considerable problem is the visual image resolution. In the current system, the 3D image is created by only 36,864 dot light sources for a cylindrical region of 48 mm height and 146 mm diameter. Therefore, the purpose of this research is to clarify the condition of contact model and haptic feedback for efficient handling of 3D virtual objects. Important parameters are the threshold between contact and out-of-contact states, friction coefficient, and dynamics parameters of the virtual object. It is also crucial to examine the problems of optical occlusion and finger penetration through the virtual object surface. In this conference, we perform a demonstration to evaluate the feasibility of 3D-virtual-object handling. Attendees can experience the usability under typical interaction models.

Fig. 3
figure 3

Virtual object in unity and optical 3D image in real world

Full size image