Keywords

1 Introduction

Recently, various head-mounted displays (HMDs) for consumers, such as Oculus Rift S and Oculus Quest 2 [1], have been developed and sold. As the latest devices, standalone HMDs, such as Oculus Quest 2, have attracted enormous attention and gained popularity because they are inexpensive, perform highly, and do not require a connection to a PC, allowing virtual reality (VR) experience with them alone.

However, in the current VR technology, the development of devices that present sensations other than visual and audio sensations has not progressed. In particular, it is paramount to present haptic feedback as the next step; this will lead to the widespread use of VR technology in education, entertainment, and medical fields, such as rehabilitation.

Based on the above, we had developed a wearable HMD-integrated force feedback device called SPIDAR-HMD in our previous work [2]. SPIDAR-HMD was developed exclusively for the Oculus Rift CV1 and had a problem that it could not be easily attached to other HMDs with different shapes and operating systems. In addition, the update frequency of force feedback was as low as 300 Hz, and there was a problem that it was less than 1 kHz, which was necessary for the stable force feedback [3]. We describe the details of this device in the next section.

In this study, we selected Oculus Quest, a standalone HMD that allows users to move freely, as the target device; in addition, we aimed to improve its force feedback update frequency. Therefore, we designed and manufactured a mounting frame for Oculus Quest, changed the microcomputer used for the controller to a higher performance one, and tried a communication method of Bluetooth and Wi-Fi. Further, we constructed a system that allows two users to share a VR space simultaneously and experience haptic sensation using the hand tracking function built into Oculus Quest. Then, we measured the magnitude of the force presented by the device and evaluated the force feedback update frequency and screen frame rate with different communication methods.

2 SPIDAR-HMD

SPIDAR-HMD was developed for Oculus Rift CV1—a VR device for consumers developed by Oculus (currently Facebook Technologies LLC) [4]. Figure 1 shows the device in use. The device consists of two plastic frames made by a three-dimensional (3D) printer attached to the front and rear of the HMD. Three direct-current (DC) motors with rotary encoders are attached to the front frame, and a control unit is attached to the rear frame. The ends of the strings from the pulleys of each DC motor are connected at a point to which a ring (end effector) is attached. When the user moves a finger with the end effector on the finger, the length of the strings from each pulley changes, and the 3D position of the finger is measured by the rotary encoder. When the finger collides with a virtual object, the tensions of the strings are controlled by changing the duty ratio of the pulse-width modulation (PWM) signals to the DC motors, and the appropriate reaction force is felt at the fingertip. A mobile battery is used as the power source for the device, and serial communication using Bluetooth serial port profile (SPP) is used for communication with a PC. Table 1 shows the specifications of SPIDAR-HMD. The fingertip position measurement has three translational degrees of freedom (DOFs), and the force presentation has almost one DOF.

3 The Proposed Device

3.1 Outline of the Proposed Device

As shown in Figs. 2 and 3, the same as SPIDAR-HMD, the proposed device has three DC motors equipped with rotary encoders fixed to the upper left and right and the lower center of the front of the HMD, respectively, as well as a control unit attached to the rear end of the head strap.

Fig. 1.
figure 1

SPIDAR-HMD in use

Full size image
Table 1. Haptic specifications of SPIDAR-HMD
Full size table
Fig. 2.
figure 2

Configuration of the proposed device

Full size image
Fig. 3.
figure 3

State of wearing

Full size image

3.2 System Configuration

As shown in Fig. 4, the system configuration consists of an HMD, which is the audiovisual presentation part, and the force feedback device. Unity—a game engine—was used as the software development environment for this device, and the force control values and the rotary encoder pulse count values were sent and received by Unity scripts. This communication was performed in Wi-Fi UDP STA mode via a wireless router, Wi-Fi UDP AP mode, or Bluetooth SPP. On the force feedback device, the HMD sends the force control value simulated by Unity to the device via Wi-Fi or Bluetooth as three force control values (each 10-bit unsigned integer). The force feedback device determines the duty ratios of the PWM signals based on the received values and sends these signals to the motor driver circuit to present the force sensation to the fingertip. This device also counts the pulses output from the three rotary encoders and sends the values (each 16-bit signed integer) to the HMD. The HMD uses these values to determine the 3D position of the finger on a Unity script. On the visual presentation part, Unity updates the VR space using the position of the finger and draws the image according to the position and angle of the HMD. In addition, using the original hand tracking function of the HMD, an avatar of the user’s hand can be displayed.

3.3 Control Unit

As shown in Fig. 5, we used a development board—ESP32-DevKitC-32D (ESPRESSIF SYSTEMS CO., LTD.) [5]—equipped with a microcomputer—ESP-WROOM-32D—as the control unit. The DC motor used was 1724T006SR with rotary encoder IEH2-512 (FAULHABER MINIMOTOR SA) [6]. Figure 6 shows the flowchart of the program installed.

Fig. 4.
figure 4

System configuration

Full size image
Fig. 5.
figure 5

Control unit

Full size image

3.4 Simulation System

As shown in Fig. 7, we created a test content that allows the user to touch a flat wall with a fingertip and check the reaction force. In this content, a spherical object representing the proxy of the fingertip for reaction force calculation is displayed to check the operation of the program. The hand avatar is displayed by the hand tracking function, which is an original feature of Oculus Quest. Unity calculates the reaction force generated when the fingertip collides with an object in front of the user. During the collision, the finger penetrates the object, but the proxy does not penetrate the object, instead, it moves along its surface, following the movement of the finger. The reaction force (force sense information) is calculated via a spring-damper model using the difference between the fingertip and proxy positions. Figure 8 shows the flowchart of the force calculation and finger coordinate calculation of the simulation system. After establishing a connection with the motor controller, the simulation system sends the initial force control values. Next, the system receives the count values and calculates the coordinates of the finger position. Afterward, the system performs the collision check with the virtual object, and if there is a collision, it calculates the reaction force and sends the force control values to the control unit; otherwise, it sends the initial values.

Fig. 6.
figure 6

Flowchart of the microcomputer control program

Full size image
Fig. 7.
figure 7

Visual presentation of test content

Full size image

4 Evaluation Experiment

4.1 Purpose

The purpose of this experiment is to measure the magnitude of the force presented and evaluate the performance in the force feedback update frequency and the screen frame rate due to the communication method.

4.2 Measurement of the Magnitude of the Force Presented

The plane formed by the three string outlets is the x–y–z plane, with the x-axis facing right toward the HMD, the y-axis facing upward, and the z-axis facing forward. The positional coordinates of the three string outlets were (98, 82, 0), (−98, 82, 0), and (0, −82, 0), respectively, in mm. At the measurement point (0, 0, 100), the presentation force was measured ten times each while increasing the duty ratio from 0% to 100% using 5% increments. The mean values and standard deviations are shown in Fig. 9.

4.3 Measurement of Force Feedback Update Frequency

The data transmission frequency \({f}_{s}\) from the HMD is determined by a fixed timestep, which is the update period set in the FixedUpdate function of Unity. Each time the microcomputer receives data from the HMD, it switches between high and low on a certain output pin. Since one cycle of this signal is two times the update period, the reception frequency \({f}_{r}\) of the microcomputer can be obtained. Ten measurements were made at each set frequency of \({f}_{s}\), and the mean and standard deviation were calculated. The results are shown in Figs. 10, 11, 12, 13, 14. The blue dots represent the mean value of \({f}_{r}\), whereas the vertical bars represent the standard deviation (left-hand axis). The solid straight line represents \({f}_{r}={f}_{s}\), and the gray straight lines above and below represent by the equation \({f}_{r}={f}_{s}\pm 10\) [%], respectively (left-hand axis). The red dots represent the average screen frame rate (right-hand axis).

4.4 Results and Considerations

Evaluation Experiment of the Magnitude of the Force Presented. The approximate line shown in Fig. 9 is a linear approximation using the least-squares method. When the duty ratio is \(x\) [%] and the presenting force is \(F\) [N], the approximate equation is given by

$$F=0.017x-0.1136.$$

This approximation shows linearity from the coefficient of determination \({R}^{2}=0.9931\approx 1\). Therefore, the above equation can be used to present a force of approximately 0 –1.6 N by changing the duty ratio of the PWM control. The maximum force of this device (1.6 N) was two times that of the conventional SPIDAR-HMD shown in 2.1 as 0.8 N, which was due to the change in the diameter of the pulley attached to the motor from 20 to 10 mm.

Evaluation Experiment of Force Update Frequency

Bluetooth SPP

In Figs. 10 and 11, the screen frame rate was kept at 72 Hz in all cases. The reception frequency \({f}_{r}\) was within −10% of the transmission frequency \({f}_{s}\), except for the case with multiple players with hand tracking. Even for multiple players with hand tracking, it was within −10% in \({f}_{s}\le 800\) Hz.

Wi-Fi UDP STA

In Figs. 12 and 13, the screen frame rate was kept at 72 Hz in all cases. The upper limit of the reception frequency \({f}_{r}\) was in the range of 200–350 Hz when it was within −10% of the transmission frequency \({f}_{s}\), which was significantly inferior to other communication methods. This was probably because the system was connected to a LAN connected to the Internet and was therefore affected by other communication traffic.

Wi-Fi UDP AP

This was only the case without multiple players since hand tracking communication with another device over a normal Wi-Fi connection was impossible in this case. As shown in Fig. 14, the screen frame rate slightly decreased after 900 Hz but remained above 70 Hz, so there was no problem. The reception frequency \({f}_{r}\) was within  −10% in \({f}_{s}\le 800\) Hz.

4.5 Consideration of Update Frequency

From 4.4.2.1 to 4.4.2.3, it is considered appropriate to use Bluetooth SPP, which can ensure at least a communication speed close to 1 kHz, for all conditions in this environment.

Fig. 8.
figure 8

Flowchart of force and coordinate calculation

Full size image
Fig. 9.
figure 9

Relationship between duty ratio and the magnitude of the force presented

Full size image
Fig. 10.
figure 10

Bluetooth SPP (single player).

Full size image
Fig. 11.
figure 11

Bluetooth SPP (multiple players).

Full size image
Fig. 12.
figure 12

Wi-Fi UDP STA (single player).

Full size image
Fig. 13.
figure 13

Wi-Fi UDP STA (multiple players).

Full size image
Fig. 14.
figure 14

Wi-Fi UDP AP (single player).

Full size image

5 Conclusion and Prospects

In this study, we adapted SPIDAR-HMD, which was made for Oculus Rift CV1 in our previous study, to Oculus Quest, which stands alone and is easy to handle. The frame of the SPIDAR-HMD was redesigned, and the microcomputer of the controller was changed to ESP32 to build an environment that could function with various communication methods. Then, we evaluated the performance of the proposed device in terms of force feedback.

In the evaluation of the magnitude of the force presented, the proposed device could present a force of 1.6 N, which was two times that of the conventional SPIDAR-HMD. In addition, the relationship between the duty ratio and the presentation force was clarified.

For measuring force feedback update frequency, it was considered appropriate to use Bluetooth SPP, which could ensure at least a communication speed close to 1 kHz for all conditions in the environment. However, the increase in computational load of the HMD due to the complexity of VR space and increase in load due to the Internet connection for sharing VR space will affect the communication speed.

As a prospect, we would like to conduct haptic experiments of various shapes of virtual objects and explore the improvement possibilities of the proposed device. In addition, to achieving higher and more stable performance, we plan to make this device compatible with Oculus Quest 2, which is a successor to Oculus Quest with improved CPU performance.