Keywords

1 Introduction

The sense of touch has an essential role in human interactions in social and emotional communication and is particularly significant in social interactions. A short touch by another person can elicit emotionally grounding and engaging experiences, from the comforting knowledge of being touched by one’s spouse to the experience of anxiety when touched by a stranger [6]. The sense of touch can be stimulated in several ways: pressure, vibration, pain, temperature, movement, and position. Research into haptic displays has explored a variety of submodalities of touch such as vibrotactile [10], thermal [21], clenching [2], pressure [7], and dragging [8]. These haptic modalities were tested on various body parts using multiple form factors from head-mounted displays [3] to small thermo-electric wearables [17]. Haptic displays can render nuanced touch-based information, either tactile, kinesthetic, or both, to users in real, augmented, or virtual environments [3]. Touch can also enhance the meaning of other forms of verbal and non-verbal communication, e.g., touch intensifies emotions in addition to that displayed by the face and voice [14]. Shaking, hugging, patting, squeezing, and stroking are examples of touch that convey emotions [4], such as love and anger.

Peltier units have been embedded into wearable devices to apply thermal stimulation. While there has been some exploration of thermal stimuli in spatiotemporal cues, high-dimensional patterns have not been investigated. Cang and Israr [1] explored the communication of socio-emotional sentiment using vibrotactile stimulation, but multimodal stimulation, such as thermal and vibrotactile, has potential to better simulate social touch and other multimodal non-verbal cues found in human interactions. This research aims to design, demonstrate, and test a multimodal haptic display using eccentric rotating mass (ERM) vibration motors and ceramic Peltier units on the forearm as a medium to transfer vibrotactile-thermal (vibrothermal) patterns toward simulating social touch cues.

2 Related Work

2.1 Haptic Feedback

Thermal Feedback.

Wilson et al. [30] suggested that warm temperatures (>32 °C) could be used to convey the physical or social presence of other people. In contrast, cool temperatures (<30 °C) could convey people’s absence. Additionally, users strongly agreed on the application of thermal feedback in social communication and rating-re-lated representations. Hot under the collar [31] made the first attempt to map a range of thermal stimuli to dimensional models of emotions and suggested that the distribution of points better fitted a vector model than the circumplex model, thereby making it difficult to convey the full range of emotions through thermal feedback alone, and could widen the range of emotions represented in the presence of other feedback modalities. While researchers have explored the utility of thermal bracelets [21] for spati-otemporal feedback, more complex patterns of higher dimensionality have not been investigated. A fascinating insight by Tewell et al. [26] is that increasing the stimulated skin area eases the perception of thermal signals. Thermal Feedback Identification in a Mobile Environment [29] tested two-dimensional thermal icons while sitting or walking, and concluded that the direction of change was a valuable thermal feedback parameter in mobile environments.

Lee and Lim [16] developed a structured approach about the quality of heat (thermal expression unit, thermal expression, and the two levels of thermal expression composition) and discovered the critical expression elements (temperature, duration, location, and temperature change rate). According to the results from K. Suhonen et al. [13], when touch was mapped to thermal stimuli, they found that cold messages conveyed negative feelings, whereas warmth communicated positive feelings. When thermal icons are designed based on the responses of the skin, they can be accurately determined with little training [24]. Researchers considered a lower limit of about 15 °C-17 °C and an upper limit of 45 °C-52 °C to avoid pain irritations associated with thermoreceptors [19]. A minimum of 15 s between different trials has been suggested to avoid the thermal adaptation effects. Wilson et al. [30] highlighted that the role of emotions could vary depending on the scenarios in which thermal stimulation is applied (e.g., happy memories, social closeness). Emotions are considered to be connected to sensed variations in temperature [23, 25], where warm temperatures were found to be comfortable, pleasant, and promoted social proximity, while colder temperatures were perceived as being uncomfortable to most. “Warm” temperatures have been associated with words like generosity, happiness, humor, and sociability [31].

Vibrotactile Feedback.

A library of effects was created in Feel Effects [10], which altered the duration and intensity of tactile icons for testing on the backs of users. Turchet et al. [28] tested a walker’s emotional states using a planar vibrotactile display embedded into footwear. Salminen et al. [22] studied emotional experiences and behavioral responses to haptic stimulations. Yoo et al. [37] identified parameters like perceived intensity, carrier frequency, duration, and envelope frequency to have a clear relationship to tactile stimuli’s emotional responses, and develop design guidelines for tactile icons that have desired emotional properties. Vibro-glove [15] delivered seven facial expressions to the back of the hand through spatiotemporal vibrotactile feedback. Two-dimensional tactile moving strokes with varying frequency, intensity, velocity, and direction of motion were tested using Tactile Brush [9] on the backs of users. To study vibrotactile patterns and the value they contain in eliciting emotions, the Haptic Face Display (HFD) was designed and evaluated using a large set of spatiotemporal patterns to evoke affective responses [18]. Finally, the results from Haptic Empathy show that people can use vibratory feedback as a medium for expressing specific subjective feelings [12].

2.2 Multimodal Vibrotactile and Thermal Feedback

Thermal Icons [5] was the first study of intramodal and intermodal thermal and vibrotactile communication, where it was demonstrated that thermal and vibrotactile stimuli together do not appear to hinder interpretation across these submodalities, and hence, thermal changes may be a helpful addition to tactile icons. Wilson et al. [32] tried to expand the emotional expressivity of interfaces and provided examples of combining different modalities for affective states. Yang et al. [34] created a testbed using pin-array tactile and thermal feedback for psychophysical studies relating human touch sensation and perceived feelings of stimuli. Yang et al. [35] concluded through their study that multiple actuators are advantageous for simultaneously displaying combinations of amplitude and frequency. The perceived magnitude of vibrotactile stimuli was affected by temperature variation only for high frequency (>150) vibrotactile stimuli [33]. Yoo et al. [36] explored thermal cues’ emotional responses through constant-temperature thermal stimuli and concluded that thermal and vibrotactile parameters have clear and somewhat independent effects on emotional responses.

3 Prototype Design

3.1 Hardware Design

The proposed vibrothermal haptic display was designed as a wearable device that fits the forearm and can produce high-dimensional spatiotemporal patterns—see Fig. 1. Peltier units of size 20 mm × 20 mm × 5.1mm were used to generate thermal feedback, and Eccentric Rotating Mass (ERM) motors of 8 mm diameter were used to generate vibrotactile feedback. Peltier units of this size were selected as they are smaller than the traditional ceramic Peltiers and can be easily arranged into a matrix, thereby providing scalability. This device was built using an Arduino UNO, where each of the actuating components was controlled through a MOSFET. Power to these components was provided through a 5V power supply or a battery bank for portable use. An HC-05 Bluetooth module connected the haptic display to custom software, through which pre-defined and user-defined unimodal and multimodal patterns can be created and sent to the haptic device for real-time display. Figure 2 depicts the arrangement of Peltier units and ERM vibration motors for the proposed design. The ceramic Peltiers and the ERM motors are arranged in a 3 × 3 staggered matrix pattern. The surface area of the skin covered by this device is 10 cm × 10 cm. Figure 3 depicts another view of the proposed display when not worn.

3.2 Software Design

An Android application was created to provide users with an interface for controlling the proposed haptic display. The Arduino UNO controls the temperature changes of the Peltier units and the vibrations of the ERM motors. According to the commands received from the android application, the output voltage of each pin of the Arduino UNO is precisely controlled through Arduino code. The Android application consists of different screens enabling control over the different modalities available for the device. The user can also select between pre-defined patterns or create new patterns using the keypad provided. The android application interacts with the hardware using a Bluetooth module connected to the Arduino UNO. Figure 4 displays different screens from the Android application.

Fig. 1.
figure 1

Vibrothermal device on the forearm.

Full size image
Fig. 2.
figure 2

Arrangement of peltiers and ERM motors

Full size image

The main screen for the application is shown in Fig. 4a. The user can select from three available modalities. Once a particular modality is selected, the application moves to the next screen as shown in Fig. 4b. On this screen, the user can choose between pre-defined patterns or user-defined patterns. Figure 4c displays the various patterns that the user can select and play on the device, and Fig. 4d displays the keypad for drawing unique patterns. The locations on the keypad coincide with the actuators on the device to ease the creation of user-defined patterns. Before the user plays different patterns, the android application should be connected to the device using Bluetooth from the CONNECT button provided in the application.

Fig. 3.
figure 3

Prototype of vibrothermal display.

Full size image
Fig. 4.
figure 4

Android application screens.

Full size image

The proposed vibrothermal display can generate high-dimensional patterns in thermal and/or vibrotactile modalities. The main aim of testing vibrothermal patterns is to gather information about their relatability to social touch cues found in social interactions. Eight pre-defined vibrothermal patterns were designed. Figure 5 depicts the proposed pre-defined patterns for evaluation, namely UP-DOWN, DOWN-UP, LEFT-RIGHT, RIGHT-LEFT, LEFT-DIAGONAL, RIGHT-DIAGONAL, HOLD, and SINGLE TAP. Patterns like UP-DOWN, DOWN-UP, LEFT-RIGHT, and RIGHT-LEFT were commonly found in previous research [9, 15, 20]. The patterns on the diagonals, namely LEFT-DIAGONAL and RIGHT-DIAGONAL, were tested to research the change in spatiotemporal patterns across the forearm. Given their similarity to social touch cues commonly encountered during social interactions, patterns such as HOLD and SINGLE TAP were introduced to explore their reliability to social touch.

Fig. 5.
figure 5

Different patterns using vibrothermal modality.

Full size image

4 Study Design

The aim of the proposed experiment was to determine the absolute identification accuracy of social touch patterns simulated in the vibrothermal modality, and their relatability to touch cues used during social interactions. The study was performed with ten participants (average age was 26 years old; 7 males and 3 females participated) recruited from Arizona State University (ASU). The study was approved by ASU’s Institutional Review Board prior to recruitment. Two participants wore this device on their left hand while remaining subjects wore it on their right hand. The device was placed on the ventral part of the forearm, and a total of eight vibrothermal patterns were tested as shown in Fig. 5. A 15–20 s gap was provided between patterns to avoid thermal adaptation [11].

The study was conducted in the following steps:

  • Informational Phase: All human subjects were informed about the research and its potential benefits.

  • Familiarization Phase: All eight pre-defined patterns were presented and named for subjects for familiarization purposes before beginning training.

  • Training Phase: Each pattern was presented to subjects in the order of HOLD and SINGLE TAP, followed by the diagonal patterns, and finally, the horizontal and vertical patterns. For each pattern, subjects were asked to recognize the stimulus. If the pattern was identified correctly, the guess was confirmed; otherwise, if the pattern was misidentified, the user was informed and given the correct answer to enhance learning. To proceed to the next phase of this experiment, subjects needed to attain 75% accuracy within a set of the eight pre-defined patterns, or else the training phase was repeated until that accuracy was attained (no more than three repetitions were allowed).

  • Testing Phase: 24 (8 pre-defined patterns × 3 trials) vibrothermal patterns were presented to subjects in a random order. No feedback regarding correct or incorrect guesses was provided. The research team noted the responses of subjects.

  • Post-Experimental Survey: Participants completed a post-experiment survey consisting of Likert-scale questions inquiring about the patterns and their relatability to social touch cues experienced during social interactions.

5 Results and Discussion

5.1 Absolution Identification of Vibrothermal Patterns

Figure 6 displays the confusion matrix of the eight pre-defined patterns tested using the proposed vibrothermal device. The pattern SINGLE TAP was highly accurate with an accuracy of 100%. The HOLD and RIGHT-DIAGONAL patterns have the same accuracy of 96.67%. The patterns RIGHT-LEFT, LEFT-DIAGONAL and LEFT-RIGHT were reasonably accurate with accuracies of 93.33%, 86.67%, and 80%, respectively. The patterns DOWN-UP and UP-DOWN were less accurate, with 66.67% and 63.33% accuracies, respectively. It can also be noted that the pattern UP-DOWN was more often confused with LEFT-RIGHT and DOWN-UP, whereas less often confused with the RIGHT-LEFT pattern. The DOWN-UP pattern was more frequently confused with RIGHT-LEFT, LEFT-RIGHT, and less frequently confused with the UP-DOWN pattern. The LEFT-RIGHT pattern was muddled with the DOWN-UP and RIGHT-LEFT patterns. The participants had a little confusion, of about 6.67%, with the RIGHT-LEFT pattern, which is relatively less than the previous patterns. In addition, the LEFT- DIAGONAL was confused with the LEFT-RIGHT and RIGHT-DIAGONAL patterns. The RIGHT-DIAGONAL and HOLD patterns had an identification accuracy of 96.67%, with around 3.34% being confused with the LEFT- DIAGONAL and SINGLE TAP patterns.

Fig. 6.
figure 6

Pattern identification confusion matrix.

Full size image
Fig. 7.
figure 7

Graphical interpretation of mean accuracies.

Full size image

5.2 Relatability of Vibrothermal Patterns to Social Touch Cues

Responses on the relatability of the pre-defined patterns to social touch cues were noted on a 5-point Likert-scale and are depicted in Fig. 7. The patterns SINGLE TAP and HOLD were highly relatable, with mean values of 4.90 and 4.70, respectively. The patterns on the diagonals, i.e., the LEFT DIAGONAL and RIGHT DIAGONAL, had a mean value of 4.10 with a standard deviation of 1.5, which indicates some degree of interpersonal variation. The remaining four patterns, namely UP-DOWN, DOWN-UP, LEFT-RIGHT, and RIGHT-LEFT, had mean values between 3.50 and 3.80, which shows that these patterns were less relatable to social interactions. Patterns like SINGLE TAP and HOLD were intuitive in conveying social touch cues experienced during social interactions.

Summary.

For most subjects, the pre-defined patterns were easily recognizable with little training. A few participants found that the SINGLE TAP and HOLD were easy to identify, whereas other patterns were more confusing and required more concentration for identification. One participant noted, “It was easier to tell if the heat was on the Left or Right, but whether the heat was on the Top or Bottom was hard”. Most of the participants agreed that this device could be used to augment social interactions. Specifically, a user could use the technology to feel a social touch from a distance. A notable statement from one of the subjects mentioned that this device could be used in places where vocal communication is difficult, such as extreme environments; in particular, this participant mentioned interest in using the device while diving to communicate with diving partners. During the experiment’s training phase, all participants obtained 75% accuracy—i.e., 6 out of 8 patterns were identified correctly in a single attempt across all subjects. This shows that with little training, it is possible to familiarize and identify spatiotemporal vibrothermal patterns on the forearm.

6 Limitations and Future Work

Currently, ERM motors are used to produce vibrotactile stimulation. Further experimentation is needed to identify the best actuator option for producing rich spatiotemporal vibrotactile patterns that are perceptible. Peltier units also have limitations; these units have significant lag to reach target temperatures, which could be problematic during real-time social communication. Future implementations of the proposed design will investigate other methods and technologies for real-time thermal stimulation. Moreover, Peltier units are bulky, and vibrotactile actuators cannot be placed directly on top of these heating elements. This separates the stimulated body sites of thermal and vibrotactile stimuli, which may reduce the intuitiveness of multimodal haptic patterns; however, future work is needed to investigate any perceptual differences, and to identify the optimal configuration of thermal and vibrotactile actuators on human skin.

Furthermore, future improvements to the user interface will enable users to directly draw patterns on a mobile device and interpret them on the proposed device, similar to [1]. Such a method may provide valuable insight when testing the device during a real-time phone call or video conversation.

Finally, a study to explore the emotional response to vibrothermal patterns using this device is needed to understand mappings that have potential for use in social scenarios. The main objective for such a study would be to explore a 3-dimensional graph of valence, arousal, and dominance/power (dominant vs. submissive) domains with a set of pre-defined patterns. A 3-dimensional projection [27] will provide researchers with more in-depth insight into the overall structure of emotional space than the previous 2-dimensional circumplex or vector model.

7 Conclusion

The proposed vibrothermal haptic display prototype can be deployed to provide rich vibrotactile and thermal stimulation for tactile interactions. This paper discussed a preliminary study to confirm the identification of such spatiotemporal patterns and their relatable to social touch cues. According to the results in Fig. 6, the confusion matrix for warm stimuli has more misclassifications for a particular pattern, which aligns with previous research performed [21]. Patterns like SINGLE TAP and HOLD were very relatable to social touch cues experienced during social interactions. Other patterns like UP-DOWN, DOWN-UP, LEFT-RIGHT, RIGHT-LEFT were less relatable to social interactions experienced by the users.

It was also noted that the spatial location of the vibrothermal patterns relative to the arm significantly affected their identification. Identifying patterns on the distal and proximal areas of the ventral side of the forearm was difficult for subjects. Detecting patterns on the periphery of the forearm was easier compared to the center of the forearm. Patterns drawn along the diagonals of the display were discernible, but their relatability to social interactions varied among subjects. Using these insights, future work will evaluate the detailed mapping of vibrothermal patterns to emotions within the specific context of social interaction. We believe that by utilizing the submodalities of touch, more tactile information could be conveyed in a compact form factor to elicit emotional responses from the users.