Keywords

1 Introduction

Touchscreens have become a new standard for mobile devices as they enable a natural user interaction by directly touching and interacting with the items of interest, rather than using an external peripheral to map the gestures to the display and possible actions (e.g. a mouse). Unfortunately, current touchscreens are still devoid of rich haptic feedback, such as being able to render different textures and localised multitouch haptic feedback. Yet, rich haptic feedback on a surface presents several benefits. For instance, it is helpful for typing on a virtual keyboard as it increases performance and reduces typing errors [4, 11]. It can also enrich the interaction by providing different sensations to different actions to help differentiate them (e.g. long vs short click, etc.) or to enrich the overall experience (e.g. in a game) [9, 1]. Non-visual interactions could even be envisioned in order to help visually impaired access digital contents [15, 13].

Consequently, rich localised haptic feedback on a surface can open up many new interaction possibilities. However, delivering localised haptic feedback on a surface can be very challenging technically. The work conducted on user interactions often either directly equips the fingers of the users [13] or relies on a single or two actuators at most [15, 4, 11], thus avoiding the issues of vibration propagation when multiple actuators are required, but then with limited feedback possibilities. Few methods are currently available to render localised haptic feedback on a surface [6, 5, 2, 8]. The time reversal approach focuses on bending waves to produce localised impulsive displacements on transparent glass plates  [6]. Another method relies on a phased actuator array in a surface, which can focus ultrasounds to create localised mid-air sensations [5]. These two methods work in ultrasonic frequencies, far beyond the tactile sensitivity range. The perception of such stimulation then relies on nonlinear demodulation phenomena that are not easily controlled. Another method uses an array of electromagnets and a magnetorheological fluid [8] but this technique is not very accurate and thus, it cannot isolate the effect to areas as small as a single fingertip and thus is not suitable for multitouch. Another approach investigated the use of the vibration modes of a surface to localise haptic feedback [2] in the tactile sensitivity range. It combines the different vibration modes of the surface to vibrate chosen locations while canceling others. However, this method does not permit to choose the desired frequency of the haptic feedback. The Inverse Filter Method has been proposed [7] that provides such flexibility with a similar idea. This method uses a glass surface equipped with piezoelectric actuators glued on the bottom of the glass and produces a localised haptic feedback at the locations of these actuators while canceling the signals at the other actuator positions. A user study was conducted demonstrating that the stimulus on a finger was better perceived than without the method. However, no studies were conducted evaluating the recognition of multiple simultaneous stimuli. Recently, this method has been improved [14] by permitting localised haptic feedback at any point of a surface without the necessity to put the fingers on top of the actuators. Furthermore, this method provides a good resolution of about 1.5 cm, far below the wavelength \(\lambda _{250Hz}\) = 19 cm. However, this novel method has not yet been tested with users, which is the contribution presented in this paper.

Therefore, this paper presents a user study of two-point based pattern discrimination using the Inverse Filter Method (IFM) to investigate the effect of varying the temporal difference between stimuli on pattern recognition and subjective evaluation. The setup and results are presented in the following sections.

2 Experimental Setup

2.1 Setup Description

The user study carried out in this paper was conducted on the setup depicted on the left of Fig. 1. Eleven piezoelectric actuators (Murata 7BB-35-3, 35 mm diameter, 0.51 mm thickness) were glued on the bottom of a 96 \(\times \) 162 \(\times \) 1 mm touchscreen (7” pingbo PB70DR8272-R1), which can detect up to five fingers. An Arduino receives the finger localisation data through an I2C protocol. After decoding the data with Python, the number of control points can be obtained as well as their X, Y coordinates. Actuators were driven individually with piezo haptic drivers (DRV8662-Texas Instrument) delivering up to 200 Vpp. Surface displacements were measured by a laser vibrometer (Polytec OFV-5000/MLV-100) mounted on a motorised three axis platform. The acquisition device (NI-9264, NI-9205 and cDAQ-9174) allowed for a synchronous emission and acquisition of actuators and vibrometer signals. All signals were sampled at \(F_{s}\) = 10 kHz.

Fig. 1.
figure 1

Left: experimental setup. Right: finger positions of the participants.

Full size image

2.2 System Calibration

In order to apply the IFM, a matrix of impulse response is needed (more details in  [14]). It captures the mechanical transduction of the actuators as well as the propagation, reverberation and attenuation of waves into the touch surface at calibration points spaced at 2 mm. These entries of the matrix of impulse response are calculated as the ratio between the displacement above a calibration point and the driving signal sent to an actuator, in the frequency domain for each actuator on the surface. We chose to acquire experimentally the matrix with the multiple sweep method [12]. Each entry was measured by sending an exponential sweep sine signal sequence of duration T = 2 s, with frequency varying linearly from 0 to \(F_{s}/2\) = 5 kHz to all the actuators. All actuators were driven simultaneously with a time shifted version of the same exponential sweep while measuring synchronously the resulting displacement at the calibration point. Then, the matrix is the ratio of the displacement by the sweep signal in the frequency domain. The same procedure was repeated for all the actuators and all the calibration points. When the user places his fingers on the screen, the matrix of impulse response at the current positions of the fingers is computed. Using the initial matrix of impulse response and an interpolation function, we can calculate the matrix of impulse response of the current different fingers (Fig. 2).

Fig. 2.
figure 2

Calculation of the matrix of impulse responses under the four fingers.

Full size image

When the output signal sent to the different actuators is calculated thanks to the IFM, a rescaling is applied to this signal in order to avoid saturation at the output of the amplifiers, which in turn causes inaccurate localisation rendering. However, this rescaling induces an amplitude variation between 1.5 and 5 \(\upmu \)m under the different fingers. As a consequence, this amplitude is not controlled, in favour of an optimal localisation rendering, and depends on the position of the fingers on the plate. However, this is not an issue for the user study as the focus was on multitouch pattern recognition with an optimal setup. The haptic signal delivering to the different fingers was a burst at 250 Hz with 25 cycles.

3 User Study

The purpose of this user study was to conduct preliminary investigations of multitouch pattern recognition on a tactile surface using the IFM. Specifically, this study investigated whether users could discriminate vibrotactile stimuli on two different fingers and in particular simultaneous stimuli. To that effect, four different timing differences were chosen, i.e. 0 ms, 100 ms, 200 ms and 300 ms.

3.1 Methodology

Participants. The study was conducted with 12 participants (4f–8m), aged between 14 and 45 (M = 28.25, SD = 7.5). 11 participants were aged between 25 and 45 and only one minor participated. All but two participants were right-handed. Half of the participants were very familiar with haptic technologies (researchers recruited within the laboratory), while the rest had limited or no knowledge about haptics (a college student, a security coordinator, a project manager in construction project management and three researchers from the vision laboratory). None reported any issues with their fingers or sensitivity.

Technical Settings. The device was placed on a table, in front of the participant (see Fig. 1). The participants were instructed to place their fingers onto the tactile screen for the trials, at the positions that were comfortable to them, as displayed on Fig. 1, with their wrists on resting supports to minimize the fatigue. They wore noise-canceling headphones during the trials with pink noise to cancel any biais due to the noise generated by the setup. The experimenter used a standard Windows laptop both for running the Python application controlling the feedback and for logging the verbal answers.

Procedure. The experiment was a within-subject repeated measures design with four conditions (0, 100, 200, 300 ms), tested in different sessions. The order of the sessions was counterbalanced between participants as well as the direction of the stimuli within a session (i.e. either playing from left-to-right, or from right-to-left). There were 10 trials per stimuli/finger combination (depicted on Fig. 3) with half from each direction, accounting for 60 trials per session. In total, each participant performed 240 trials. The experiment lasted one hour on average.

In each trial, the task was to identify the positions/finger combination that received the haptic stimuli with the fingers numbered from 1 to 4 (e.g. stimuli on the index fingers of each hand corresponded to ‘2–3’, see Fig. 3). The participants were instructed to provide the answer verbally as soon as they recognised the stimuli, which was provided only once, whilst their hands remained on the device. The experimenter logged the answer by first, pressing a button to measure the response time and then, typing the given answer. This was a forced-choice experiment: if participants had doubts, they were asked to answer the most likely option. To accustom participants and reduce the impact of learning effects, prior to each condition, participants were presented with each of the stimuli twice per direction and performed a blind test.

After each session, the participants were asked whether they perceived a difference with the previous condition and to describe it. They were also asked to rate the difficulty of discrimination on a continuous numeric scale from 0 to 10 (10: very difficult). At the end of the last session, the participants were asked which timing difference they preferred and general comments about the perception. As for quantitative measures, the interface collected the responses and the response times. The response time was collected to provide trends, as the experimenter logging the responses induced a bias, in particular in terms of longer hesitations to answer for a condition.

Fig. 3.
figure 3

Stimulation number corresponding to the stimulated fingers.

Full size image
Fig. 4.
figure 4

Left: average performance (in %). Right: average response time (in s).

Full size image

3.2 Results

Recognition Rates. The average recognition rates are displayed left of Fig. 4. The distribution was normal for all the conditions except for 0 ms. Therefore, a Friedman’s ANOVA was conducted and revealed that the recognition rates were significantly different between the timing conditions, \(\chi ^2(3)\) = 13.07, p < .05 (\(M_{0}\) = 49.58 or 82.64%, \(SE_{0}\) = 6.1, \(M_{100}\) = 51.83 or 86.39 %, \(SE_{100}\) = 4.78, \(M_{200}\) = 54.75 or 91.25%, \(SE_{200}\) = 3.52, \(M_{300}\) = 55 or 91.67%, \(SE_{300}\) = 4.59). Post hoc tests were conducted based on the following inequality  [3]: \(\left| \overline{R_{u}} - \overline{R_{v}} \right| \ge z_{\alpha /k(k-1)} \sqrt{k(k+1)/6N}\) (Eq. 1), with \(R_{u,v}\) the mean rank of a group, z the statistic from the table of the standard normal distribution, k the number of conditions and N the total sample size. We computed the critical difference (right side of Eq. 1) as being equal to 1.39 with a z value of 2.64. We then calculated the differences between the mean ranks of the groups with the most likely significant differences, i.e. 0–200, 0–300, 100–200 and 100–300. The inequality of Eq. 1 indicates that if the differences between mean ranks is greater than or equal to the critical difference, then that difference is significant. In this case, the pairs 0–200 (\(\left| 1.67-3.13\right| \)) and 0–300 (\(\left| 1.67-3.17\right| \)) have values of 1.45 and 1.5, greater than 1.39, thus their difference is significant. Overall, the results indicate that participants recognised well patterns made of two distinct stimuli on the surface using the IFM, even simultaneous ones (82.64% recognition, chance level at 17%). However, to further confirm this hypothesis, studies involving mixed stimuli on one to several fingers need to be conducted as the knowledge of having only two stimuli could have biased the conditions with a lower temporal difference by guessing. There were no major differences between participants judged as haptic experts and non-experts, if anything, non-experts had higher scores than experts. As expected, participants performed better at longer timing differences with a significant difference between 0 and 200 ms onwards.

For further analysis of the participants’ performance, we computed confusion matrices for each of the conditions (see Fig. 5). They show that the pairs ‘12’ and ‘34’ are nearly always recognised, no matter the timing difference between stimuli. Most of the confusion happened when involving pairs of fingers from different hands. In particular, for 0 and 100 ms, the pairs with the lowest scores were ‘13’, ‘14’ and ‘24’ and the confusion most often happened with one adjacent finger. Further analysis of the direction of the stimuli could inform us whether it had any effect on the confusions. Preliminary analysis of the results of amplitude differences between patterns did not indicate any notable impact on the perception. For 0 ms, the amplitude varied on average between 2.95 (patterns ‘13’ and ‘24’) and 4.16 \(\upmu \)m (pattern ‘23’), for 100 ms between 2.75 (pattern ‘24’) and 3.72 (pattern ‘23’), for 200 ms between 2.69 (patterns ‘13’ and ‘14’) and 3.75 (pattern ‘23’) and for 300 ms between 2.78 (patterns ‘12’ and ‘13’) and 3.8 (pattern ‘23’). There was no clear correlation between this difference in amplitude and the recognition rates, as for instance ‘23’ always had the highest amplitudes, but not the highest recognition rates. For 100 ms the lowest recognition was for ‘13’ and yet had an amplitude of 3.06 \(\upmu \)m. Further analysis will be conducted to assess the impact of the different amplitudes.

Fig. 5.
figure 5

Confusion matrices. From left to right: 0 ms, 100 ms, 200 ms and 300 ms

Full size image

Response Times. The average response times are displayed right of Fig. 4. The distribution was normal for all the conditions except for the 300 ms condition. Therefore, a Friedman’s ANOVA was conducted and revealed that the response times did not significantly change between the different timing conditions, \(\chi ^2(3)\) = 3.6, p > .05 (\(M_{0}\) = 2.06, \(SE_{0}\) = 0.57, \(M_{100}\) = 2.05, \(SE_{100}\) = 0.37, \(M_{200}\) = 2.03, \(SE_{200}\) = 0.41, \(M_{300}\) = 2.07, \(SE_{300}\) = 0.37). This shows that participants had no particular difficulty according to the timing difference, even with simultaneous stimuli, which is confirmed by the relatively high recognition rates.

Qualitative Feedback. During the study, after each session, participants were asked to rate the difficulty of discrimination for each condition. 0 ms obtained an average score of 6.42 out of 10, 100 ms a score of 3.63, 200 ms a score of 3.36 and 300 ms an average score of 2.33. This shows that despite good recognition rates for the 0 ms condition, participants felt less confident, some participants reported feeling a movement rather than two distinct points. This echoes the work on ‘out of the body’ phantom sensations on a surface [10] and warrants further exploration. On the contrary, the 300 ms condition was deemed the less difficult as participants could clearly feel and distinguish the two stimuli. From a pattern recognition point of view, this is an interesting result as the 0 ms could produce patterns or textures that need to be perceived as continuous movements, whereas timing delays above 200 ms could be used to ensure the discrimination of several points. In concordance with the perceived difficulty ratings, 7 participants preferred the 300 ms condition as they were more confident about the perception, with stimuli well separated, whereas 5 participants preferred the 100 ms where the stimuli were still well perceived and the rhythm was faster. Some participants reported perceiving different intensities on their fingers in a trial, though not consistently. This could be explained by the uncontrolled amplitude of the setup, though the difference in amplitude of the stimuli were constant for a given pattern in a trial, which contradicts user perceptions.

4 Conclusion

This paper reported the results of an initial user study on 2-point based pattern recognition on a surface using the Inverse Filter Method, with different timing differences between stimuli. The results are promising as participants could discriminate rather well the different patterns with averaged rates of 83% for simultaneous stimuli and up to 92% for stimuli separated by 300 ms, without any significant differences in response times. The sensations reported varied between a fast movement to two clearly distinct points depending on the timing difference, thus opening up possibilities for rich patterns. A lot of data remains to be analysed to assess the impact of stimuli direction and uncontrolled amplitudes on the observed confusions. This initial study also opens up many future leads for experiments by evaluating 3 to more actuated fingers at a time, perceptual illusions and for future design of patterns using this method.