1 Introduction

Vibro-tactile feedback provides an inexpensive and effective way to enhance the interaction experience. Smart phones, perhaps the most popular handheld computing devices today, provide vibro-tactile feedback using a single vibration actuator embedded within the phone, and one way to further enrich the tactile feedback is to employ multiple actuators. However, such an approach can incur significant costs and can cause problems in packaging and usability. An alternative might be to use just a few (2–4) actuators and instead to rely on illusory tactile perception.

Funneling and saltation are two major examples of such tactile illusion. Funneling and/or saltation refer to illusory tactile sensations occurring “away” from and between two places on the continuous skin on which the actual vibratory stimulations are given either simultaneously or with time intervals. The intended location of the phantom sensation (ITL) can be changed by modulating the intensity (funneling) or the inter-stimulus time interval (saltation). In fact, funneling and saltation have been applied to reduce the number of tactile actuators needed for tactile interaction design [13, 25].

Recently, researchers have discovered such phantom sensations can be extended to “out of the body” sensations [19] (see Fig. 1), thus making it possible to generate phantom tactile sensations as if they were coming from an external handheld object (which thus extends out of or is connected to the body). This result is appealing in its potential application in mobile devices. For instance, it might be feasible to make a user perceive phantom tactile sensations emanating from the handheld device in a form that is richer than a single actuator scheme (e.g., middle of the display), but supplying actual vibrations only to the “natural” holding locations.

Fig. 1
figure 1

Concept of “out of the body” tactile experience from a handheld object. When vibratory stimulations are given to the fingertips with proper parameters, phantom tactile sensations can be felt as they are occurring in the middle of the handheld object

Full size image

However, while basic studies regarding the “out of the body” experience had been carried out by Miyazaki et al. [19], the same effect has not yet been verified to properly function in an actual application for an everyday mobile device. In the Miyazaki’s original experiment, a 10-cm-long metal ruler placed between two fingertips was used as an external object. In a scenario of a mobile device application, the device corresponding to the external object would be heavier and would be shaped differently. It would be held or gripped (rather than simply placed) at points that are not necessarily at the fingertips. Thus, we believe a reaffirmation of the illusory phenomenon is necessary.

Moreover, to be useful for mobile devices, it would be desirable to extend the phantom sensation phenomenon to 2D space (e.g., mapped to the mobile display). If verified, the results can serve as a basis to realize more flexible and less expensive tactile/haptic feedback for mobile interfaces, without using too many vibrators or actuators.

Accordingly, in this paper we conduct and discuss results from the following three experiments: (1) to validate the “out of the body” illusory experience using an actual mobile device (i.e., smart phone), (2) to assess the effects of our newly proposed 2D extension to the original 1D “out of the body” phantom sensation, and (3) to apply “out of the body” funneling to a real-world application and evaluate the user experience compared to that of a conventional single vibrator-based reference system.

In the next section, we first review prior research literature related to phantom tactile sensations and application in practical interaction design for mobile devices. Then, we describe the three validation experiments and report their results. Finally, we conclude the paper with a discussion and directions for future research.

2 Related work

Funneling and saltation are the two major perceptual illusion techniques specific to vibro-tactile feedback. Funneling refers to simultaneous stimulation of the skin at two different locations along a contiguous length of the body, such as along an arm, with different amplitudes, eliciting phantom sensations between the two (i.e., 1D phenomenon) [4]. Several researchers have applied this phenomenon in human interfaces, experimenting with different ways of modulating the vibration amplitudes for detailed controlling of the intended target phantom sensation locations [1, 3]. Mizukami has used funneling and apparent motion to generate phantom sensations in 2D and has applied the results to tactile character recognition [20]. In their work, a sparse tactile array grid was used by controlling the relative amplitudes between pairs of actuators at the grid points. It was possible to elicit illusory perception of the character stroke, that is, 1D funneling was applied between pairs of vibrators, but this required an area of the body to be in full contact with the 2D vibrator grid. Ooka et al. used funneling by placing a 2 × 2 grid of vibrating micro-pins on the fingertips to simulate and display a sense of force feedback for virtual object manipulation. Their work demonstrated that it was possible to induce a sense of mass, slippage, and rotational torque in object grasping actions [22].

In saltation, the skin is stimulated at two locations with proper inter-stimulus intervals (ISIs) and the phantom sensation of directional “hopping” is felt in between (also originally discovered as a 1D phenomenon) [10]. In the usual form, two weak, timed stimulations are given at one skin location, and then a stronger one is given at the other. Saltation has been further investigated by other scientists in terms of the effects with different ISI values [7–9, 24, 28]. Note that saltation produces a dynamic effect (e.g., directional hopping), while the phantom sensation from funneling is more localized. However, it is also possible to make saltation produce a localized phantom sensation with proper ISI.

In a manner similar to funneling, 1D saltation has been applied to user interfaces. For instance, Hoggan et al. [13] experimented by using three vibrators on a mobile device to emulate a tactile progress indicator bar. The moving progress indicator metaphorically matched the saltation’s hopping effect. Tan applied saltation to an implementation in a tactile chair using a 3 × 3 tactile array for pattern recognition [29]. The saltation effect made it possible to generate a sensation of a smooth gesture stroke. Similar work was done by Israr and Poupyrev [14] and Seo and Choi [26]. Again, note that even though the tactile array had a 2D structure, it consisted of 1D saltation between pairs of vibrators. Moreover, an area of the body had to be in full contact with the vibrator array.

Recently, Miyazaki has discovered that saltation can be extended to body-worn (e.g., handheld) objects and can create an “out of the body” tactile experience [19]. In their experiment, a small (~10 cm) metal ruler was placed between two index fingers (one from the left hand and the other from the right) where saltation-based vibratory stimulations were given at the two fingertips. Subjects perceived a phantom sensation as if it were coming from the middle of the ruler, i.e., out of the subjects’ body. However, subjects were not able to sense it as such when the ruler was not present. It is projected that this peculiar phenomenon is related to the concept of the body map (homunculus), and the ruler “extended” and virtually became part of the body map.

This form of illusion seemingly has a practical application for handheld mobile interaction. Mobile devices are often held by two hands (on the far two ends of the device), and it might be equally possible to generate tactile feedback to be felt for events or objects presented on the device without direct contact by using only a limited number of actuators (See Fig. 2).

Fig. 2
figure 2

a 2D tactile grid (and the hand in full contact with the array) versus b using the “out of the body” effect. In the latter, the feedback from the middle of the screen is felt despite not being in direct contact with the interaction location. The number of vibrators can be reduced, making the interaction more convenient

Full size image

Another illusion similar to saltation is called apparent motion, in which stimulations appear to or feel to move between two positions (1D) when the amplitudes of the stimulations are temporally modulated (like a dynamic blending effect in visual animation) [14, 20, 23, 26]. Although not often stated explicitly, many tactile pattern systems (e.g., those mentioned above) actually use this phenomenon. For example, SemFeel [32] was a vibro-tactile feedback system for a mobile touch screen device using five vibrators attached to the backside of a device (four corners and one center). It was capable of producing ten different perceptible vibration patterns for expressing richer semantic information, compared to that of the usual single vibrator, and was based on apparent motion.

Finally, many human computer interfaces also rely on cross-modal illusions to enrich the user experience by fusing the feedback of different modalities. For instance, a mere single vibrator-based tactile feedback, when combined with other modalities like visual and aural, can induce a phantom directional haptic feedback and will enrich the user experience [21]. Generally, multimodal feedback is also known to improve task performance, as shown in the survey and meta-study of multimodal feedback systems conducted by Burke et al. [6]. Multimodal interfaces can even alter perceptions across the modalities. The McGurk effect [18] is one of the most famous of such phenomena in which the visual information a person gets from seeing a person speak changes the way they hear the sound. This effect is often put into use in interfaces for people with perceptual disabilities [33].

Thus, to summarize, the distinguishing point of our work is to newly extend “out of the body” funneling and saltation to 2D mobile interaction and to assess the resulting user experience. Most previous applications of tactile illusion to human interfaces have been developed within the context of realizing more flexible tactile array displays, and they suffer from inherent usability problems of having a large constant contact area. Our focus is more in reducing the number of actuators by taking advantage of the illusion and multimodal fusion effects. We are also particularly interested in whether the “out of the body” tactile illusion scheme can significantly improve the user experience relative to the usual single vibrator scheme.

3 Experiment I: “Out of the body” funneling and saltation on mobile device

Before extending funneling or saltation to 2D, Experiment I was first conducted to validate whether the same 1D “out of the body” phenomenon still applies when a mobile device is used as the body extending object instead of a simple handheld object like the ruler used in Miyazaki’s original experiment [19]. For one, the mobile device is different from the ruler in size (117 × 61 × 15 mm vs. 8 cm in length placed above the two fingertips), shape (rectangular, 3.5-inch screen), weight (~120 g), and in the way it is held. Thus, the contact points at which stimulations are applied may not necessarily be the fingertips. Furthermore, this validation experiment is regarded to be a prerequisite to the next step which is the extension of the “out of the body” phenomenon to 2D using a mobile device. Experiment I was carried out separately in two parts: one to validate funneling and the other to validate saltation. Both experiments mostly use the same overall experimental design, setup, and procedures. The differing detailed procedures are explained subsequently.

3.1 Experimental design and setup: Parts A and B

In these experiments, the user experienced two types of stimulation: (1) nominal 1D tactile array using no phantom sensations such as funneling or saltation, and (2) funneling (Part A) saltation based (Part B) as shown in Fig. 3 and Fig. 4. A mobile device was attached with five (Part A) or three vibrators (Part B) on its back in line with equal spacing. The two vibrators on each end of the line where placed at convenient positions to be in contact with the holding hands/fingers, and the others were placed between those two. Thus, all the vibrators (five or three) were used for the nominal stimulations, while only the vibrators on the left and right ends were used for the funneling or saltation stimulation.

Fig. 3
figure 3

Experimental setup for Experiments I-A comparing the effect of the nominal tactile array (five vibrators laid out linearly in 8 cm, ~1.8 cm apart)-based stimulation to that of funneling. Lower right buttons on the screen for collecting user response

Full size image
Fig. 4
figure 4

Experimental setup for Experiments I-B comparing the effect of the nominal tactile array (three vibrators laid out linearly in 8 cm, ~2.5 cm apart)-based stimulation to that of the saltation

Full size image

Each experiment was designed to be psychophysical in the form of an appearance-based N-alternative forced choice (N-AFC). That is, one stimulation was given either in non-funneling or in non-saltation form, and the users were to answer the position among the five (A) or three (B) vibrators at which the tactile sensation was perceived. Note that even though the response choices for Parts A and B are different in number, our purpose was only to confirm the existence of the phantom sensation occurring somewhere between the two fingertips. The details of the way the stimulations were prescribed in Parts A and B are described in the next subsections.

A common coin-type vibrator was placed on the respective fingertip and controlled in part by an Arduino board [2]. The vibrators interfaced indirectly to the smart phone experimental program through a PC, and the vibration motor used in our experiment had the same specification as that reported in by Jung and Choi [16]. The vibrators were controlled by a voltage input using a pulse width modulation signal with an amplitude between 0 and 5 V which in turn produced vibrations with frequencies between 0 and 250 Hz and associated amplitudes between 0 and 2G (measured in acceleration, or 0–18 μm in position). According to Burdea and Coiffet, Gunther et al., Jung and Choi, Seo and Choi, and Sherrick [5, 12, 16, 26, 27], these values are well above human’s normal detection threshold of about 6–45 db SL.

Our hypothesis was that there would be no statistically significant difference between the effects of non-funneling, non-saltation, and funneling- or saltation-based stimulations. In other words, the 1D phantom sensations can also be elicited on the mobile device. Our hypothesis was simply based on the observation that despite external differences, there was not much qualitative difference to the brain between holding a ruler in the fashion tested by Miyazaki et al. [19] to the usual gripping of a mobile device in terms of extending the internal body map to external objects.

3.2 Detailed procedure: Part A (funneling)

Ten paid subjects (five males, five females) participated in Experiment I-A. They had mean age of 25.2, and after collecting their basic background information, the subjects were briefed on the purpose of the experiment and were given instructions for the experimental task. A short training of 5 min was given to the subjects to get familiarized with the experimental process and the tactile device. None of the participants were aware of the funneling phenomenon. The subjects wore ear muffs to prevent any bias from the sounds of the vibration. The ear muffs were tested such that no sound could be heard in order to prevent bias in the experimental results.

Nominal stimulation was given in the following simple way: five locations were designated from L1 to L5 (Fig. 3), and to elicit a tactile feedback at the intended target location (ITL), the vibrator at that location was enacted with full amplitude (denoted “255” in Table 1) with a duration of 2 s.

Table 1 Stimulation values for the five vibrators in Experiment I-A
Full size table

For the funneling, only the two vibrators at the two ends, L1 and L5, were used. To elicit phantom sensations at various positions in between, linearly interpolated amplitudes were administered following the approach described by Alles [1]. The exact values are shown in Table 1. Note that the nominal and funneling stimulation patterns coincide when the ITL is at L1 or L5, making eight distinct tactile stimulation patterns in total.

During each session, the subjects made a total of 200 trials administered in five blocks. Thus, each block consisted of 40 trials with eight patterns delivered five times in a balanced order. The entire session took approximately 50 min. Different types of stimulations were given in a balanced order, and each stimulus was separated by a 10-s break. The subjects were asked to indicate the nearest location where the sensation was felt at among the five possible locations by touching a button on the screen (Fig. 3, lower right). Subjects were asked to select among discrete locations because the subjects expressed significant difficulties in directly indicating the perceived location in a continuous scale in our pilot study. It appeared that the perception resolution was at least in the order of centimeters, although this result was not verified experimentally. In addition, the five vibrators were already quite close to each other in the 8-cm span if they were attached as in the nominal case, given the relatively large size of the vibrators (Fig. 3).

3.3 Detailed procedure: Part B (saltation)

Ten paid subjects (five males, five females) with the mean age of 23.6 years participated in Experiment I-B. Subjects were asked to answer whether a phantom sensation was felt at or close to L2 or not. That is, L2 was the only ITL. Note again that our purpose was only to verify the “out of the body” phenomenon with saltation on a mobile device. As such, the experiment was designed as a 1-AFC yes/no style.

Nominal stimulation was given in the following way: the first vibrator on the right end (L1 at T1) was activated, followed by the one in the middle (L2 at T2), and then the left (L3 at T3). The ISI varied at six different levels (T1–T2: 800 ms × T2–T3: 80, 100, 120, 150, 200, 300 ms). The ISIs were chosen around values that were known from prior literature to be successful in inducing saltation [10, 19, 23, 24]. The stimulation was given in two directions, half starting from the right and vice versa.

The saltation stimulation was given in a similar way except that the second stimulation was delivered at location L1 (or L3 when given from right to left). An example is given in Table 2.

Table 2 Example of right (L1) to left (L3) stimulation pattern example with T1–T2 = 800 ms and T2–T3 = 80 ms for non-saltation-based and saltation-based stimulations
Full size table

During the actual sessions, each subject was asked to perform a total of 96 trials for about 50 min with two types of feedback given with respect to two directions, six different ISIs, and four repetitions. Different types of stimulations were given in a balanced order, and each stimulus was set apart by a 10-s break. The subjects were asked to indicate where the sensation was felt among the three locations (L1, L2, or L3). However, we were only interested in whether there was a phantom sensation from L2 with comparable strength to that in the corresponding nominal case.

3.4 Results

The main purpose of our work was to validate whether there was no significant perceptual difference between saltation and funneling-driven phantom sensations and actual sensations. For this reason, Chi-square analysis and ANOVA were applied rather than signal detection theoretic (SDT) analysis. We felt SDT to be more appropriate to derive results such as “Just Noticeable Difference” and amplitude perception. ANOVA was more convenient yet still valid in comparing effects of these two types of stimulations across different stimulation intervals.

3.5 Part A: Out of the body funneling on mobile device

Figure 5 shows comparative user performance between the cases of nominal and funneling stimulation. A score was given according to how far the response was from the ITL with 20 pointsFootnote 1 being a 100 % match; 10 points1 were deducted for a one-interval difference. For instance, if P2 was the ITL, and the subject responded that the sensation was felt from the same location, 20 points were given. If P4 was the ITL, but the subject answered the sensation location to be at P3 or P2, only 10 or 0 points were given, respectively. Then, the score was normalized to a scale of 100. Such a scoring scheme was used to compensate for bias possibly introduced by the presence of a discrete selection (vs. designation in a continuous scale).

Fig. 5
figure 5

User performance with 1D funneling versus nominal stimulation. There is no statistically significant difference, thus confirming the existence of the out of the body funneling on the mobile device

Full size image

Interestingly, funneling tended to produce higher performance than nominal stimulation. The Chi-square test had revealed that there was a significant correlation between the type of stimulus and the response results in the P2 and P3 regions, but no statistically significant difference was found according to ANOVA. Note that in the case of the nominal stimulation, the perceptions of vibrations coming from the real vibrators installed at the ITLs (e.g., middle of the screen) were indirectly felt at the grasping ends. On the other hand, in the funneling case, the vibrations were always directly coming from the grasping fingers with different amplitudes to produce the illusion that the vibration was coming from the ITLs. In other words, in the nominal treatment, the perception was real with indirect stimulation, while in the funneling treatment it was virtual and direct. Thus, it is highly plausible that the subjects with direct sensation of relative vibration amplitudes produce better localization performance since the indirect nominal stimulations get muddled through the physical medium in reaching the grasping fingers, making it more difficult to correctly assess the ITLs. At any rate, the most important result was that the results from the funneling case were not statistically different from those of nominal, strongly showing that the “out of the body” experience was possible even during typical mobile device usage.

3.6 Part B: Out of the body saltation on mobile device

Figure 6 shows the comparative user behavior between the nominal and saltation stimulations across six different ISI’s. For this experiment, we applied both the Chi-square test and ANOVA to evaluate the responses. While there was a trend of generally weaker responses for saltation, the Chi-square analysis had revealed no correlation between the type of the stimulus (nominal or saltation) and the results across all ISI’s. Similarly, ANOVA showed no statistically significant difference between the two groups across all ISI’s. Note that even in the case of the nominal stimulation, it would be difficult to produce a 100 % “appearance” response (e.g., always feeling a sensation from the middle when the middle vibrator is activated) because the stimulation is still indirect since there is no direct contact to the middle vibrator at L2.

Fig. 6
figure 6

Comparative user behavior nominal and saltation across different ISI’s. No statistically significant differences are found between the two for all ISI’s

Full size image

In effect, the subjects were not able to distinguish stimulation from the middle vibrator from that of a phantom sensation elicited without the use of the middle vibrator. Given previous studies [15, 19], these results strongly indicate the same “out of the body” tactile illusion can successfully be applied to an actual mobile device usage situation with two hands.

4 Experiment II: Extending “out of the body” funneling and saltation to 2D

4.1 Experimental design and setup

For Experiment II, we used five vibrators attached to four rear corners, where the fingers would be touching in a normal two-handed grip, and the middle of the mobile device (Fig. 7). Three types of stimulations were tested with the objective of inducing indirect sensation emanating from the 2D display/device space: (1) single in which only one vibrator (in our case, the middle one) was used, (2) funneling in which simultaneous vibrations were generated from the four corner vibrators with interpolated amplitudes, and (3) saltation in which timed vibrations were generated from the four corner vibrators.

Fig. 7
figure 7

Experimental setup for Experiment II comparing the effects of single-, funneling-, and saltation-based stimulations in 2D. Numbers indicate the timing order of the stimulation, and “S” indicates the intended target-sensed location (ITL) by the user

Full size image

The experiment was also designed as a psychophysical one in the form of a two-alternative forced choice (2AFC). The stimulations were given in one of the three types: either to elicit real (single) or phantom (funneling or saltation) sensations. The display space was divided into 15 (5 × 3) grid sections,Footnote 2 and the subject was to make a response to the stimulation as shown in Fig. 8. After experiencing a stimulus, two markings appeared at two grid locations. One was at the ITL of the real or the phantom sensation, and the other was at a location away from the first one. The subject was to tap on the display to make his or her response, selecting which location marker felt closer to the sensed location. For the sake of argument, assuming that a correct answer meant choosing the marking that coincides with the ITL, the subject had 50 % probability of getting the correct answer, a baseline performance in a 1AFC experiment [11, 17]. Note that in reality, there is no notion of “correctness” because none of the three stimulation types guarantee the fine control of the real/phantom sensation locations. We use the term “pseudo-correct” answers to reflect this aspect.

Fig. 8
figure 8

Making a response to a stimulus. a A stimulation is given with an ITL for phantom sensation (indicated with an ‘S’). b The user is presented with two markings in the 5 × 3 grid and is to choose between the two. One marking coincides with the ‘S’ location (pseudo-correct answer) and the other reasonable deviated from it

Full size image

There is an operational reason why the experiment was designed as a binary selection task instead of having the subject directly indicates their perceived source location of the tactile feedback over the 2D screen. One important purpose of our experiment was not only to assess the feasibility of 2D extension, but also to compare its effect to the case where a single vibrator is used. When a subject was given a stimulation with a single vibrator, we wanted to ascertain that the user would perceive it as coming from or near the middle location of the vibrator (at least in most cases). We felt that if the subject was left to simply indicate the source location without any visual feedback, such a naturally expected behavior would be corrupted by uncontrollable and difficult to identify external biases (a pilot task indeed showed such subject behavior). Moreover, in most cases of a real application, tactile feedback will be accompanied with corresponding visual effects. Thus, the task was framed as a forced selection task, to equalize the experimental conditions and processes. The same task was used for the two other treatment cases, and in a manner similar to that of Experiment I, the space was discretized for subjects’ convenience and perception resolution was observed. The details of the way in which the stimulations were prescribed are described in the next subsection. Otherwise, the same vibrator and system setup as in Experiment I were used.

4.2 Detailed procedure

Ten paid subjects (five males, five females) with the mean age of 24.5 participated in the experiment. After collecting their basic background information, the subjects were briefed on the purpose of the experiment and were given instructions for the experimental task. A short training was given for the subjects to become familiarized with the experimental process. None of the participants were aware of the funneling or saltation phenomena. The subjects wore ear muffs to prevent any bias from the sounds of the vibration.

A single stimulation was given by activating the middle vibrator. The intention was to elicit sensations from a 2D position with only one vibrator similarly to how current handheld devices operate. Obviously, it was expected that it would be difficult to control the location of the phantom sensation in 2D space with only one fixed vibrator.

The funneling stimulation was given by activating four vibrators simultaneously with linearly interpolated amplitudes according to the ITL of the phantom sensation. Since the display space was divided into a 5 × 3 grid, there were mainly two cases of stimulus prescription. When the ITL was on the boundary (12 regions except for B2, C2, and D2 in Fig. 9), only two vibrators in the corresponding row or column were used. For instance, for D1 in Fig. 9a, the top two vibrators were used and their amplitudes were interpolated linearly in 1D. As for the three remaining regions, all four vibrators were used as shown in Fig. 9b. The vibration amplitudes of the right/left set of vibrators were determined as explained above, but since these locations are vertically in the middle, the amplitudes were mirrored in the vertical dimension. Each stimulation lasted for 80 ms. Previous studies have used durations of 40–50 ms. However, we have adjusted this value based on user feedback from our pilot tests.

Fig. 9
figure 9

Funneling stimulations for the 5 × 3 grid display. a When the ITL is on the boundary, only two vibrators are used with interpolated amplitudes. b For locations B2, C2, and D2, all four are used

Full size image

Similarly, for saltation, different ISI’s and vibration patterns were used to create ITLs at the 15 grid positions as summarized in Table 3 and illustrated with examples in Fig. 10. For instance to elicit a sensation in the corner (e.g., upper left corner, A1), timed stimulation was delivered from two directions from the right and bottom toward the A1 with T2–T3 set to 50 ms (A3 → A1, E1 → A1). Three vibrators (A3, E1, A1) were used; 50 ms has been found to be about the lower bound for a person to feel any saltation effect [10, 19, 24]. Thus, for extreme corner positions (like A1), the T2–T3 ISI was set at this value (Fig. 10a).

Table 3 Methods of stimulation for saltation case
Full size table
Fig. 10
figure 10

Five main cases of saltation-based stimulation for eliciting sensations on the 2D space. Sensation locations are categorized in five groups according to the stimulation methods: a A1, A3, E1, E3 (four corners), b B1, B3, D1, D3, c C1, C3, A2, E2, d B2, D2, and e C2 (middle). Arrows indicate the direction of the stimulation. Dashed arrows are alternative stimulation directions

Full size image

Let’s take another example. For the B2 location, four vibrators were used since it was located in the middle row (Fig. 10d). The vibration started from the right since B2 was slightly to the left and went toward the left direction. Since B2 was not all the way to the left, the T2–T3 ISI was set to 65 ms. Timed stimulations from the E1 to A1 and E3 to A3 directions were used to create a phantom sensation at B2.

4.3 Results

As was performed in the Experiment I, both the Chi-square analysis and ANOVA were applied to analyze the results. Figure 11 shows the statistics for the existence of the “out of the body” phantom sensation effect in 2D. In particular, the graph illustrates that for the single condition, the pseudo-correct performance was at about the 50 % probability level. This is an expected result as only one vibrator is used to induce feelings for 15 different locations. Near chance performance is expected at most locations except for where the vibrator is actually located (middle, C2). As for funneling and saltation, over 75 % responses pseudo-correct responses, with collective statistically significant differences (p < 0.005 and p < 0.005, respectively), were obtained indicating a distinct psychophysically sensed event [11, 17]. In other words, subjects were, to some degree, able to sense phantom sensation as modulated by the respective funneling or saltation methods used in this experiment.

Fig. 11
figure 11

Subject’s performance in making pseudo-correct responses from all regions. The single condition shows close to 50 % performance, whereas funneling and saltation show about 75–80 % performance

Full size image

When analyzed in more detail with respect to the 15 individual regions, most regions followed the trend of the whole. That is, they showed a distinct and better-than-random (~75–80 % correct) performance with funneling and saltation. Figure 12 shows two examples of such results from regions A1 and B2.

Fig. 12
figure 12

Subject’s performance making pseudo-correct responses from regions A1 and B2. The single condition shows close to 50 % performance, whereas funneling and saltation show over 75–80 %

Full size image

We analyzed the pseudo-correct performance using ANOVA. As a result, except for the center regions (C1, C2, and C3), all other 12 regions showed statistically significant differences. As for the regions in the middle (C1, C2, and C3), there were no statistically significant differences found among the three treatments (Fig. 13). In addition, all three treatments showed over 75 % performance, more or less. This is in line with our expectation that since the lone vibrator used for the single condition was located in the middle, the sensation was naturally felt from there (or near there). Thus, this is a positive result in the sense that funneling and saltation exhibited non-differential behavior from this natural single sensation. If these statistics are subtracted from the whole, the figures of the overall performance would further improve (Fig. 14).

Fig. 13
figure 13

Subject’s performance in making pseudo-correct responses from region C (C1, C2, and C3). No statistically significant differences are exhibited between funneling or saltation conditions

Full size image
Fig. 14
figure 14

Subject’s performance in making pseudo-correct responses from all regions without the statistics from the middle region. The single condition shows close to 50 % performance, whereas for funneling and saltation show about 80–85 %

Full size image

The analysis is muddled in the sense that 75 % came out by adding all the figures from Fig. 13. However, this is at best an ad-hoc interpretation. Nevertheless, it is still true that there was a marked difference in performance between when the middle region was considered or not because that was the location of the middle vibrator for the single condition.

5 Experiment III: Applying 2D funneling to a real-world application

Finally, based on the results from Experiments I and II, we now investigate whether the 2D phantom sensation scheme can add to a more enriched interactive experience in an actual mobile application relative to a simple, conventional single vibrator-based feedback. Funneling was chosen to be tested mainly due to the relative simplicity in its control.

5.1 Purpose and hypothesis

We have compared the user/tactile experience of a mobile game application for when (1) no vibro-tactile feedback was given (none, the base case), (2) a single vibrator-based feedback was given (single), and (3) illusory tactile feedback with multiple vibrators was given (funneling). Again, note that in a usual game play setting, interaction events are typically presented multimodally (e.g., visual, aural and tactile), and a single vibrator scheme could very well suffice as the tactile component of the combined perceived effect.

Our hypothesis is that the proposed multiple vibrator-based illusory vibro-tactile funneling feedback would contribute to eliciting a significantly higher level of user experience compared to when only a single vibrator is used, even when presented as a component of multimodal feedback.

5.2 Experimental design and setup

The subject played a game on a mobile device (Samsung Galaxy A) attached with five vibrators on its back (the same as in Experiments I and II) at its four corners, where the fingers would normally be touching in a normal two-handed grip and in the center (Fig. 15). The center vibrator was used only for the single case. Participants were asked to hold the device freely, but with the two fingers from each hand to be lightly gripping (not too tight) around the vibrators on the four corners in the back side.

Fig. 15
figure 15

a Experimental device (mobile phone fitted with five vibrators) and b playing the “Arkanoid”-like game by tilting

Full size image

The game the subject played in the experiment was a custom version of “Arkanoid,” an old arcade game [31]. This game was chosen for its simplicity and physical interaction (e.g., breaking the stack of blocks and controlling the game by tilt motion). The game is played by controlling the lower paddle to bounce the ball to hit the bricks above and make them disappear. The game ends when the ball passes by the paddle below the screen. Note that the paddle is controlled by tilting the device (left/right) without any touch; thus, the finger grip remains natural and comfortable.

As for the single case, the vibrator is turned on and off when the ball hits the paddle or the side wall. Tactile feedback was not given when hitting the bricks because once the ball reaches the other end of the brick wall, it could bounce off too frequently and cause overly continuous vibrations. The frequency and duration of the vibration were set at 200 Hz (~1.5 G, also see later part of the section) and 140 ms, respectively, after several pilot trials and was based on figures from previous research [14, 25, 30]. Aural feedback (beep) was also given at the time of the bounce. Note that the center vibrator shown in Fig. 15 is used, rather than the one already embedded in the smart phone, to reduce any bias due to differences in the vibratory device or the way the contact is made.

The funneling stimulation was given by activating a subset of four vibrators in the corners simultaneously, with logarithmically interpolated amplitudes [1] according to the ITL of the phantom sensation. The display space was divided into a 5 × 3 grid based on the results from the Experiment II. Based on the direction of the bouncing ball, the ITL was chosen. Such a rendering was given to convey not only the indication of the “bounce” event but also its directionality afterward. For example, in Fig. 16, as the ball just bounces off the paddle and heads toward the upper diagonal direction, the direction vector is computed and quantized at the 5 × 3 resolution based on the flying orientation, and the ITL representing the ball direction is computed. Then, proper vibrators (and their relative intensities) were chosen to elicit the phantom sensation at the ITL. For example, ITLs in the upper/lower boundary are elicited with interpolated intensities of two vibrators, and the rest with four vibrators. The frequency and duration of vibration were also set at the same 200 Hz (~1.5 G) and 140 ms similarly. As was with the single case, an aural beep was also given at the time of the bounce.

Fig. 16
figure 16

Funneling stimulations on the 5 × 3 grid display. The bouncing direction is converted into a representative ITL. Two or four vibrators are used to generate the phantom sensation at the designated ITL (starry figures actual stimulations; cloud-like figures illusory sensation)

Full size image

In summary, the experiment was designed as a one-factor (three conditions) within-subject experiment. In the none condition, only visual and aural feedback were given. In the single or funneling conditions, all visual, aural, and tactile feedback were supplied. Although not the focus of this study, task performance was measured in terms of the average time the subject went on without missing the ball.

Eight survey questions were answered in a seven Likert scale asking about various aspects of the tactile and user experience, namely (Q1) event perception, (Q2) intensity perception, (Q3) direction perception (Q4) force feedback/inertial perception, (Q5) user experience, (Q6) helpfulness for task, (Q7) multimodal consistency, and (Q8) general satisfaction (Table 4).

Table 4 Detailed questionnaires of the UX survey in Experiment III
Full size table

5.3 Detailed procedure

Sixteen paid subjects (seven males, nine females) with the mean age of 26 (between 22 and 30) participated in the experiment. After collecting the subject’s basic background information, he or she was briefed on the purpose of the experiment and was given task instructions. A short training (5–10 min) was administered for game play. The subjects wore headphones through which they heard pink noise (and occasional beep sounds) to prevent any bias from the sounds of the vibration. All participants were familiar with the game “Arkanoid.”

Each subject tried out the three treatments, in a balanced order, for 5 min. They were asked to do their best not to miss the ball. If the subject missed the ball before 5 min, the game was simply restarted. After each trial, the subject was asked to fill out the survey.

5.4 Results

Figure 17 shows the responses to the eight UX survey questions. A Kruskal–Wallis test was used to evaluate the experimental results, and Mann–Whitney was used for the post hoc test. To guard against possible type 1 error, Bonferroni’s method was used for statistical adjustment. No statistical differences were observed between single and funneling for the event detection (Q1) and intensity perception (Q2), for which single vibrator scheme is deemed sufficient to provide anyway. However, responses to Q3–Q7 all exhibited statistical differences between the “Single” and “Funneling.” This goes to show that subjects did perceive the qualitative difference in the tactile feedback even though they were conveyed collectively in a multimodal fashion. The overall satisfaction level (Q8) was statistically the same between the “Single” and “Funneling” despite the possibility of the multi-vibrator scheme and feedback variety to cause fatigue. As for the quantitative performance, it took on average 54.9, 58, and 54.7 s until missing the ball for the “None,” “Single,” and “Funneling” condition, respectively. No statistically significant differences were found according to ANOVA (Fig. 18).

Fig. 17
figure 17

Results for the UX survey questionnaire. The lines between the bars indicate statistically significant differences with p values less than 0.05 (we omit the detailed statistics)

Full size image
Fig. 18
figure 18

Results of the quantitative performance (e.g., time taken until a ball is missed) among the three treatments. No significant statistical differences are found

Full size image

Post-briefing revealed that subjects generally felt the vibration feedback to be clearly helpful for improving the user experience, immersion, and even task performance. Most subjects also appreciated the novel and illusory vibro-tactile sensation as distinct from single vibrator feedback. The combined modality feedback seemed to express the more dynamic nature of the interaction and have improved the overall user experience. As for the task performance (e.g., time the subject could go on without missing the ball), no statistical differences were found between the “Single” and “Funneling” cases. As the subjects indicated, the directionality was not sufficiently helpful for the task. It seems, compared to the Single case, the added effects of illusory tactile feedback improved the experience but not necessarily the performance because regardless of the tactile feedback type, the visual feedback provided sufficient guidance for reasonable performance (at least for this particular task). More experiments and research are needed to exactly characterize the “added” effects to assess its wider and general applicability and benefits to cost.

6 Discussion and conclusion

In this paper, we presented three experiments in which the “out of the body” phantom sensation effects (saltation and funneling) were verified to exist with actual handheld media devices and, furthermore, were shown for the first time that it could be extended to 2D. Both saltation and funneling techniques exhibited similar effects in the 2D extension of the original 1D “out of the body” effect. In our experiment, the extent of the controllable resolution for the phantom sensations was about 5 × 3 for a 3.5-inch-sized 2D display. We applied the 2D funneling effect to an actual mobile game and showed the user experience to significantly improve over single vibrator-based feedback in several categories, such as phantom direction/inertia perception, helpfulness, and user experience. Thus, we have shown the distinct possibility of upgrading the user/tactile experience through the use of tactile illusion with only a small number of vibrators. Even though the simple single vibrator scheme relies on the multimodal effect of the associated visual and aural cues, it still has limitations to the depth and type of tactile experience it can convey. The proposed scheme is cost-effective because it only requires a relatively small number of vibrators (not more than four), while other non-illusory approaches need at least five or more depending on the vibration patterns they intend to generate. However, the rendering method for illusory feedback needs to be more reflective of the task to maximize its effects. Thus, we plan to explore other vibro-tactile illusory rendering methods and their effects.

To be useful in practical applications, more research is needed in finding the way to more precisely modulate the location of the phantom sensation, and testing is also needed with other situations and a larger number of subjects. We also expect the phantom tactile sensations be even more effective when combined with other modalities.