Abstract
We tested whether auditory sequences of beeps can modulate the tactile perception of sequences of taps (two to four taps per sequence) delivered to the index fingertip. In the first experiment, the auditory and tactile sequences were presented simultaneously. The number of beeps delivered in the auditory sequence were either the same as, less than, or more than the number of taps of the simultaneously presented tactile sequence. Though task-irrelevant (subjects were instructed to focus on the tactile stimuli), the auditory stimuli systematically modulated subjects’ tactile perception; in other words subjects’ responses depended significantly on the number of delivered beeps. Such modulation only occurred when the auditory and tactile stimuli were similar enough. In the second experiment, we tested whether the automatic auditory-tactile integration depends on simultaneity or whether a bias can be evoked when the auditory and tactile sequence are presented in temporal asynchrony. Audition significantly modulated tactile perception when the stimuli were presented simultaneously but this effect gradually disappeared when a temporal asynchrony was introduced between auditory and tactile stimuli. These results show that when provided with auditory and tactile sensory signals that are likely to be generated by the same stimulus, the central nervous system (CNS) tends to automatically integrate these signals.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Our everyday interactions with the environment provide us with continuous stimulation of our various sensory channels, each of which allows the transduction of specific physical signals (such as photons on the retina, mechanical pressure on tactile sensors) into electrical signals. This means that the central nervous system (CNS) has to deal with a pool of multimodal signals that provides information on various aspects of the body/environment relationship. In many cases, the occurrence of a specific value of the signal in one sensory modality is accompanied by a corresponding specific signal in one or more other modes. For instance, when knocking on a door, one receives congruent visual, tactile and auditory feedback. However, these congruent signals are usually part of a whole pool that does not exclusively contain signals related to the same single event or stimulus. For instance, while knocking on the door, one might hear the voice of a neighbor talking. To derive a coherent and unified percept, the CNS has to sort the pool of incoming signals and combine the ones that are likely to be generated by the same stimulus. In the present study, we investigated the integration of auditory and tactile signals for the perception of sequences of tactile events.
Shams et al. (2000, 2002) recently showed that when a single visual flash is accompanied by multiple task-irrelevant auditory beeps, the single flash is perceived as multiple flashes. Hötting and Röder (2004) reproduced this experiment in the audio-tactile domain and found that one tactile tap accompanied by multiple auditory tones is perceived as more than a single tap. Here, we investigated whether auditory stimuli can also bias the perception of sequences of multiple tactile taps. More specifically, we tested whether the perception of tactile sequences of two to four taps delivered to the index fingertip can be modulated by simultaneously presented sequences of auditory beeps when the number of beeps differs (less or more) from the number of taps. This design allowed us to systematically test whether auditory stimuli can increase and/or decrease the perceived number of taps. We were therefore able to determine whether task-irrelevant auditory signals can really modulate (influence in both directions) the perception of tactile taps, or whether the results of Hötting and Röder (2004) merely reflected an original but very specific illusion.
In the first experiment, the auditory and tactile sequences were always presented simultaneously. In all but one auditory condition, the beep sequences were similar to the tactile sequences. A control condition where the beep duration was obviously too long to correspond to the brief taps allowed us to determine whether auditory-tactile combination can also occur with clearly dissimilar auditory and tactile stimuli. In the second experiment, we tested whether the auditory and tactile stimuli are integrated when the timing between auditory and tactile sequences is manipulated. The sequences of beeps and taps were always similar, but for some timing conditions they were presented in temporal asynchrony.
Experiment 1
Methods
Subjects
16 right-handed subjects (aged 18–40 years) participated in the experiment. None of the subjects had a history of sensorimotor or auditory disorder, and they gave their informed consent before taking part in the experiment, which was performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.
Experimental set-up
The experimental set-up is schematically represented in Fig. 1A. The subjects were seated with their right forearm and hand resting palm-up at belly level on a table (72 cm high). A PHANTOM (SensAble Technologies) force-feedback device fixed to the table was used to generate the tactile stimuli (taps of 1 N indenting subjects’ skin by approximately 2 mm) via a metallic pin of 1 mm in diameter. A curtain-like piece of black cloth located in-between the table and the subjects prevented them from seeing their hand or the force-feedback device. For the whole duration of the experiment, subjects wore earphones emitting white-noise (71 dB) to mask any external auditory disturbance. The earphones were also used to present the auditory stimuli (beeps, 790 Hz, 74 dB).
A Experimental set-up. B Temporal profiles of the auditory and tactile stimuli in Experiment 1. The delay between two successive beeps in the auditory sequence was adjusted so that the onsets of the first and last beeps coincided with the respective onsets of the first and last taps
Procedure
For each trial, a sequence of two to four taps was delivered to the subjects’ index fingertips. The subjects’ task was to report how many taps they felt. No visual or direct auditory feedback about the delivered taps was provided. Each tap lasted 20 ms and the delay between the onsets of two successive taps was 135 ms. These parameters were set in a pilot study so that subjects can estimate the number of delivered taps below a ceiling level of 100% of correct answers (85%, 74% and 61% of correct answers for the two-, three- and four-taps sequences, respectively). Indeed, presenting tactile stimuli that are too “salient” would have dramatically reduced the probability of getting an auditory-evoked bias (Ernst and Banks 2002, Alais and Burr 2004, Ernst and Bülthoff 2004). In the experiment, five different auditory conditions were associated with the tactile sequences. Subjects were explicitly instructed that the auditory stimuli did not relate to the tactile stimuli. In the “No Beep” auditory condition, no auditory signal (apart from the white noise) was presented during the tactile sequence. This condition established baseline performance for identifying the number of taps for the different tactile sequences. For the “One Beep Less”, “Same Number” and “One Beep More” auditory conditions, the number of beeps was one less (number of beeps equals the number of taps minus one), the same number (number of beeps equals the number of taps) or one more (number of beeps equals the number of taps plus one) than the number of taps of the associated tactile sequence, respectively. These conditions tested the influence of task-irrelevant auditory stimuli on subjects’ perception of tactile taps. The beeps and tactile sequences were always presented simultaneously. Each beep lasted 50 ms and the delay separating the onsets of two successive beeps varied so that the onsets of the first and last beeps coincided with the respective onsets of the first and last taps (see Fig. 1B). For each trial in which beeps were presented, the inter-beeps delay in milliseconds (Δo) was defined according to the following formula:
For the “Control Beep” auditory condition, a unique long beep was presented at the same time as the tactile sequence. This auditory condition was designed to control for a possible non-specific influence of the auditory stimulation on the identification of tactile sequences. Three different durations (270, 405 and 540 ms) were used for the control beep, to control for any influence of beep duration on tactile perception. In any case, the duration of the control beep made the auditory signal obviously dissimilar to the presented tactile input.
On the whole, the experiment lasted about 20 minutes and consisted of fifteen different experimental conditions, combining three tactile conditions (sequences of two, three and four taps) with five auditory conditions (“No Beep”, “One Beep Less”, “Same Number”, “One Beep More”, “Control Beep”). Subjects performed twelve trials per experimental condition, for a total of 180 trials. For each of the three experimental conditions involving the control beep, the twelve trials actually consisted of three subsets of four trials (four trials for each duration). All fifteen experimental conditions were intermixed and the trials presented in a random order. After each trial, verbal responses reporting the number of perceived taps were given by the subjects.
Results
The average perceived numbers of taps in the different auditory conditions are presented in Fig. 2. To determine whether the auditory stimuli altered the perceived number of taps, the individual averages were entered in a 5×3 [number of auditory conditions (“No Beep”, “One Beep Less”, “Same Number”, “One Beep More”, “Control Beep”) × variations on number of delivered taps (two, three, four)] repeated-measures analysis of variance (ANOVA). When necessary, post hoc comparisons using a Sidak adjustment for multiple comparisons (p<0.05) were then performed. The auditory condition significantly influenced the perceived number of taps [F(4,60)=24.53, p<0.001]. Indeed, the perceived number of taps was significantly higher for the “One Beep More” than for any other auditory condition (Fig. 2A), and lower for the “One Beep Less” than for all but the “No Beep” condition (p=0.06). This auditory-evoked modulation was consistent across the different tactile conditions, the perceived number of taps always being significantly lower in the “One Beep Less” than in the “One Beep More” auditory condition. On the other hand, the “No Beep”, “Same Number” and “Control Beep” conditions did not statistically differ from one another (Fig. 2B). The number of taps delivered also influenced the perceived number of taps [F(2,30)=448.11, p<0.001]. All conditions were significantly different from each other; the higher the number of taps actually presented, the higher the number of taps perceived by the subjects.
Number of perceived taps as a function of both the actual number of delivered taps and the auditory condition. The error bars represent between-subjects standard errors. A When the number of beeps of the auditory sequence was one less than the number of taps of the simultaneously presented tactile sequence (“One Beep Less”), the subjects perceived significantly less taps. Similarly, presenting more beeps in the auditory than in the tactile sequence (“One Beep More”) significantly increased the perceived number of taps. B When the auditory sequence consisted either of a unique long beep (“Control Beep”) or of a number of beeps similar to the number of taps in the tactile sequence (‘Same Number’), the perceived number of taps was not different to when only the tactile sequence was presented (“No Beep”)
We then tested whether providing subjects with redundant auditory and tactile signals rather than tactile signals alone reduced the variability of the estimates. The individual standard deviations were entered in a 2×3 [number of auditory conditions (“No Beep”, “Same Number”) × variations on number of delivered taps (two, three, four)] repeated-measures ANOVA. The responses of the subjects were significantly less variable when redundant tactile and auditory signals were presented (“Same Number”) than with tactile signals alone (“No Beep”) [F(1,15)=11.96, p<0.01].
To test whether the duration of the control beep had an influence on the perceived number of taps, the data from the “Control Beep” condition were submitted to a 3×3 [variations on control beep duration in milliseconds (270, 405, 540) × variations on number of delivered taps (two, three, four)] repeated-measures ANOVA. This analysis revealed no effect of the duration of the control beep on the perceived number of taps.
Finally, to determine whether the observed auditory influence on tactile perception was consistent across subjects, the perceived number of taps was analyzed for each subject separately using a 5×3 [number of auditory conditions (“No Beep”, “One Beep Less”, “Same Number”, “One Beep More”, “Control Beep”) × variations on number of delivered taps (two, three, four)] repeated measures ANOVA in each case. Figure 3 presents the average perceived number of taps in the different auditory conditions for six representative subjects. The perceived number of taps was significantly altered by the auditory condition for 13 out of 16 subjects, confirming the between-subjects consistency of the auditory-evoked modulation of tactile perception.
Number of perceived taps as a function of both the actual number of delivered taps and the auditory condition for six representative subjects. The perceived number of taps was significantly lower in the “One Beep Less” than in the “One Beep More” auditory condition. This was true for 13 out of the 16 subjects that took part in the experiment
Discussion
When required to estimate the number of tactile taps delivered on the index fingertip, subjects were significantly influenced by the simultaneous presentation of task-irrelevant auditory stimuli. Indeed, the perceived number of taps not only depended on the actual number of delivered taps, but also on the number of simultaneously presented auditory beeps. This auditory modulation of tactile tap perception proved to be rather robust since subjects’ performance was quite consistent and the effect was observed for all tactile conditions. These results are in line with previous findings showing that for non-spatial tasks, task-irrelevant auditory stimuli can bias visual (Bermant and Welch 1976, Bertelson and Radeau 1981, Fendrich and Corballis 2001, Morein-Zamir et al. 2003, Shams et al. 2000, 2002) and tactile perceptual estimates (Guest et al. 2002, Jousmäki and Hari 1998, DiFranco et al. 1997, Hötting and Röder 2004). More specifically, our results can be paralleled with the ones obtained by Shams et al. (2000, 2002) in the audio-visual domain, and Hötting and Röder (2004) in the audio-tactile domain. These authors showed that single visual flashes or single tactile taps are perceived as multiple flashes or taps when multiple auditory beeps are presented simultaneously. In these experiments, however, auditory stimuli only biased single events (one visual flash or one tactile tap). Moreover, the auditory stimuli were shown to increase, but not to decrease, the perceived number of visual or tactile events. For instance, in the Hötting and Röder’s experiment, presenting only one tone with several taps (2–4) did not affect tactile perception. Therefore, the possibility remained that the observed bias reflected an “isolated” illusion only applying to very specific conditions rather than a general combination of tactile (or visual) and auditory inputs. By showing that (1) task-irrelevant auditory stimuli not only increase but also decrease the perceived number of tactile taps, and that (2) this effect holds for different sequences of tactile stimuli, our results demonstrate that tactile tap perception can be systematically modulated by task-irrelevant auditory inputs.
Another interesting point is the fact that subjects’ responses were significantly less variable when redundant tactile and auditory signals were presented (“Same Number” condition) rather than tactile signals alone (“No Beep” condition). This suggests that even though auditory signals were irrelevant to the task, tactile and auditory signals were probably integrated according to a Maximum Likelihood Estimation (MLE) model. Indeed, one of the predictions of the MLE model of multimodal integration is that cross-modal perceptual estimates should have a lower variance than uni-modal estimates (Ernst and Banks 2002, Alais and Burr 2004, Ernst and Bülthoff 2004). The auditory-evoked modulation of tactile perception that we observed therefore probably results from a real multimodal integration rather than just constituting some response bias. Concerning the fact that the overall tendency to underestimate the number of delivered taps (especially for the three and four taps conditions) was not improved when a congruent stimulus was provided in the auditory modality (in other words, average accuracy of the estimate in the “No Beep” versus “Same Number” condition), we should mention here that the MLE model of multimodal integration makes no prediction about the bias of the estimate.
Another statement from the MLE model of multimodal integration is that the relative weight allocated to the different available channels is inversely proportional to the relative variance of each channel. In line with this, the more salient (or reliable) a signal is, the less susceptible to bias this signal should be. In the same way, the more reliable a “biasing” signal is, the more bias it should induce. Therefore, the fact that auditory signals can bias both visual and tactile perception probably indicates that, when counting the number of events presented in a sequence, auditory signals are more reliable than both visual and tactile signals. In the studies by Shams et al. (2000, 2002), a single visual flash was perceived as two when two beeps were simultaneously presented. When compared to this bias, the effects observed on tactile perception are relatively small. This is true for both the present study and the Hötting and Röder’s experiment (2004). According to the abovementioned statements for the MLE model, this difference in the magnitude of the auditory-evoked effects likely reflects a higher saliency of tactile than visual signals in this kind of non-spatial task.
Finally, our results stress that for an auditory modulation of tactile perception to occur, the task-irrelevant auditory stimulus must present sufficient similarity with the tactile stimulus. Indeed, when the auditory stimulus was obviously dissimilar to the tactile sequence (“Control Beep” auditory condition), the perceived number of taps did not significantly differ from the conditions where no beep was presented (“No Beep” auditory condition) or where the number of beeps was identical to the number of taps (“Same Number” auditory condition). In the second experiment, we tested whether simultaneity constitutes a necessary factor for an integration to occur.
Experiment 2
Methods
Subjects
Sixteen right-handed subjects (aged 19–44 years) participated in the second experiment, which was also performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki. None of the subjects had a history of sensorimotor or auditory disorders, and all subjects gave informed consent before taking part in the experiment.
Procedure
The first and second experiments were identical except for two points. First, only four auditory conditions (“No Beep”, “One Beep Less”, “Same Number”, “One Beep More”) were associated with the tactile sequences (two, three or four taps). Second, we manipulated the timing between the auditory and tactile sequences. For the three auditory conditions in which beeps were presented (“One Beep Less”, “Same Number”, “One Beep More”), five different onset asynchronies between auditory and tactile stimuli were used. These five “timing” conditions are schematically represented in Fig. 4. For the “Auditory 200 ms Before” and “Auditory Immediately Before” timing conditions, the auditory sequence finished 200 ms before and right at the onset of the tactile sequence, respectively. It started immediately and 200 ms after for the “Auditory Immediately After” and “Auditory 200 ms After” timing conditions, respectively. For the “Simultaneous” timing condition, the timing between auditory and tactile stimuli was the same as in the first experiment.
Temporal profiles of the auditory and tactile stimuli for the different timing conditions in Experiment 2. The tactile and auditory stimuli were identical to those used in Experiment 1, but the onset asynchrony between the tactile and auditory sequences was systematically varied between the different timing conditions
In total, the experiment lasted approximately one hour and consisted of 48 different experimental conditions. The “No Beep” auditory condition was only combined with the three tactile conditions (two, three and four taps sequences). These three experimental conditions were used as a baseline of subjects’ performance in identifying the number of taps for the different tactile sequences. The three other auditory conditions (“One Beep Less”, “Same Number”, “One Beep More”) were combined with the three tactile conditions (two, three and four taps sequences), and each of the resulting nine conditions were combined with the five timing conditions (“Auditory 200 ms Before”, “Auditory Immediately Before”, “Simultaneous”, “Auditory Immediately After”, “Auditory 200 ms After”) for a total of 45 experimental conditions. Subjects performed ten trials per experimental condition, resulting in a total of 480 trials. All 48 experimental conditions were randomly intermixed. After each trial, the subjects responded using a number keypad.
Results
Figure 5 presents the average perceived number of taps in the “One Beep Less”, “Same Number” and “One Beep More” auditory conditions, for the five timing conditions. These data can be compared to the average perceived number of taps for the “No Beep” auditory condition (dashed line). For each of the five timing conditions, we tested whether the auditory condition influenced tactile perception using a 4×3 [number of auditory conditions (“No Beep”, “One Beep Less”, “Same Number”, “One Beep More”) × variations on the number of taps (two, three, four)] repeated-measures ANOVA in each case. Because each of these five analyses included the same “No Beep” auditory condition, the α level was adjusted using a Bonferroni correction (so that the α level was set to 0.01 instead of 0.05). When necessary, post hoc comparisons using a Sidak adjustment for multiple comparisons (p<0.01) were performed. The auditory condition only had a significant effect for the “Simultaneous” [F(3,45)=17.858, p<0.001] and the “Auditory Immediately After” [F(3,45)=4.776, p<0.01] timing conditions. The auditory-evoked effect marginally failed to reach significance for the “Auditory Immediately Before” timing condition, and no effect at all was observed for the “Auditory 200 ms Before” and “Auditory 200 ms After” timing conditions. For the “Simultaneous” timing condition, subjects perceived significantly less taps in the “One Beep Less” than in the “No Beep” and the “One Beep More” auditory conditions. The number of taps perceived for “One Beep More” was also significantly higher than for “Same Number”. For the “Auditory Immediately After” timing condition, subjects perceived significantly less taps in the “One Beep Less” than in the “One Beep More” auditory condition. For each of the five ANOVAs, the perceived number of taps depended significantly on the actual number of delivered taps. A significant interaction between the two factors was not found for any of the five timing conditions.
Influence of the auditory stimuli on the perceived number of taps for the different timing conditions. The dashed line is a comparison “baseline” representing the average perceived number of taps in the “No Beep” auditory condition. The perceived number of taps was significantly influenced by the auditory stimuli (more and less presented beeps increased and decreased the perceived number of taps, respectively) when the tactile and auditory sequences were presented simultaneously. This was also true when the auditory sequence was presented immediately after the tactile sequence. On the other hand, when the auditory sequence immediately preceded the tactile sequence, the auditory stimuli marginally failed to significantly alter the perceived number of taps. Finally, with an onset asynchrony of 200 ms between the tactile and auditory sequences, there was no influence of the auditory stimuli on tactile perception
Discussion
The results from this second experiment confirmed that task-irrelevant auditory stimuli can modulate tactile tap perception when similar tactile and auditory stimuli are presented simultaneously. Indeed, in the “Simultaneous” timing condition, the perceived number of delivered taps depended significantly on the number of presented beeps. Moreover, the second experiment showed that the auditory modulation of tactile perception is dependent on the timing between the stimuli presented in the two modalities. As compared to the “Simultaneous” timing condition, the auditory modulation of tactile perception was weaker when the auditory stimuli were presented immediately before the onset or after the end of the tactile sequences. This modulation completely vanished with a 200 ms gap between the auditory and tactile sequences (see Fig. 5). According to the fact that, in everyday life, multimodal signals related to the same stimulus are generally simultaneous, it is perhaps surprising to observe some auditory modulation when the auditory and tactile sequences are non-overlapping. Indeed, in the “Auditory Immediately Before” and “Auditory Immediately After” timing conditions, the stimulus onset asynchrony (SOA) ranged from 155 ms (when only two taps were delivered) to 425 ms (when four taps were delivered). In their investigation of audio-visual integration, Shams et al. (2002) found that the temporal window in which audition can bias the perceived number of visual flashes is about 100 ms, with almost no effect for SOAs above 160 ms. Similarly, Slutsky and Recanzone (2001) reported that the ventriloquism effect (the visual bias of the perceived location of sound) vanishes when the temporal disparity between the visual and the auditory stimuli exceeds 100 ms. Our results therefore suggest that the temporal window of auditory-tactile integration might be slightly wider than for auditory-visual integration.
The auditory sequence significantly biased tactile perception when presented immediately after the tactile sequence, but not when presented immediately before. In this latter case, the auditory-evoked effect failed to reach significance. This slight asymmetry in the timing effect could be related to the fact that in everyday life, due to the time it takes for sound to reach the ears, sounds are generally delayed with respect to tactile stimuli and viewed “contacts” (for instance, when seeing someone knocking on a door or shutting a car’s door). As a consequence, tactile signals might be integrated more easily with delayed than preceding auditory signals. Similar asymmetries in the temporal window of integration have been observed in the visuo-auditory domain (Shams et al. 2002; Slutsky and Recanzone 2001). However, if one considers the sound resulting from a tap delivered on the hand, the time for sound to reach the ears would be less than three milliseconds, since the distance between the hand and ears is less than one meter. Therefore, the sole “natural” temporal disparity between auditory and tactile stimuli cannot account for the asymmetry of the timing effect. An alternative explanation relates to the conduction and processing time differences between auditory and tactile signals (Fraisse 1980). Indeed, the tactile sensory signals from the fingertip take longer to reach the brain than the auditory signals coming from the ears. Therefore, for an auditory and a tactile event to be centrally represented as simultaneous, the tactile stimulus has to precede the auditory one. For instance, when subjects are asked to synchronize simple tap movements with auditory clicks produced by a metronome, the taps generally precede the clicks by about 20–80 ms, furnishing direct evidence of the processing asynchrony between auditory and tactile sensory signals (see Aschersleben and Prinz 1995, Aschersleben et al. 2002 for a review). A similar phenomenon is likely to underlie the asymmetry in the timing effect observed in our experiment. According to the abovementioned processing asynchrony, the CNS should integrate tactile and auditory stimuli more easily when the tactile stimulus precedes the auditory one slightly (in other words, when they are centrally represented as simultaneous) than when the auditory stimulus comes first (in other words, when the temporal asynchrony between the auditory and tactile representation is enhanced).
General discussion
Task-irrelevant auditory signals have been reported to modulate tactile perception of roughness (Guest et al. 2002, Jousmäki and Hari 1998) and stiffness (DiFranco et al. 1997). The present contribution shows that task-irrelevant auditory stimuli can also modulate the tactile perception of sequences of taps delivered on the skin. More generally, this finding confirms that when presented with a pool of multimodal sensory signals, the CNS tends to automatically integrate the ones that are likely to be generated by a common event (Bermant and Welch 1976, Bertelson and Radeau 1981, Fendrich and Corballis 2001, Morein-Zamir et al. 2003, Shams et al. 2000, 2002, Guest et al. 2002, Jousmäki and Hari 1998, Hötting and Röder 2004). This automatic integration likely results from the fact that our everyday experience continuously provides us with co-occurring multimodal signals. Therefore, the CNS associates sets of redundant sensory signals and identifies every single set as elicited by the same unique event or stimulus. Functionally, automatically “sorting” the incoming sensory signals and integrating the ones that are likely to be generated by the same stimulus can be thought of as an optimization process that takes advantage of information redundancy. Indeed, multimodal perceptual estimates have been shown to reduce the variance of the estimates (Wu et al. 1999, Ernst and Banks 2002, Gepshtein and Banks 2003, Alais and Burr 2004) and to enhance the detection of the stimuli (Gielen et al. 1983, Hershenson 1962, Bernstein et al. 1969, Morell 1968, Nickerson 1973).
At the structural level, the mechanisms underlying auditory-tactile integration are likely based on multimodal neurons whose activity is modulated by both auditory and tactile sensory signals. In monkeys, such “auditory-tactile neurons” have been found in several brain structures like the posterior parietal cortex (Hyvärinen and Poranen 1974), the superior temporal sulcus (Desimone and Gross 1979, Hikosaka et al. 1988), and the temporo-parietal cortex (Leinonen et al. 1980). Recent imaging studies evidenced similar audio-tactile interactions in the corresponding structures of the human brain—the posterior parietal cortex and the temporo-parietal cortex (Gobbelé et al. 2003)—as well as in regions traditionally considered as unisensory, like the somatosensory cortices (Foxe et al. 2000, Lütkenhöner et al. 2002), the posterior auditory cortices (Foxe et al. 2000) or a sub-region of the auditory cortex located in the superior temporal gyrus (Foxe et al. 2002). In these areas, simultaneous stimulation in both auditory and tactile modalities resulted in a significantly greater neural activity than the summed responses from auditory-alone and tactile-alone stimulations. This suggests that, when provided with multiple rather than single sensory inputs, multimodal neurons are more likely to fire and/or their firing frequency is likely to increase. The fact that not only “strictly” congruent but also similar cues gathered by different sensory channels are interdependent likely reflects the probabilistic nature of the process of neural integration.
Conclusion
We found that the tactile perception of taps delivered on the skin can be modulated by task-irrelevant auditory stimuli. Our results emphasized that such modulation: 1) is bidirectional (it can not only increase but also decrease the perceived number of taps); 2) can reduce the variability of the perceptual estimates when redundant bimodal signals are presented; 3) depends on both the similarity and the timing of occurrence between the auditory and tactile stimuli, and; 4) presents some temporal flexibility (strict simultaneity between the auditory and tactile stimuli is not necessary for a modulation to occur). These results suggest that: 1) the CNS tends to automatically integrate auditory and tactile signals that are likely to be generated by the same event; 2) this automatic integration likely results from a real multimodal integration, and; 3) the integration can occur even when the auditory and tactile stimuli are presented with a slight temporal asynchrony.
References
Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14:257–262
Aschersleben G, Prinz W (1995) Synchronizing actions with events: the role of sensory information. Percept Psychophys 57:305–317
Aschersleben G, Drewing K, Stenneken P (2002) Temporal coordination of simple movements. Cognitive Processing 1:37–60
Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Motor Skill 42:487–493
Bernstein IH, Clark MH, Edelstein BA (1969) Effects of an auditory signal on visual reaction time. J Exp Psychol 80:567–569
Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29:578–584
Desimone R, Gross CG (1979) Visual areas in the temporal cortex of the macaque. Brain Res 178:363–380
DiFranco DE, Beauregard GL, Srinivasan MA (1997) The effect of auditory cues on the haptic perception of stiffness in virtual environments. P ASME 61:17–22
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169
Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophys 63:719–725
Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Cognitive Brain Res 10:77–83
Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D, Ritter W, Murray MM (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. J Neurophysiol 88:540–543
Fraisse P (1980) Les synchronizations sensori-motrices aux rythmes [The sensorimotor synchronization of rhythms]. In: Requin J (ed) Anticipation et comportement. Editions du CNRS, Paris, pp 233–257
Gepshtein S, Banks MS (2003) Viewing geometry determines how vision and haptics combine in size perception. Curr Biol 13:483–488
Gielen SCAM, Schmidt RA, van den Heuvel PJM (1983) On the nature of intersensory facilitation of reaction time. Percept Psychophys 34:161–168
Gobbelé R, Schürmann M, Forss N, Juottonen, K, Buchner H, Hari R (2003) Activation of the human posterior parietal and temporoparietal cortices during audiotactile interaction. NeuroImage 20:503–511
Guest S, Catmur C, Lloyd D, Spence C (2002) Audiotactile interactions in roughness perception. Exp Brain Res 146:161–171
Hershenson M (1962) Reaction time as a measure of intersensory facilitation. J Exp Psychol 63:289–293
Hikosaka K, Iwai E, Saito H, Tanaka K (1988) Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. J Neurophysiol 60:1615–1637
Hötting K, Röder B (2004) Hearing cheats touch, but less in congenitally blind than in sighted individuals. Psychol Sci 15:60–64
Hyvärinen J, Poranen A (1974) Function of the parietal associative area 7 as revealed from cellular discharges in alert monkeys. Brain 97:673–692
Jousmäki V, Hari R (1998) Parchment-skin illusion: sound-biased touch. Curr Biol 8:190
Leinonen L, Hyvarinen J, Sovijarvi AR (1980) Functional properties of neurons in the temporo-parietal association cortex of awake monkey. Exp Brain Res 39:203–215
Lütkenhöner B, Lammertmann C, Simoes C, Hari R (2002) Magnetoencephalographic correlates of audiotactile interaction. NeuroImage 15:509–522
Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cognitive Brain Res 17:154–163
Morell LK (1968) Temporal characteristics of sensory interaction in choice reaction times. J Exp Psychol 77:14–18
Nickerson RS (1973) Intersensory facilitation of reaction time: Energy summation or preparation enhancement? Psychol Rev 80:489–509
Shams L, Kamitani Y, Shimojo S (2000) What you see is what you hear. Nature 408:788
Shams L, Kamitani Y, Shimojo S (2002) Visual illusion induced by sound. Cognitive Brain Res 14:147–152
Slutsky D, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. NeuroReport 12:7–10
Wu WC, Basdogan C, Srinivasan MA (1999) Visual, haptic, and bimodal perception of size and stiffness in virtual environments. P ASME 67:19–26
Acknowledgements
This work was supported by the Max-Planck Society and by the 5th Framework IST Program of the EU (IST-2001–38040, TOUCH-HapSys). We thank Kari Hoffman and Roland Fleming for helpful comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bresciani, JP., Ernst, M.O., Drewing, K. et al. Feeling what you hear: auditory signals can modulate tactile tap perception. Exp Brain Res 162, 172–180 (2005). https://doi.org/10.1007/s00221-004-2128-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-004-2128-2