Abstract
When one sensory response occurs in the presence of a different sensory stimulation, the sensory response is often amplified. The variety of sensory enhancement data tends to obscure the underlying rules, but it has long been clear that weak signals are usually amplified more than strong ones (the Principle of Inverse Effectiveness). Here we show that for many kinds of sensory amplification, the underlying law is simple and elegant: the amplified response is a power law of the unamplified response, with a compressive exponent that amplifies weak signals more than strong. For both psychophysics and cortical electrophysiology, for both humans and animals, and for both sensory integration and enhancement within a sense, gated power law amplification (amplification of one sense triggered by the presence of a different sensory signal) is often sufficient to explain sensory enhancement.
Similar content being viewed by others
Introduction
Sensory enhancement or amplification is common when one sensory activity occurs in the presence of another1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24. For example, lights can appear brighter when a sound comes from the same direction in space4. Often that amplification seems to be intensity-dependent, with weak stimuli being amplified more than stronger stimuli. Moreover, neural systems responding weakly to a single stimulus often have more enhanced multisensory neural responses than neurons with strong unisensory responses. In sensory integration this is called the Principle of Inverse Effectiveness3, but this can also be true for amplification that occurs within a sense. Amplification is studied in many ways and presented in many formats, but has thus far evaded lawful quantification. Here we find a curious near-universal invariance: the amplified response is a power law function of the unamplified response, with a compressive exponent that explains much of intensity-dependent enhancement. The only exception we have found is for bimodal cells, especially in superior colliculus. For both psychophysics and cortical spike rate data, power law exponents are clustered around 0.85. This is true for vision amplified by auditory modulation and for audition amplified by somatosensory information. This is also true for amplification seen within a sense (in this case for human color vision). Similar power laws govern other electrophysiological measures. The power laws’ r2 values are generally extremely high, suggesting that gated (triggered) amplification, not nonlinear combination, is often sufficient to explain sensory enhancement.
Results
Data showing that sensory signals are often enhanced or amplified in the presence of another sensory activity are shown in Fig. 1. The circumstances and nature of the amplification are diverse, but weakly effective stimuli generally receive relatively more amplification – the influential Principle of Inverse Effectiveness from sensory integration3. The details of inverse effectiveness are a subject of some controversy in sensory integration11,18,20, but it is a useful rule-of-thumb and the nature of intensity-dependent amplification is a matter of importance; a more general and fundamental rule for amplification would be welcome. Such a rule is shown in Fig. 2, where all four data sets from Fig. 1 are plotted in the same framework. Here the amplified or enhanced response is plotted as a function of the unamplified response. All four amplification functions − despite their differences in Fig. 1 − turn out to be power laws of the form,
Note that because both amplified and unamplified response have the same units, the fitting parameters a and n are unitless. This may be useful for comparing electrophysiology and psychophysics.
Implications of all four cases following the same rule
There are four immediate implications of these findings. First, it is unlikely that the amplification effects stem from attention, which is also known to enhance sensory signal processing25. Compare (in Fig. 2) visual brightness amplification via auditory modulation (the red squares and line) to the firing rate increase in cat visual neurons (black triangles and line) when an auditory signal is present. Because the neural data was collected in anesthetized cats17, attention (in the classic sense) is unlikely to be involved. (We acknowledge the existence of preconscious elements to the range of phenomena involved in attention, but doubt their applicability here). By extension, the similarity of the psychophysical brightness amplification function to the neural data suggests that whatever neural mechanism amplifies the neural response in unconscious cats would suffice to account for the human psychophysical data as well.
Second, the quality of the fits in Fig. 2 are superb, with r2 values that range from 0.989 to 0.999 (Table 1). Since almost all of the variance is accounted for by a function of a single variable, the enhancements seen here are consistent with simple gated amplifications (a) of the unenhanced responses. The role of the modulating stimulus is to trigger the amplification of the modulated stimulus. It has long been known that there are essentially two classes of multisensory neurons: true multimodal neurons (bimodal and trimodal) and subthreshold multimodal neurons (also known as modulated unisensory neurons). True multimodal neurons, prominent in midbrain (superior colliculus/optic tectum) but also found in cortex, produce amplifications based on two or more inputs, and can produce remarkable amplifications (as much as 1200%3). The subthreshold multimodal neurons fire to only one kind of sensory stimulus, but are modulated by another, are studied mainly in cortex, and produce more modest amplifications, roughly a factor of two for weak stimuli in many studies15,17,19. It has been argued (based on refs6,7,17) that the modest amplifications produced by subthreshold neurons are more in keeping with amplifications seen in psychophysics than the remarkable amplifications produced by true multimodal neurons24. The data in Fig. 2 are consistent with that argument. The neural data shown in black in Figs 1 and 2 are from subthreshold multisensory neurons and the form of the subthreshold neurons’ amplification function is quite similar to the most relevant psychophysical data (red).
Third, a psychophysical measure − chromatic brightness spectral sensitivity − can be predicted from luminance spectral sensitivity; chromatic brightness is well described by a simple power law of luminance. Moreover this amplification function that describes chromatic brightness spectral sensitivity as a simple amplification of luminance spectral sensitivity (Fig. 2, blue data) follows the same rule as sensory integration data, suggesting similar (perhaps generic) amplification mechanisms can mediate both multisensory integration and sub-modal interactions (in this case hue-luminance interactions) within a single sense like color vision. This surprising result requires some background to be properly appreciated. Luminance spectral sensitivity and chromatic brightness spectral sensitivity were originally alternative ways of probing the spectral effectiveness of different color lights. Luminance was measured by using criteria that detected spatial or temporal discontinuities under conditions that discount color vision; chromatic brightness was measured by equating apparent brightness of different wavelength stimuli26. Luminance turned out to predict human performance (e.g., acuity) better, while chromatic brightness was the better predictor of color appearance. Chromatic brightness as a function of wavelength is broader than luminance; blue and red lights are brighter than they are luminous26. Luminance spectral sensitivity is associated with the spectral sensitivity of the achromatic channel (and can be measured physiologically), but there is no known color vision mechanism that corresponds to chromatic brightness. Because hue-opponent mechanisms like the red-green and blue-yellow channels have strong responses in the long- and short-wavelength (reddish and bluish) ends of the spectrum where luminance is weak, it has long been thought that chromatic brightness might be a weighted combination of hue and luminance responses and there is evidence that hue and luminance signals combine synergistically (via probability summation) at threshold. In particular, vector models of chromatic brightness have been popular (see refs26,27 for review). In these models chromatic brightness is the vector sum of weighted contributions of luminance and the absolute magnitude of the hue opponent (red-green and blue-yellow) channels. However, the fit of the power law amplification function in Fig. 2 contradicts theories that suprathreshold chromatic brightness is a combination of luminance and hue signals. Chromatic brightness is almost completely accounted for as a simple power law amplification of luminance (r2 = 0.993) leaving little variance for the hue channels to account for. The fit suggests that chromatic brightness is a simple amplification of luminance; likely this amplification is triggered by the presence of the hue signal, just as the visual neurons in Fig. 1b fire harder for vision stimuli when auditory stimuli are present.
Fourth, the mildly compressive exponent n shown by all four data sets results in stronger amplification for weak (or weakly effective) signals than for strong ones. This is broadly consistent with the Principle of Inverse Effectiveness, but does not clarify some details like the extra enhancement of some weak − but not weakest − stimuli found in some studies11,13,17,18,20. Some examples of this can be seen in in Fig. 1b and in Fig. 3, which features data from cat extrastriate visual cortex. These are modulated unisensory neurons that respond to visual stimuli, do not respond to auditory stimuli alone but show enhanced spike rates when both visual and auditory stimuli are present. If we replot the data in Fig. 3a as percent enhancement in Fig. 3b, there is much neural variability, but it can be clearly seen that most of the strongest enhancements on a percent basis are for relatively low (2–6 spikes/sec) unenhanced spike rate responses, consistent with the Principle of Inverse Effectiveness.
Establishing the domain of the power law
It is worth seeking excursions from the amplification function power law behaviors, both to establish the domain of Eq. 1 and because exceptions can be instructive. Figure 4 explores three cases.
Figure 4a shows power law fits to two sets of bimodal cells, one set from superior colliculus and one set from cortical area AES, both taken from the same study23. Since the amplified response produced by bimodal cells is (by definition) a function of two variables, a simple power law should not produce as good a fit to bimodal cell firing rates as it does to the firing rates of subthreshold neurons, and indeed it does not, especially for superior collicular (midbrain) neurons (r2 = 0.63). However, the fit is somewhat better for cortical AES bimodal cells (r2 = 0.83) and the resulting exponent (0.84) is within the range of the exponents found for subthreshold cells (Table 1). Similar results were found for other cortical bimodal cells in areas AES and PLLS (see Table 1). This is interesting because superior collicular bimodal and cortical subthreshold neurons are sometimes thought of as two points along a multisensory processing continuum22, and Meredith’s findings23 from AES (reanalyzed in Fig. 4a) argue that cortical bimodal neurons lie somewhere in the midst of that continuum.
Figure 4b explores electrophysiological data other than spike rates. The data shown are for current source densities and multiunit activity in macaque auditory cortex12. Current source densities (CSD) and multiunit activity (MUA) are thought to reflect massed synaptic and axonal activity respectively12. Figure 4b shows only the data from one neural layer, but both fits from each of three sampled cortical layers are shown in Table 1. Although the exponents of the power laws vary, the power law fits are quite good and consistent with a function of a single variable (a gated amplification). This speaks to a hypothesis raised by Laurienti et al.7, that measures of mass activity in cortex will likely reflect the actions of the more cortically numerous neurons that do not produce large amplifications (a group that would include subthreshold neurons).
Finally Fig. 4c looks at the predictions of a model of spike rate amplification in simulated synaptically-coupled subthreshold multisensory neurons24. The amplification function produced by the model is shown by a dotted red line. The amplification function is well approximated by a power law (solid red line; n = 0.83; r2 = 0.97), but a cursory examination shows small systematic deviations from the power law at low spike rates, suggesting a venue for additional modeling. Real data from the cat cortical neurons in Fig. 1b (black triangles)17 and the Eq. 1 fit to that data, are superimposed on the model for comparison only (e.g., the model is computed from first principles, instead of being fit to the data).
Discussion
In summary, sensory enhancement for psychophysical data and for all electrophysiological data (except data from bimodal cells) is consistent with gated (triggered) power law amplification. Nothing about the second signal − other than its presence as an amplification trigger − is required to predict the amplification of the enhanced sensory response from the unenhanced response. The power law exponent varies from slightly compressive (0.9) to more compressive (0.6), consistent with the amplification of weak stimuli, relative to strong ones, found throughout the sensory literature. If the generality of these findings hold up, this may have other consequences. For example, S.S. Stevens and his associates spent much effort describing power law responses for single sensory responses; there is a rich database here covering many sensory variables28. Figure 1a shows a nice illustration of this – Stein’s lab4 finds a Stevens’ law for brightness of (quiet) lights, with an exponent of 0.6. They also found that the Stevens’ law for enhanced data has an exponent of 0.5. The generality of the results of this paper suggest that estimates for Stevens’ laws for multisensory amplifying conditions (in various senses), could be obtained by multiplying the exponents that Stevens obtained by about 0.85 (the average of the four psychophysical exponents from the amplification functions shown in Table 1).
Compressive nonlinearities like those usually found in Stevens’ law are common in electrophysiology and psychophysics and contribute to a larger dynamic range for sensory systems. The compressive exponents found here can be a second order effect arising from interactions of already compressive sensory systems. For example, in Fig. 4c, a binding-theory-based model of amplification employs mutual excitatory synaptic connections between two responding sensory systems, and has been used to model amplification in rattlesnakes and cats24. The mutual excitation results in amplification as each system enhances the other’s firing rates. The compression in this case arises partly from spike rate compression in the firing neurons and partly from a simulated synaptic depression, which tempers amplification at high firing rates more than low.
The similarity of the amplification functions for sensory integration and chromatic brightness (Fig. 2, Table 1) are intriguing for three reasons. First, it implies that the mechanisms involved may be quite generic and may be applicable to other psychophysical phenomena. Second, it provides a fascinating alternate explanation of the origins of chromatic brightness, a fundamental color percept. Third, it prompts consideration of the relationship between different kinds of neural information integration problems. This has a precedent in multisensory integration. For example, rattlesnake optic tectum integrates infrared information from their facial pits and visual information from their eyes and this is considered multisensory integration29,30, but if the rattlesnake’s infrared transduction had taken place in the retina, the integration of the infrared and visual spectral bands would almost certainly be considered an aspect of color vision.
Finally, it is surprising that so much perceptual and neural data can be accounted for as a gated amplification of an underlying sensory variable, because there is a rich literature on linear and nonlinear weighted cue combinations, and on strong- and weak-fusions of sensory information (see31 for recent reviews). The phenomena described here and the phenomena described in the cue combination literature seem to belong to different perceptual regimes and it would be interesting to understand whether the gulf between them is fixed and sharply delineated, or to explore the perceptual processing regime between them.
Theoretical Methods
Data from ref.17 was obtained from the authors. Data from refs10,32,33 were taken from published tables. Data from refs4,9,12,15,19,23 were digitized from greatly magnified graphs using digital calipers (Lakatos provided better graphs than those available in ref.12 for this purpose). Datasets that had too many closely packed unresolvable data points were not employed unless the author made the data available. The data (red dotted line) shown in Fig. 4c (labeled neural model) was computed from a model24 that consists of two simulated Hodgkin-Huxley-like neurons with mutual excitatory synaptic coupling. The eight coupled differential equations that represent these spiking neurons and their synapses were integrated using a fourth-order Runge-Kutta algorithm in MATLAB.
References
Meredith, M. A. & Stein, B. E. Interactions among converging sensory inputs in the superior colliculus. Science 221, 389–391 (1983).
Meredith, M. A. & Stein, B. E. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640–662 (1986).
Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).
Stein, B. E., London, N., Wilkinson, L. K. & Price, D. D. Enhancement of a perceived visual intensity by auditory stimuli: A psychophysical analysis. J. Cogn. Neurosci. 8, 497–506 (1996).
Lovelace, C. T., Stein, B. E. & Wallace, M. T. An irrelevant light enhances auditory detection in humans: A psychophysical analysis of multisensory integration in stimulus detection. Cogn. Brain Res. 17, 447–453 (2003).
Perrault, T. J. Jr., Vaughan, J. W., Stein, B. E. & Wallace, M. T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J. Neurophysiol. 90, 4022–4026 (2003).
Laurienti, P. J., Perrault, T. J., Stanford, T. R., Wallace, M. T. & Stein, B. E. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp. Brain Res. 166, 289–297 (2005).
Holmes, N. P. & Spence, C. Multisensory integration: Space, time and superadditivity. Current Biol. 15, 762–764 (2005).
Allman, B. & Meredith, M. A. Multisensory processing in ‘unimodal’ neurons: cross-modal subthreshold auditory effects in cat extrastriate visual cortex. J. Neurophysiol. 98, 545–549, https://doi.org/10.1152/jn.00173.2007 (2007).
Gillmeister, H. & Eimer, M. Tactile enhancement of auditory detection and perceived loudness. Brain Res. 1160, 58–68, https://doi.org/10.1016/j.brainres.2007.03.041 (2007).
Holmes, N. P. The law of inverse effectiveness in neurons and behavior: Multisensory integration verses normal variability. Neuropsychologica 45, 3340–3345 (2007).
Lakatos, P., Chen, C. M., O’Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in the primary auditory cortex. Neuron 53, 279–292 (2007).
Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C. & Foxe, J. J. Do you see what I’m saying? Exploring visual amplification of speech comprehension in noisy environments. Cerebral Cortex 17, 1147–1152 (2007).
Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. NeuroReport 18, 787–792 (2007).
Allman, B. L., Keniston, L. P. & Meredith, M. A. Subthreshold auditory inputs to extrastriate visual neurons are responsive to parametric changes in stimulus quality: Sensory-specific vs non-specific coding. Brain Res. 1242, 95–101 (2008).
Kayser, C., Petkov, C. L. & Logothetis, N. K. Visual modulation of neurons in auditory cortex. Cerebral Cortex 18, 1560–1574 (2008).
Allman, B. L., Keniston, L. P. & Meredith, M. A. Not just for bimodal neurons anymore: The contribution of unimodal neurons to cortical multisensory processing. Brain Topogr. 21, 157–167 (2009).
Holmes, N. P. The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topogr. 21, 168–176 (2009).
Meredith, M. A. & Allman, B. L. Subthreshold multisensory processing in cat auditory cortex. NeuroReport 20, 126–131 (2009).
Stein, B. E. et al. Challenges in quantifying multisensory effectiveness: Alternative criteria, models, and inverse effectiveness. Exp. Brain Res. 198, 113–126 (2009).
Schirillo, J. A. Cross-modal detection using various temporal and spatial configurations. Attent., Percept. & Psychophys. 73, 237–246 (2011).
Kayser, C., Lakatos, P. & Meredith, M. A. Cellular physiology of cortical multisensory processing. In The New Handbook of Multisensory Processing (ed. Stein, B. E.) 115–134 (MIT Press, 2012).
Meredith, M. A., Allman, B. L., Keniston, L. P., Clemo, H. R. Are bimodal neurons the same throughout the brain? In The Neural Bases of Multisensory Processes (eds Murray M. M. & Wallace M. T.) 51–64 (CRC Press, 2012).
Billock, V. A. & Tsou, B. H. Bridging the divide between sensory integration and binding theory: Using a binding-like neural synchronization mechanism to model sensory enhancements during multisensory interactions. J. Cogn. Neurosci. 26, 1587–1599 (2014).
Hilllyard, S. A., Vogel, E. K. & Luck, S. J. Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. Phil. Trans. Roy. Soc. B 353, 1257–1270 (1998).
Wyszecki, G. & Stiles, W. S. Color Science: Concepts and Methods, Quantitative Data and Formulae, 2 nd Edition (Wiley, 1982).
Billock, V. A. & Tsou, B. H. Sensory recoding via neural synchronization: integrating hue and luminance into chromatic brightness and saturation. Journal of the Optical Society of America A 22, 2289–2298 (2005).
Stevens, S. S. Psychophysics (Transaction Publishers, 1975).
Hartline, P. H., Kass, L. & Loop, M. S. Merging of modalities in the optic tectum: infrared and visual integration in rattlesnakes. Science 199, 1225–1229 (1978).
Newman, E. A. & Hartline, P. H. Integration of visual and infrared information in bimodal neurons in the rattlesnake optic tectum. Science 213, 789–791 (1981).
Trommershäuser, J., Körding, K. P. & Landy, M. S. (eds) Sensory Cue Integration (Oxford Univ. Press, 2011).
Ikeda, M. & Nakano, Y. Spectral luminous-efficiency functions obtained by direct heterochromatic brightness matching for point sources and for 2° and 10° fields. J. Opt. Soc. Am. A 3, 2105–2108 (1986).
Sharpe, L. T., Stockman, A., Jagla, W. & Jägle, H. A luminous efficiency function, V*(λ), for daylight adaptation. J. Vision 5, 948–968 (2005).
Acknowledgements
We thank Angela Brown, Rhea Eskew, Michael Loop, Steven Shevell and Andrew Stockman for helpful discussions, and Brian Allman, Alex Meredith, and Peter Lakatos for sharing some data from their studies. This research was funded by NSF #1456650 and by an ORISE/AFRL Faculty Fellowship to V. Billock.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The authors declare no competing interests.
Additional information
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Billock, V.A., Havig, P.R. A Simple Power Law Governs Many Sensory Amplifications and Multisensory Enhancements. Sci Rep 8, 7645 (2018). https://doi.org/10.1038/s41598-018-25973-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-018-25973-w
- Springer Nature Limited