Abstract
Conventional views of brain organization suggest that regions at the top of the cortical hierarchy processes internally oriented information using an abstract amodal neural code. Despite this, recent reports have described the presence of retinotopic coding at the cortical apex, including the default mode network. What is the functional role of retinotopic coding atop the cortical hierarchy? Here we report that retinotopic coding structures interactions between internally oriented (mnemonic) and externally oriented (perceptual) brain areas. Using functional magnetic resonance imaging, we observed robust inverted (negative) retinotopic coding in category-selective memory areas at the cortical apex, which is functionally linked to the classic (positive) retinotopic coding in category-selective perceptual areas in high-level visual cortex. These functionally linked retinotopic populations in mnemonic and perceptual areas exhibit spatially specific opponent responses during both bottom-up perception and top-down recall, suggesting that these areas are interlocked in a mutually inhibitory dynamic. These results show that retinotopic coding structures interactions between perceptual and mnemonic neural systems, providing a scaffold for their dynamic interaction.
Similar content being viewed by others
Data availability
Data are available on Open Science Framework (https://osf.io/sm2xf).
Code availability
Code used for data analysis is available on Open Science Framework (https://osf.io/sm2xf).
References
Libby, A. & Buschman, T. J. Rotational dynamics reduce interference between sensory and memory representations. Nat. Neurosci. 24, 715–726 (2021).
Kiyonaga, A., Scimeca, J. M., Bliss, D. P. & Whitney, D. Serial dependence across perception, attention, and memory. Trends Cogn. Sci. 21, 493 (2017).
Summerfield, C. & de Lange, F. P. Expectation in perceptual decision making: neural and computational mechanisms. Nat. Rev. Neurosci. 15, 745–756 (2014).
Rademaker, R. L., Chunharas, C. & Serences, J. T. Coexisting representations of sensory and mnemonic information in human visual cortex. Nat. Neurosci. 22, 1336–1344 (2019).
Favila, S. E., Lee, H. & Kuhl, B. A. Transforming the concept of memory reactivation. Trends Neurosci. 43, 939–950 (2020).
Yassa, M. A. & Stark, C. E. L. Pattern separation in the hippocampus. Trends Neurosci. 34, 515–525 (2011).
Holmes, G. Disturbances of vision by cerebral lesions. Br. J. Ophthalmol. 2, 353–384 (1918).
Wandell, B. A., Dumoulin, S. O. & Brewer, A. A. Visual field maps in human cortex. Neuron 56, 366–383 (2007).
Guclu, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).
Groen, I. I. A., Dekker, T. M., Knapen, T. & Silson, E. H. Visuospatial coding as ubiquitous scaffolding for human cognition. Trends Cogn. Sci. https://doi.org/10.1016/j.tics.2021.10.011 (2022).
Popham, S. F. et al. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nat. Neurosci. 24, 1628–1636 (2021).
Huntenburg, J. M., Bazin, P. L. & Margulies, D. S. Large-scale gradients in human cortical organization. Trends Cogn. Sci. 22, 21–31 (2018).
Margulies, D. S. et al. Situating the default-mode network along a principal gradient of macroscale cortical organization. Proc. Natl Acad. Sci. USA 113, 12574–12579 (2016).
Bellmund, J. L. S., Gärdenfors, P., Moser, E. I. & Doeller, C. F. Navigating cognition: spatial codes for human thinking. Science (1979) 362, eaat6766 (2018).
Szinte, M. & Knapen, T. Visual organization of the default network. Cereb. Cortex 30, 3518–3527 (2020).
Christiaan Klink, P., Chen, X., Vanduffel, W. & Roelfsema, P. R. Population receptive fields in non-human primates from whole-brain fMRI and large-scale neurophysiology in visual cortex. eLife 10, e67304 (2021).
Dumoulin, S. O. & Wandell, B. A. Population receptive field estimates in human visual cortex. Neuroimage 39, 647 (2008).
Silson, E. H., Chan, A. W. Y., Reynolds, R. C., Kravitz, D. J. & Baker, C. I. A retinotopic basis for the division of high-level scene processing between lateral and ventral human occipitotemporal cortex. J. Neurosci. 35, 11921–11935 (2015).
Kundu, P., Inati, S. J., Evans, J. W., Luh, W. M. & Bandettini, P. A. Differentiating BOLD and non-BOLD signals in fMRI time series using multi-echo EPI. Neuroimage 60, 1759–1770 (2012).
Steel, A., Garcia, B. D., Silson, E. H. & Robertson, C. E. Evaluating the efficacy of multi-echo ICA denoising on model-based fMRI. Neuroimage 264, 119723 (2022).
Wang, L., Mruczek, R. E. B., Arcaro, M. J. & Kastner, S. Probabilistic maps of visual topography in human cortex. Cereb. Cortex 25, 3911–3931 (2015).
Thomas Yeo, B. T. et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. J. Neurophysiol. 106, 1125–1165 (2011).
Shmuel, A. et al. Sustained negative BOLD, blood flow and oxygen consumption response and its coupling to the positive response in the human brain. Neuron 36, 1195–1210 (2002).
Steel, A., Billings, M. M., Silson, E. H. & Robertson, C. E. A network linking scene perception and spatial memory systems in posterior cerebral cortex. Nat. Commun. 12, 1–13 (2021).
Hasson, U., Levy, I., Behrmann, M., Hendler, T. & Malach, R. Eccentricity bias as an organizing principle for human high-order object areas. Neuron 34, 479–490 (2002).
Dilks, D. D., Julian, J. B., Paunov, A. M. & Kanwisher, N. The occipital place area is causally and selectively involved in scene perception. J. Neurosci. 33, 1331–1336 (2013).
Epstein, R. & Kanwisher, N. A cortical representation the local visual environment. Nature 392, 598–601 (1998).
Breedlove, J. L., St-Yves, G., Olman, C. A. & Naselaris, T. Generative feedback explains distinct brain activity codes for seen and mental images. Curr. Biol. https://doi.org/10.1016/j.cub.2020.04.014 (2020).
Favila, S. E., Kuhl, B. A. & Winawer, J. Perception and memory have distinct spatial tuning properties in human visual cortex. Nat. Commun. 13, 5864 (2022).
Bastos, A. M. et al. Canonical microcircuits for predictive coding. Neuron 76, 695–711 (2012).
Knapen, T. Topographic connectivity reveals task-dependent retinotopic processing throughout the human brain. Proc. Natl Acad. Sci. USA 118, e2017032118 (2021).
Silson, E. H., Zeidman, P., Knapen, T. & Baker, C. I. Representation of contralateral visual space in the human hippocampus. J. Neurosci. 41, 2382–2392 (2021).
Zabelina, D. L. & Andrews-Hanna, J. R. Dynamic network interactions supporting internally-oriented cognition. Curr. Opin. Neurobiol. 40, 86–93 (2016).
Shulman, G. L. et al. Common blood flow changes across visual tasks: II. Decreases in cerebral cortex. J. Cogn. Neurosci. 9, 648–663 (1997).
Fox, M. D. et al. The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proc. Natl Acad. Sci. USA 102, 9673–9678 (2005).
Raichle, M. E. The brain’s default mode network. Annu Rev. Neurosci. 38, 433–447 (2015).
Robertson, C. E. et al. Neural representations integrate the current field of view with the remembered 360° panorama in scene-selective cortex. Curr. Biol. 26, 2463–2468 (2016).
Braga, R. M. & Buckner, R. L. Parallel interdigitated distributed networks within the individual estimated by intrinsic functional connectivity. Neuron 95, 457–471.e5 (2017).
DiNicola, L. M., Braga, R. M. & Buckner, R. L. Parallel distributed networks dissociate episodic and social functions within the individual. J. Neurophysiol. 123, 1144–1179 (2020).
Silson, E. H., Steel, A. D. & Baker, C. I. Scene-selectivity and retinotopy in medial parietal cortex. Front Hum. Neurosci. 10, 412 (2016).
Silson, E. H., Steel, A., Kidder, A., Gilmore, A. W. & Baker, C. I. Distinct subdivisions of human medial parietal cortex support recollection of people and places. eLife 8, e47391 (2019).
Deen, B. & Freiwald, W. A. Parallel systems for social and spatial reasoning within the cortical apex. Preprint at bioRxiv https://doi.org/10.1101/2021.09.23.461550 (2021).
Ranganath, C. & Ritchey, M. Two cortical systems for memory-guided behaviour. Nat. Rev. Neurosci. 13, 713–726 (2012); https://doi.org/10.1038/nrn3338
Pertzov, Y., Avidan, G. & Zohary, E. Multiple reference frames for saccadic planning in the human parietal cortex. J. Neurosci. 31, 1059 (2011).
Gardner, J. L., Merriam, E. P., Movshon, J. A. & Heeger, D. J. Maps of visual space in human occipital cortex are retinotopic, not spatiotopic. J. Neurosci. 28, 3988 (2008).
Fetsch, C. R., Wang, S., Gu, Y., DeAngelis, G. C. & Angelaki, D. E. Spatial reference frames of visual, vestibular, and multimodal heading signals in the dorsal subdivision of the medial superior temporal area. J. Neurosci. 27, 700–712 (2007).
Golomb, J. D. & Kanwisher, N. Higher level visual cortex represents retinotopic, not spatiotopic, object location. Cereb. Cortex 22, 2794–2810 (2012).
Silson, E. H., Groen, I. I. A., Kravitz, D. J. & Baker, C. I. Evaluating the correspondence between face-, scene-, and object-selectivity and retinotopic organization within lateral occipitotemporal cortex. J. Vis. 16, 14–14 (2016).
Swisher, J. D., Halko, M. A., Merabet, L. B., McMains, S. A. & Somers, D. C. Visual topography of human intraparietal sulcus. J. Neurosci. 27, 5326–5337 (2007).
Yeatman, J. D. et al. The vertical occipital fasciculus: a century of controversy resolved by in vivo measurements. Proc. Natl Acad. Sci. USA 111, E5214–E5223 (2014).
Peirce, J. W. PsychoPy—psychophysics software in Python. J. Neurosci. Methods. 162, 8–13 (2007).
Weiner, K. S. et al. Defining the most probable location of the parahippocampal place area using cortex-based alignment and cross-validation. Neuroimage 170, 373–384 (2018).
Li, X., Morgan, P. S., Ashburner, J., Smith, J. & Rorden, C. The first step for neuroimaging data analysis: DICOM to NIfTI conversion. J. Neurosci. Methods. 264, 47–56 (2016).
Fischl, B. FreeSurfer. NeuroImage 62, 774–781 (2012); https://doi.org/10.1016/j.neuroimage.2012.01.021
Fischl, B. et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron 33, 341–355 (2002).
Dale, A. M., Fischl, B. & Sereno, M. I. Cortical surface-based analysis. I. Segmentation and surface reconstruction. Neuroimage 9, 179–194 (1999).
Saad, Z. S. & Reynolds, R. C. SUMA. Neuroimage 62, 768–773 (2012).
Cox, R. W. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Computers Biomed. Res. 29, 162–173 (1996).
Jo, H. J. et al. Effective preprocessing procedures virtually eliminate distance-dependent motion artifacts in resting state FMRI. J. Appl. Math. 2013, 935154 (2013).
DuPre, E. et al. TE-dependent analysis of multi-echo fMRI with *tedana*. J. Open Source Softw. 6, 3669 (2021).
DuPre, E. et al. ME-ICA/tedana: 0.0.6. Zenodo https://doi.org/10.5281/ZENODO.2558498 (2019).
Evans, J. W., Kundu, P., Horovitz, S. G. & Bandettini, P. A. Separating slow BOLD from non-BOLD baseline drifts using multi-echo fMRI. Neuroimage 105, 189–197 (2015).
Argall, B. D., Saad, Z. S. & Beauchamp, M. S. Simplified intersubject averaging on the cortical surface using SUMA. Hum. Brain Mapp. 27, 14–27 (2006).
Julian, J. B., Fedorenko, E., Webster, J. & Kanwisher, N. An algorithmic method for functionally defining regions of interest in the ventral visual pathway. NeuroImage 60, 2357–2364, https://doi.org/10.1016/j.neuroimage.2012.02.055 (2012).
Gomez, J., Barnett, M. & Grill-Spector, K. Extensive childhood experience with Pokémon suggests eccentricity drives organization of visual cortex. Nat. Hum. Behav. 3, 611–624, https://doi.org/10.1038/s41562-019-0592-8 (2019).
Gomez, J. et al. Development of population receptive fields in the lateral visual stream improves spatial coding amid stable structural-functional coupling. NeuroImage 188, 59–69 (2019).
R Core Team. R: a language and environment for statistical computing (R Foundation for Statistical Computing, 2013); www.r-project.org/
Lawrence, M. A. ez: easy analysis and visualization of factorial experiments. R package version 4.0.2 (2016).
Acknowledgements
We thank I. Groen for assistance with specific analysis code. This work was supported by the National Institute of Mental Health under award number R01MH130529 (C.E.R.). A.S. was supported by the Neukom Institute for Computational Science and E.H.S. by the Biotechnology and Biological Sciences Research Council award number BB/V003917/1.
Author information
Authors and Affiliations
Contributions
A.S., E.H.S. and C.E.R. conceived of and designed the experiment. E.H.S. and B.D.G. contributed stimulus code. A.S. and B.D.G. collected the data. A.S. processed the data. A.S. and E.H.S. analyzed the data. A.S., E.H.S. and C.E.R. wrote and edited the paper.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Neuroscience thanks Christopher Baldassano, Alexander Huth and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Transition from positive to negative-amplitude population receptive fields (+pRF, -pRF) moving anteriorly from posterior cerebral cortex is evident in individual participants.
Figure depicts amplitude maps from all participants’ left hemispheres. Only vertices surviving the threshold applied in the main text (R2 > 0.08) are shown. Individual participant SPAs and PMAs used for analysis are drawn in white (PMAs) and black (SPAs).
Extended Data Fig. 2 Retinotopic coding in SPAs and PMAs.
To quantify the extent to which retinotopic coding is expressed within each ROI we first calculated the percentage of suprathreshold pRFs (R2 > 0.08) within our ROIs for each subject separately before testing each against a non-retinotopic prediction using t-tests (that is, t-test versus zero, with Bonferroni correction). Retinotopic coding was significantly present within each ROI (LH; OPA: t(12.51)=, pcorr = 3.35-9, D = 2.99; PPA: t(16) = 8.68, pcorr = 5.67-7, D = 2.17; LPMA: t(16) = 6.23, pcorr = 3.58-5, D = 1.59; VPMA: t(16) = 6.65, pcorr = 1.64-5, D = 1.66; RH; OPA: t(16) = 12.15, pcorr = 5.10-9, D = 3.03; PPA: t(16) = 11.97, pcorr = 6.32-9, D = 3.12; LPMA: t(16) = 8.75, pcorr = 5.05-7, D = 2.18; VPMA: t(16) = 6.39, pcorr = 2.68-5, D = 1.55). Bars represent the mean percentage of suprathreshold pRFs (R2 > 0.08) in each ROI/hemisphere for the lateral (left) and ventral (right) surfaces, respectively. Individual data points are overlaid. Each ROI exhibited a significant percentage of suprathreshold pRFs, ***ptwo-tailed < 0.001.
Extended Data Fig. 3 Comparison between the location of the SPAs, PMAs, and default mode network in one participant.
Comparison between the location of the SPAs, PMAs, and default mode network in one participant (example participant from Main text Fig. 2). This pattern was consistent in all individuals and at the group-level (Main text Fig. 1b). Default mode network defined using the Yeo et al., 2011 parcellation24.
Extended Data Fig. 4 Correlation in trial × trial activation during memory recall aggregated across participants.
Mean BOLD response amplitude relative to baseline during place recall trials for each ROI (OPA, LMPA) and pRF population (+/−).
Extended Data Fig. 5 Differential interaction between pRFs in SPAs with −/+ pRFs in memory areas is evident across all trials.
Each scatter plot and corresponding correlation values depict the unique correlation between pRFs in the SPAs with -pRFs (blue) and +pRFs (red) in the PMAs (for example, correlation between +pRFs in OPA with -pRFs in LPMA, controlling for +pRFs in LPMA) quantified using Pearson’s correlation. Each data point represents the z-scored activation on a given trial for all pRFS in the population (that is, all -LPMA pRFS) for a given subject on a trial.
Extended Data Fig. 6 Trial x trial interaction between −/+ pRFs in the place memory areas and scene perception areas exhibit push-pull interaction in independent data.
Recall trials were identical to the trials used in the localizer. Participants fixated on a dot projected in the center of the screen. They were then cued with the stimulus to be recalled for 1 second, followed by a 1 s dynamic mask, and 10 seconds of imagery. Trials were separated by a 4-8 s jittered interstimulus interval. Participants completed 32 imagery trials (16 for each landmark) separated into two imaging runs. One participant was excluded from the analysis for lack of familiarity with the landmarks; the remaining participants were familiar with the locations and had lived in the Hanover area for at least one year. Two participants did not have -pRFs in the ventral surface regions of interest. We tested for the relationship between +/− pRFs in the scene perception and place memory area using the same approach described in the Main text. We examined the unique correlation between the -/+ pRFs in the place memory areas and scene perception areas (that is, correlation between activation of -pRFs in memory areas with pRFs in scene perception areas, while controlling for activation of +pRFs in the memory areas). Using paired t-tests, we found evidence for the opponent interaction between -pRFs and +pRFs in this independent sample. We found that the relationship between the -/+ pRFs in the memory areas with the scene perception area pRFs was significantly different (Lateral – t(8) = 2.61, p = 0.018; Ventral – t(6) = 7.82, p < 0.0001). As we observed in our original analysis, the majority of participants showed a negative correlation in the trial x trial activation of the -pRFs in the place memory areas with pRFs in the scene perception areas (Ventral – 6/7 participants: t(6) = 2.79, p = 0.031; Lateral – 6/8 participants: t(8) = 1.79, p = 0.11). Likewise, most participants showed a positive relationship between activation of + pRFs in the memory areas and pRFs in the perception areas (Ventral – 7/7 participants; t(6) = 7.77, p = 0.0002; Lateral – 7/8 participants; t(8) = 3.30, p = 0.01). This result gives us confidence that our original analysis was not influenced by potential circularity. * = ptwo-tailed < 0.05, *** = ptwo-tailed < 0.005.
Extended Data Fig. 7 Activation during recall of personally familiar places.
Mean BOLD response amplitude relative to baseline when familiar scenes were presented in each lower quadrant (hemifield: ipsilateral and contralateral) for each ROI (OPA, LPMA) and pRF population (+/−).
Supplementary information
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Steel, A., Silson, E.H., Garcia, B.D. et al. A retinotopic code structures the interaction between perception and memory systems. Nat Neurosci 27, 339–347 (2024). https://doi.org/10.1038/s41593-023-01512-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41593-023-01512-3
- Springer Nature America, Inc.
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.