Skip to main content
SpringerLink
Log in

Decoding Imagined Sound

  • Chapter
  • First Online:
Sounds from Within: Phenomenology and Practice

Part of the book series: Numanities - Arts and Humanities in Progress ((NAHP,volume 18))

  • 448 Accesses

Abstract

The experience of hearing sounds not present in the air around us has been explored by composers, artists, and researchers for decades. Understanding and communicating the experience of these imagined sounds is often at the core of these explorations. This chapter presents guidelines and best practices for designing frameworks to decode imagined sound. The guidelines explore various design considerations and provide a detailed taxonomy of imagined sound for additional clarity. The second-half of the chapter details the design and results of a contemporary example of the decoding of imagined sound through a brain imaging study where decoding involves machine learning of the pattern of brain activity associated with specific properties imagining of sound, such as pitch-class and timbre, and then estimating those properties from novel data. The discussion of the study outlines the successful decoding of pitch-class and timbral information from brain scans of trained musicians imagining individual musical notes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Image here refers to the mental representation of the object which can include multiple senses simultaneously and does not necessitate an imagined visual component.

  2. 2.

    Specifically, the front-mic’d legato samples from the Hollywood Brass collection was used for the trumpet (East West 2019a) and the sustained, non-vibrato samples from the Hollywood Orchestral Woodwinds collection was used to generate the clarinet sound (East West 2019b).

  3. 3.

    We use the term perceived in this section strictly as it pertains to this line of inquiry. While other terms, such as primarily-acoustic or cochlear-listening, may have been used here, we chose to use perceived to maintain a clear connection with the preceding literature.

  4. 4.

    The minimum cluster size cut-off was 40 voxels and only included voxels that shared a common face.

References

  • Adrian, Edgar Douglas, and Brian H.C. Matthews. 1934. The berger rhythm: potential changes from the occipital lobes in man. Brain 57 (4): 355–385.

    Article  Google Scholar 

  • Baddeley, Alan, and Robert Logie. 1992. Auditory imagery and working memory. In Auditory Imagery, D. reisberg, ed. 179–197. Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Beisteiner, Roland, E. Altenmuller, W. Lang, G. Lindinger, and L. Deecke. 1994. Musicians processing music: measurement of brain potentials with eeg. European Journal of Cognitive Psychology 6 (3): 311–327.

    Article  Google Scholar 

  • Berger, Hans. 1933. Über das elektrenkephalogramm des menschen. European Archives of Psychiatry and Clinical Neuroscience 98 (1): 231–254.

    Google Scholar 

  • Bigler, Erin D., Sherstin Mortensen, E. Shannon Neeley, Sally Ozonoff, Lori Krasny, Michael Johnson, Lu Jeffrey, Sherri L. Provencal, William McMahon, and Janet E. Lainhart. 2007. Superior temporal gyrus, language function, and autism. Developmental Neuropsychology 31 (2): 217–238.

    Article  Google Scholar 

  • Brodsky, Warren, Avishai Henik, Bat-Sheva Rubinstein, and Moshe Zorman. 2003. Auditory imagery from musical notation in expert musicians. Perception & Psychophysics 65 (4): 602–612.

    Article  Google Scholar 

  • Bunzeck, Nico, Torsten Wuestenberg, KaiLutz, Hans Jochen Heinze, and Lutz Jancke.2005. Scanning silence: Mental imagery of complex sounds. NeuroImage 26 (4):1119–1127. issn: 10538119.https://doi.org/10.1016/j.neuroimage.2005.03.013.

  • Bzdok, Danilo, Gesa Hart, and wigsen, Andrew Reid, Angela R Laird, Peter T Fox, and Simon B Eickhoff. 2016. Left inferior parietal lobe engagement in social cognition and language. Neuroscience & Bio behavioral Reviews 68: 319–334.

    Google Scholar 

  • Cage, John. 1962. The future of music: credo. Music Journal 20 (1): 44.

    Google Scholar 

  • Casey, Michael, Jessica Thompson, Olivia Kang, Rajeev Raizada, and Thalia Wheatley. 2012. Population codes representing musical timbre for high-level fmri categorization of music genres. In Machine Learning and Interpretation in Neuroimaging, 34–41. Springer.

    Google Scholar 

  • Clark, Graeme. 2004. Cochlear Implants, 422–462. NewYork, NY: Springer.

    Google Scholar 

  • Coleridge, S. 1817. Biographia Literaria; or, Biographical Sketches of My Literary Life and Opinions: v.2. London: RestFenner.

    Google Scholar 

  • Crowder, Robert G. 1989. Imagery for musical timbre. Journal of Experimental Psychology: Human Perception and Performance 15 (3): 472.

    Google Scholar 

  • Damasio, Antonio R. 1996. The somatic marker hypothesis and the possible functions of the prefrontal cortex. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 351 (1346):1413–1420.

    Google Scholar 

  • Deutsch, Diana. 1981. The octave illusion and auditory perceptual integration. Hearing Research and Theory 1 (99): 142.

    Google Scholar 

  • Deutsch, Diana. 1991. The tritone paradox: an influence of language on music perception. Music Perception: An Interdisciplinary Journal 8 (4): 335–347.

    Google Scholar 

  • Deutsch, Diana. 2013. Psychology of Music. Elsevier.

    Google Scholar 

  • Deutsch, Diana. 2016. Illusions and Research. http://deutsch.ucsd.edu/psychology/pages.php?i=201.

  • Donnay, Gabriel F, Summer K. Rankin, Monica Lopez-Gonzalez, Patpong Jiradejvong, and Charles J. Limb. 2014. Neural substrates of interactive musical improvisation: an fmri study of ‘trading fours’ in jazz. PLoS One 9(2)

    Google Scholar 

  • Dunn, Robert E. 1997. Creative thinking and music listening. Research Studies in Music Education 8 (1): 42–55.

    Google Scholar 

  • East West. 2019a. Hollywoodbrass. [Online; accessed19-January-2020]. http://www.soundsonline.com/hollywood-brass.

  • East West. 2019b. Hollywood orchestral woodwinds. [Online; accessed19-January-2020]. http://www.soundsonline.com/hollywood-orchestral-woodwinds.

  • Eger, Evelyn, Stefan Robert Schweinberger, Raymond J. Dolan, and Richard N. Henson. 2005. Familiarity enhances invariance of face representations in human ventral visual cortex: fmri evidence. Neuroimage 26 (4): 1128–1139.

    Article  Google Scholar 

  • Esteban, Oscar, Christopher J. Markiewicz, Ross W. Blair, Craig A. Moodie, AIlkay Isik, Asier Erramuzpe, James D Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. 2019. Fmriprep: a robust preprocessing pipeline for functional mri. Nature Methods 16 (1): 111–116.

    Google Scholar 

  • Fisk, Josiah, and Jeff William Nichols. 1997. Composers on Music: Eight Centuries of Writings. UPNE.

    Google Scholar 

  • Gazzaniga, Michael S, Richard B Ivry, and G R Mangun. 2006. Cognitive Neuroscience. The Biology of the Mind,(2014). Norton: New York.

    Google Scholar 

  • Gharabaghi, Alireza, Monika Fruhmann Berger, Marcos Tatagiba, and Hans-Otto Karnath. 2006. The role of the right superior temporal gyrus in visual search-insights from intraoperative electrical stimulation. Neuropsychologia 44 (12): 2578–2581.

    Article  Google Scholar 

  • Gleick, James. 2011. The information: a history, a theory, a flood. New York, NY: Vintage.

    Google Scholar 

  • Gordon, Edwin E. 1989. Audition, imitation and notation: musical thought and thought about music. The American Music Teacher 38 (5): 15.

    Google Scholar 

  • Halpern, Andrea R. 2015. Differences in auditory imagery self-report predict neural and behavioral out comes. Psychomusicology: Music, Mind, and Brain 25 (1):37.

    Google Scholar 

  • Halpern, Andrea R., Robert J. Zatorre, Marc Bouffard, and Jennifer A. Johnson. 2004. Behavioral and neural correlates of perceived and imagined musical timbre. Neuropsychologia 42 (9): 1281–1292.

    Article  Google Scholar 

  • Helthuis, Jasper HG, Albert van der Zwan,Tristan PC van Doormaal, Ronald LAW Bleys, Anita A Harteveld, Annette van der Toorn, Mariana Brozici, Jeroen Hendrikse, and Jaco JMZ wanenburg. 2018. High resolution 7t and 9.4t-mri of human cerebral arterial casts enables accurate estimations of the cerebrovascular morphometry. Scientific reports 8 (1):14235.

    Google Scholar 

  • Herholz, Sibylle C., Andrea R. Halpern, and Robert J. Zatorre. 2012. Neuronal correlates of perception, imagery, and memory for familiar tunes. Journal of Cognitive Neuroscience 24 (6): 1382–1397.

    Article  Google Scholar 

  • Holmes, Jessica A. 2016. Singing beyond hearing. Colloquy: On the Disability Aesthetics of Music 69:542–48.

    Google Scholar 

  • Hunter, Murray, etal. 2013. Imagination may be more important than knowledge: the eight types of imagination we use. Review of Contemporary Philosophy (12): 113–120.

    Google Scholar 

  • Janata, Petr. 2001. Brain electrical activity evoked by mental formation of auditory expectations and images. Brain Topography 13 (3): 169–193.

    Google Scholar 

  • Japee, Shruti, Kelsey Holiday, Maureen D. Satyshur, Ikuko Mukai, and Leslie GUngerleider. 2015. A role of right middle frontal gyrus in reorienting of attention: a case study. Frontiers in Systems Neuroscience 9: 23.

    Google Scholar 

  • Kleber, Boris, Niels Birbaumer, Ralf Veit, Tracy Trevorrow, and Martin Lotze. 2007. Overt and imagined singing of an Italian aria. Neuroimage 36 (3): 889–900.

    Article  Google Scholar 

  • Koyama, Maki S., David O’Connor, Zarrar Shehzad, and Michael P. Milham. 2017. Differential contributions of the middle front algyrus functional connectivity to literacy and numeracy. Scientific Reports 7 (1): 1–13.

    Article  Google Scholar 

  • Krumhansl, CarolL, and Lola L Cuddy. 2010. A theory of tonal hierarchies in music. In Music Perception, 51–87. Springer.

    Google Scholar 

  • Limb, Charles J., and Allen R. Braun. 2008. Neural substrates of spontaneous musical performance: an fmri study of jazz improvisation. PLoS One 3(2):

    Google Scholar 

  • Lin, Lung-Chang, Mei-Wen Lee, Ruey-Chang Wei, Hin-Kiu Mok, and Rei-Cheng Yang. 2014. Mozart k. 448 listening decreased seizure recurrence and epileptiform discharges in children with first unprovoked seizures: a randomized controlled study. BMC Complementary and Alternative Medicine 14 (1):17.

    Google Scholar 

  • Liu, Siyuan, Ho Ming Chow, Xu Yisheng, Michael G. Erkkinen, E.S. Katherine, and wett, Michael W Eagle, Daniel A Rizik-Baer, and Allen R Braun. 2012. Neural correlates of lyrical improvisation: an fmri study of freestyle rap. Scientific Reports 2: 834.

    Google Scholar 

  • May, Lloyd, Sean Paulson, Andrea Halpern, and Michael Casey. 2020. Imagined musical scale relationships decoded from fMRI. In Prep.

    Google Scholar 

  • McKeown, Martin J., LarsKai Hansen, and Terrence J. Sejnowsk. 2003. Independent component analysis of functional mri: what is signal and what is noise? Current Opinion in Neurobiology 13 (5): 620–629.

    Article  Google Scholar 

  • Nanay, Bence. 2010. Perception and imagination: a modal perception as mental imagery. Philosophical Studies 150 (2): 239–254.

    Google Scholar 

  • Naselaris, Thomas,Cheryl A Olman, Dustin E Stansbury, Kamil Ugurbil, and Jack L Gallant. 2015. A voxel-wise encoding model for early visual are as decodes mental images of remembered scenes. Neuroimage 105:215–228.

    Google Scholar 

  • Nolan, Christopher. 2017. Dunkirk.

    Google Scholar 

  • Norman, Kenneth A., Sean M. Polyn, Greg J. Detre, and James V. Haxby. 2006. Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends in Cognitive Sciences 10 (9): 424–430.

    Article  Google Scholar 

  • Oliveros, Pauline. 2005. Deep Listening: A Composer’s Sound Practice. NE: IUniverse. Lincoln.

    Google Scholar 

  • Pfordresher, Peter Q., and Andrea R. Halpern. 2013. Auditory imagery and the poor-pitch singer. Psychonomic Bulletin & Review 20 (4): 747–753.

    Article  Google Scholar 

  • Pitt, Mark A., and Robert G. Crowder. 1992. The role of spectral and dynamic cues in imagery for musical timbre. Journal of Experimental Psychology: Human Perception and Performance 18 (3): 728.

    Google Scholar 

  • Podgorny, Peter, and Roger N. Shepard. 1978. Functional representations common to visual perception and imagination. Journal of Experimental Psychology: Human Perception and Performance 4 (1): 21.

    Google Scholar 

  • Prpa, Mirjana, and Philippe Pasquier. 2019. Brain-computer interfaces in contemporary art: a state of the art and taxonomy.In Brainart, 65–115. Springer.

    Google Scholar 

  • Radua, Joaquim, Mary L. Phillips, Natalia Lawrence TamaraRussell, Nicolette Marshall, Sridevi Kalidindi, Wissam El-Hage, Colm McDonald, Vincent Giampietro, Michael J. Brammer, et al. 2010. Neural response to specific components of fearful faces in healthy and schizophrenic adults. Neuroimage 49 (1): 939–946.

    Article  Google Scholar 

  • Repp, Bruno H. 2001. Expressive timing in the mind’sear. Musical Imagery:185–200.

    Google Scholar 

  • Rosenboom, David. 1977. Bio feedback and the arts: results of early experiments. Journal of Aesthetics and Art Criticism 35 (3): 385–386.

    Google Scholar 

  • Sewitsky, Anne. 2019. Black Mirror: Rachel, Jack and Ashley too. Netflix.

    Google Scholar 

  • Shepard, Roger N. 1964. Circularity in judgments of relative pitch. The Journal of the Acoustical Society of America 36(12):2346–2353.

    Google Scholar 

  • Stober, Sebastian, Thomas Prätzlich, and Meinard Müller. 2016. Brain beats: tempo extraction from eeg data.In Ismir, 276–282.

    Google Scholar 

  • Stober, Sebastian, Avital Sternin, Adrian M Owen, and Jessica A Grahn. 2015. Towards music imagery information retrieval: introducing the openmiir dataset of eeg recordings from music perception and imagination.In Ismir, 763–769.

    Google Scholar 

  • Straebel, Volker, and Wilm Thoben. 2014. Alvin Lucier’s music for solo performer: experimental music beyond sonification. Organised Sound 19 (1): 17–29.

    Article  Google Scholar 

  • Uddin, Lucina Q., Jason S. Nomi, Benjamin Hebert-Seropian, Jimmy Ghaziri, and Olivier Boucher. 2017. Structure and function of the human insula. Journal of Clinical Neurophysiology: Official Publication of the American Electroencephalographic Society 34 (4): 300.

    Article  Google Scholar 

  • Vidal, Jacques J. 1973. Toward direct brain-computer communication. Annual Review of Bio-physics and Bioengineering 2 (1):157–180.

    Google Scholar 

  • Wei, Tao, Xia Liang,Yong He, Yufeng Zang, Zaizhu Han, Alfonso Caramazza, and Yanchao Bi.2012. Predicting conceptual processing capacity from spontaneous neuronal activity of the left middle temporal gyrus. Journal of Neuroscience 32(2):481–489.

    Google Scholar 

  • Whitney, Carin, JamieO’Sullivan MarieKirk, Matthew A. Lambon, and Ralph, and Elizabeth Jefferies. 2011. The neural organization of semantic control: tms evidence for a distributed network in left inferior front al and posterior middle temporalgyrus. Cerebral Cortex 21(5): 1066–1075.

    Google Scholar 

  • Willey, Basil. 1947. Coleridge on Imagination and Fancy. Folcroft, PA: The Folcroft Press.

    Google Scholar 

  • Wolpaw, Jonathan, and Elizabeth Winter Wolpaw. 2012. Brain-Computer Interfaces: Principles and Practice. OUP USA.

    Google Scholar 

  • ZA/UM. 2019. Disco elysium.

    Google Scholar 

  • Zatorre, Robert J., and Andrea R. Halpern. 2005. Mental concerts: musical imagery and auditory cortex. Neuron 47 (1): 9–12.

    Article  Google Scholar 

  • Zatorre, Robert J., Andrea R. Halpern, David W. Perry, Ernst Meyer, and Alan C. Evans. 1996. Hearing in the mind’s ear: a pet investigation of musical imagery and perception. Journal of Cognitive Neuroscience 8 (1): 29–46.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dartmouth College Hanover, Hanover, NH, USA

    Lloyd May & Michael Casey

Authors
  1. Lloyd May
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Casey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lloyd May .

Editor information

Editors and Affiliations

  1. Department of Music—Composition, University of California, Riverside, Riverside, CA, USA

    Paulo C. Chagas

  2. Department of Recording Arts, University of Colorado Denver, Arvada, CO, USA

    Jiayue Cecilia Wu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

May, L., Casey, M. (2021). Decoding Imagined Sound. In: Chagas, P.C., Cecilia Wu, J. (eds) Sounds from Within: Phenomenology and Practice. Numanities - Arts and Humanities in Progress, vol 18. Springer, Cham. https://doi.org/10.1007/978-3-030-72507-5_4

Download citation

Publish with us

Policies and ethics

Search

Navigation