Can thoughts be read from the brain? Neuroscience Contra Wittgenstein

Abstract

Wittgenstein wrote: “No supposition seems to me more natural than that there is no process in the brain correlated with associating or with thinking; so that it would be impossible to read off thought-processes from brain processes.” In general, he rejects what he calls “psycho-physical parallelism.” In Sect. 1, I explain Wittgenstein’s position on this topic and how his followers defend it. In Sect. 2, I argue against Wittgenstein, contending that there is “thought” in a wider sense and that it can be “manifested” without expression. Thought does not need to be ordinarily expressed or even expressible, and it can be read off from the brain, provided one has sufficient technology, data, and understanding of the subject’s interaction with the environment. In Sect. 3, I offer results from brain-science in support of my claims.

1 Wittgenstein on thought and brain

Wittgenstein in the late 1940 s was much concerned with psychological states and activities such as thinking, feeling, remembering, wishing, hoping, willing, and expecting. According to him, they are radically different from physical brain processes so that no correspondence could be found; reading thoughts off the brain would be impossible:

No supposition seems to me more natural than that there is no process in the brain correlated [zugeordnet] with associating or with thinking; so that it would be impossible to read off thought-processes from brain processes. I mean this: if I talk or write there is, I assume, a system of impulses going out from my brain and correlated with my spoken or written thoughts. But why should the system continue further in the direction of the centre? Why should this order not proceed, so to speak, out of chaos? (RPP I 903)

Wittgenstein rejects ideas of “correlation” (Zuordnung), “correspondence” (Entsprechung), and causality that would allow for reading off thoughts from the brain.

Why should there not be psychological regularity to which no physiological regularity corresponds? If this upsets our concepts of causality then it is high time they were upset. (RPP I 905)

Belief in “psycho-physical parallelism” is mere “prejudice” that leads to believing in “a soul alongside the body, a ghostly mental nature” (RPP I 906). To illustrate his point, Wittgenstein employs an analogy:

The case would be like the following—certain kinds of plants multiply by seed, so that a seed always produces a plant of the same kind as that from which it was produced—but nothing in the seed corresponds to the plant which comes from it; so that it is impossible to infer the properties or structure of the plant from those of the seed that it comes out of—this can only be done from the history of the seed. So an organism might come into being even out of something quite amorphous, as it were causelessly; and there is no reason why this should not really hold for our thoughts, and hence for our talking and writing. (RPP I 903)

He suggests that nothing in the structure of the seed corresponds to the structure of the plant, and that similarly nothing in the brain corresponds to the structure of thought. Wittgenstein says that not even God could read our thoughts off from the brain:

If God had looked into our minds [unsere Seelen] he would not have been able to see there who we were speaking of. (PI II, p. 221)

As a matter of method, we should put “the expression of the thought in place of the thought” (PG 140). By this he means linguistic expression: “Language itself is the vehicle of thought” (PI 329).

Wittgenstein’s rejection of the idea of correlation and causal connection between brain and thoughts is related to another one of his rejections, namely that of the will as causal power. He says that our seeing something as something is “voluntary” (willkürlich) and “subject to the will” (RPP I 899), but he criticizes the idea that the will is an influence, force, or “primary action.”

[The will is not] an influence, a force, or again: a primary action, which then is the cause of the outward perceptible action. (RPP I 900)

A child stamps its feet with rage: isn’t that voluntary [willkürlich]? And do I know anything about its sensations of movement, when it is doing this? Stamping with rage is voluntary. Coming when one is called, in the normal surroundings, is voluntary. Involuntary walking, going for a walk, eating, speaking, singing, would be walking, eating, speaking etc. in an abnormal surrounding. E.g. when one is unconscious [etc.]. (RPP I 902)

Saying that one does something “voluntarily” merely means that one does it normally and under normal circumstances, while involuntariness is the exception. There is no sharp definition but merely an open list: we do something voluntarily when we are not forced, threatened, intoxicated, misinformed, unconscious, etc. There is voluntariness, but we do not need to assume a will as an additional substance, force, influence, or primary action. Wittgenstein rejects William James’ idea of the will and voluntariness as some kind of kinaesthetic feeling. He is opposed to James’ “ideo-motor theory” as well as Wundt’s “innervations theory.”¹ He rejects these attempts at reducing psychology to physiology. Instead, he suggests we had better look at the phenomena themselves and their social contexts, such as learning, familiarity, and a mere lack of surprise.²

For Wittgenstein, voluntariness is a certain social behavior and not the causal effect of an act of will, and similarly, thought is something social and linguistic and not a causal effect of events in the brain. It is therefore “natural” to him that there cannot be any physiology of thought.

What drives Wittgenstein’s intuitions in this regard? First, he is familiar with Frege’s idea of a separate “realm of thought” (Reich der Gedanken), a “third realm” (drittes Reich) besides the physical and the psychological.³ If there is such a separate realm, then parallelism with the physical and reading from the brain might seem unlikely. Second, he distinguishes voluntariness from James’ idea of will power and he rejects the latter. Similarly, he distinguishes thought from brain and he rejects brain reading. Third, Wittgenstein makes thought depend on linguistic expression, and linguistic meaning depend on use, which happens in the social sphere outside the brain. Fourth, there is behaviorism, which emphasizes outer manifestation and distrusts first-person talk of the private and inner. But the brain is not private and inner in that psychological way. It is physical, although usually not public. This point is crucial and I will return to it. Fifth, there is the deeper concern that neuroscience and psycho-analysis threaten our ethical behavior, as scientific medicine threatens human practices of comforting the sick.⁴ Thus in 1945, he writes to Malcolm: “Unless you think very clearly psycho-analysis is a dangerous and a foul practice, and it’s done no end of harm and, comparatively, very little good.”⁵ In a 1939 lecture, he says that if a policeman grabs you we will think of this as compulsion, and “if you knew all laws of nature, and could observe all the particles etc.,” someone might say: “He’s as much compelled as if a policeman shoved him.” Wittgenstein comments that we would then give up a distinction, and he says: “I would be very sorry.”⁶ In 1951, he imagined the invention of a mechanical lie detector, and asking himself whether we would change our ways of life, he writes “How could I answer that?”⁷ Thus there are places where Wittgenstein allows for the conceivability of conceptual change resulting from scientific discoveries. But in general, I think he held on to what Klagge called “Wittgenstein’s insulation thesis: science is not relevant to the resolution of philosophical problems.”⁸ This thesis is still alive. For instance, Bennett and Hacker in 2003 write: “philosophical questions are not answerable to scientific investigations.”⁹ In the following, I concentrate on the third point, Wittgenstein’s demand for outer criteria in the form of manifestation as expression. I believe this is central to Wittgenstein and many of his followers.

Hans-Johann Glock follows Wittgenstein when he argues that “mental phenomena must be manifestable”¹⁰ and that such expression must be linguistic:

For one thing, the idea of ineffable thoughts cannot be intelligibly raised within philosophical discourse. If a mystic claims that he has an ineffable belief, it is not only legitimate but obligatory to ask what the content of the belief is. For a belief is something that has content. Now, if the question can be answered at all, however tentatively or vaguely, the belief is not ineffable. […] Even ineffable thoughts would need to have the form: the thought/belief that p. (Glock, 2001, 20)

Thoughts and beliefs must have content, and such content must be of the form “that p.” Thus, there are two steps: “We identify thoughts and beliefs by identifying their content, and we identify their content by identifying their linguistic expression” (ibid.).

Glock defends the demand for manifestation in expression by saying that, if we were asked what we are thinking, we would express our thoughts in words, and if further challenged about what exactly we meant, we would “not reexamine some inner process” but “paraphrase” our utterance (ibid.).

Teichmann, too, follows Wittgenstein and makes similar claims: “What of the criteria for my having thought of Margaret Thatcher? Such criteria lie […] in how I respond to questions, how I explain myself, and the like.”¹¹ Peter Hacker also adopts Wittgenstein’s ideas when he writes: “The limits of what a being can intelligibly be said to think are the limits of its possible behavioural expression of thinking.”¹²

He not only derives the limits of thought from linguistic expression, but also its structure (if he admits such a thing at all):

If one has thought through an argument, then the expression of what one thought will be an ordered sequence of sentences. In so far as there is anything that can be called ‘the structure of thought’ or ‘the structure of thinking’, it is the structure of the expression of the argument which is thought through. (Hacker, 2013, 372)

Hacker says thought is not thinking in images or ideas.¹³ Similar to Glock, he demands expression and content. For Glock, Hacker, and Teichmann, thinking is thinking of, about, or through something. It has content p, which is different from physical brain states, or brain “signals,” as Hacker puts it: “Whatever thinking of, thinking about, thinking through and thinking up may be, they are surely not emitting BOLD (blood-oxygen-level-dependency) signals.”¹⁴ He even says that nothing needs to be going on in our minds as far a thought is concerned:

At any given moment while Le Penseur is thinking his way through a problem, nothing need be going through his mind, no mental imagery, not talking to himself in the imagination—and yet he is thinking continuously for all that. That he is thinking is determined by the context, by what happened previously, by what he can consequently do, as well as by what he would do were such-and-such circumstances to arise. (Hacker, 2013, 371)

As Wittgenstein points to the “history” of a seed and the plant that comes from it, so Hacker points to “what happened previously” and “what he can consequently do.” Nothing “need be going through his mind” that would determine what he thinks, as nothing can be found in the seed that would determine what kind of plant will grow from it.

Similarly, Severin Schröder follows Wittgenstein when he says “in philosophy it is rather misguided” from the start to call “thinking […] an activity or process.”¹⁵ Thinking simply “lacks the ‘micro-structure’ of a typical activity.”¹⁶ A thought is not a thing and not an activity either, he argues. “Thinking is not a process going on in my mind, insofar as the thoughts cannot be recognized or read off there.”¹⁷ Similar to Hacker and Glock, Schröder says “appropriate expressive behaviour is the criterion by which we ascribe thoughts and thinking to a person.”¹⁸

Teichmann adopts Wittgenstein’s idea when he says we can imagine that normal people have no brain and hence that brain reading is impossible, because there is no conceptual necessity of any realization of thought or pain in brain states. “The question I am raising is whether it is even true that someone’s pain must be ‘realized’ by any internal state at all.”¹⁹ He says “there is nothing conceptually amiss, so to speak, in the hypothesis of the brainless person who is one of us.”²⁰

Teichmann is inspired by Wittgenstein’s famous saying that sensation “is not a something, but not a nothing either” (PI 304). He argues that “speaking of remembering, or intending, or of being in pain, as processes or states” is speaking of these phenomena as a something and gets us “started down a wrong path.”²¹ He argues that if someone inherits a house, there is no particular event that corresponds to that person’s inheriting the house.²² Thinking, Teichmann suggests, is like inheriting. No particular process corresponds to it, but much context is required (a whole history, as in the analogy with the seed).

Similarly, Tim Thornton argues that “content cannot be given a reductionist explanation by appeal to brain processes” and that therefore “no account can be given in which thought-processes might be read off from brain-processes.”²³ Norman Malcolm, in his essay “The Mystery of Thought,” follows Wittgenstein in arguing that psychological states such as expectation and fulfillment “make contact” in language.²⁴

Many more examples in this vein could be provided, but these should be sufficient to motivate the discussion. I think that under the influence of Frege and Wittgenstein, these thinkers tend to identify thought with language and to take language as existing in a “third realm” (Frege) or only in social interactions (Wittgenstein) outside of us. This tendency and idea are still alive today.

In the following, I concentrate on thought, which I take to be a paradigmatic psychological state and which Wittgenstein believes must be expressible in language. Call this claim “TE.” He also believes thought is separate from the brain. Call this “T/B.” I argue against both claims. More generally, Wittgenstein believes psychological states must be expressible in behavior. Call this “PE.” He also thinks that such states are separate from brain states: “P/B.” Insofar as psychological states such as feeling, remembering, wishing, hoping, willing, and expecting involve thought, my arguments against TE and T/B generalize to arguments against PE and P/B.

2 Beyond Wittgenstein: More thought and more manifestation!

In this section, I argue for three interconnected claims: there is unconscious and pre-linguistic thought; it can be read from the brain; and there are two different kinds of manifestation. I argue for this independently from findings in neuroscience that I present in Sect. 3, where I confirm my claims. Ordinary expression of thought is “inside-out manifestation” (IOM). It is dynamically “in the loop” with brain, body, and social interaction. But there is also “outside-in manifestation” (OIM), which consists in technology-supported brain reading. This is artificial, not “in the loop,” and not intended by the subject. It is manifestation via observation. IOM occurs in first and second-person perspective: I think and I express my thoughts to you. OIM occurs in first and third-person perspective: I think and someone reads from outside. Both are manifestations, though of different kinds.

One might object that brain reading will be even less possible if we allow for an extended notion of “thought,” taking thought to be an iceberg, the largest part of which is not visible but unconscious, not ordinarily expressed, or even inexpressible. But there are good reasons to accept such a wider notion of thought while still maintaining that thought can be read from the brain. The two points are connected, because there is much going on in the brain that we may want to call “pre-linguistic” and that machine learning makes accessible. Hence, I argue for both a wider notion of thought and the possibility of reading thoughts from the brain.

To be able to “read” and make sense of fMRI pictures and other brain data requires knowledge of the subject’s interaction with the environment. It requires some record of the brain in relation to perception, expression, and action. How far such reading will go remains to be seen. Wittgenstein’s emphasis on use and practice thus applies in novel ways. I think (agreeing with Wittgenstein) that Martians who know nothing of our lives will not be able to read thoughts from our brains. But given such knowledge, I do not see any limits (contrary to what appears “natural” to Wittgenstein). Thus, in a sense, thoughts are not in the brain, because they have their meanings in relation to the world outside. But in another they can be read from the brain by whoever understands the world around and has enough technology and records at hand. Reading is not reduction.

We sometimes absentmindedly put down our eyeglasses somewhere and cannot remember where we put them. We look for them in various places and think hard, but without success. One hour later, seemingly out of the blue, when absentmindedly sitting in a chair, we suddenly remember where we left them. The mathematician Henri Poincaré recounted that after having worked hard on a mathematical problem, given up on it and gone on vacation, the solution suddenly came to him when stepping onto a bus. The mathematician and physicist Roger Penrose said he often thinks not in words but in pictures and diagrams. I myself have been a mathematician and I had such experiences, both of thinking in pictures and of suddenly knowing the solution as if it came out of the blue.

In the case of the eyeglasses and Poincaré’s anecdote, I suggest we say that we sometimes subconsciously or unconsciously continue thinking. By “subconscious” I mean what can become conscious and sometimes does so. By “unconscious” I include what ordinarily cannot become conscious. The line between the two is not sharp and I will use both expressions according to context. When we suddenly remember or when the solution strikes us, it is not that this arrives from the outside, as the words “Einfall” and “inspiration” might suggest. It is rather that we have been working on it subconsciously and unconsciously, beneath the radar of our awareness. It came from outside our conscious sphere, but not from outside ourselves. Such subconscious thought is nothing special. We never have full conscious control over our minds anyway. Conscious thought is only the tip of the iceberg of what is going on mentally. When giving a speech without preparation, ideas often come to us spontaneously and in unforeseen ways. They are then expressed in words; and the demand for expression is stimulating. But they do not come out of nowhere, nor do they come out of “chaos,” as Wittgenstein insinuates. System and order exist not only in social practices, where Wittgenstein expects to find them, but also in the brain, as I hold against Wittgenstein and those who reject any kind of psycho-physical parallelism. There are causal loops interconnecting brain, body, and environment, and there is much meaningful structure and internal looping to be found within the brain already.

Our brains make thousands of “calculations” when we look around, listen to music, attend to lectures, or even just take a walk or ride a bicycle. There are no symbols in our brains and hence strictly speaking, brains do not “calculate.” Only human beings do that. But extension of the term is permissible and innocent, as long as we do not slide into imagining that brains by themselves think and know something the way we human beings do. Calling such calculation in the brain “thinking” is a stretch, but even this we can accept to some degree. Thinking and perceiving are interwoven, ontogenetically and phylogenetically. Prototype and stereotype theories of concepts that are based on exemplars from perception are evidence for this, as are discussions about conceptual content in perception.²⁵ Similarly, thinking and speaking are interwoven. Discussion about the embodied mind has revealed that thinking is interwoven with bodily action. It is embodied, embedded, enacted, extended (4E). Thought, meaning, and language are not just outside in the social world. Nor are they in a “third realm.” The brain carries out “calculations” when we perceive, think, and speak, and most of this we are not aware of. What we are conscious of occurs at a higher level. Propositional expression is part of that higher level. There are no sharp lines between perception, thought, speech, and action, and there are also no sharp lines between higher and lower levels either. This is roughly my position about the nature of thought, and I will show that taking such a position and allowing for a wider notion of thought makes sense and allows for substantiating the idea of reading thoughts from the brain.

A pianist has to practice until she can let her fingers run without paying too much attention to their movements. Something similar can be said about a mathematician and the way he works. Poincaré had to study before he could solve problems. The solution in Poincaré’s case, and the beauty and ease of the playing in the pianist’s case, do not come out of nowhere. Poets and painters have to practice, paying attention to scenes and atmospheres and trying to find the right words, shades, and colors. The more they master their skills, the less they need to pay attention to causes and techniques. Skills are learned and practiced consciously, but when mastered we let them sink to subconscious levels to make room for new tasks to focus on. Thinking, speaking, and writing are similar to perception and bodily skills in this respect. Learning how to write can be a demanding process, requiring much practice and memorization, but once learned, writing becomes second nature and works automatically. Writing and speaking then become guiding features of thought. Sometimes we think better when holding a pen in our hand or having a computer keyboard in front of us. This is not just a metaphor. There is much mental activity that is interwoven with bodily activity, as embodied-mind theories (4E) have shown. Thinking is one of them. We hardly have words for features that are below the level of consciousness. When they become accessible via technology such as fMRI and computers with deep-learning programs, we will discover new features and invent new words. Some will become everyday and our practices and our self-understanding will change. Some of this has already happened with the invention of writing, printing, photography, cameras, telephones, computers, smartphones, and AI. Looking into people’s brains will be another step.

What was going on just before I suddenly remember where I put my eyeglasses, or before the solution of the mathematical problem suddenly appears to me, or before I suddenly know how to go on in my writing, speaking, or composing? I usually do not know. What was going on seems foggy and vague. I cannot trace it back to its source. Hence, I might be tempted to ask, as Wittgenstein did, “why should the system continue further in the direction of the centre? Why should this order not proceed, so to speak, out of chaos?” (RPP I 903). By means of introspection we cannot see where our thoughts come from. We do not feel what is going on in our brains, nor do we ordinarily look there. But this is no reason to give up on the idea of causality regarding thought. Nor is it a reason to limit thought to linguistic expression. Instead, we should both widen the scope of what counts as thinking and investigate the causes. This will reveal new systems of thought in a wider sense. There can be systems before and systems after the expression. We will have to find out how they are related. Asking, “why should the system continue further in the direction of the centre?” (RPP I 903) and believing it does not, is giving up too soon.

On first blush it is true that, as Glock said, if one were asked what one was thinking, all that one could do is spell it out in words and thereby admit that thought has linguistic content and can be expressed in language. But there is an alternative, namely to simply say that one does not know exactly what one was thinking and still take it that one was in fact thinking and that thought is more than what one ordinarily can express in words. Of course, thought needs context. But I suggest externalist arguments from Wittgenstein, Kripke, Putnam, and Burge can be extended to the pre-linguistic level and in the opposite direction, tracing causes back into the brain.

When someone is asked to clarify what exactly she means, she can give answers. The causes of the sounds she produces are in her body, especially the brain. Machines can transform the uttered sounds into writing (we already have such machines), and additional machines that go deeper and set in earlier in the causal history can transform the causes of these sounds directly into sounds and writing (we are working on developing such machines). This is a way of reading from the brain by tracing the sources of speech and action. Reading pre-linguistic thoughts will be just a further step. Computers and machine learning begin to allow for this. The deeper we go into the brain, the more distant the structures will be from ordinary language. That is why “reading” requires widening the scope of “thought.”

Thought is tuned to the environment in which we grow up, move, and act. It is in tune with ordinary expressions and forms of life. The physical (causes) and the psychological (thought) are not as separate as Wittgenstein believed them to be. Imagining thought without brains, something Wittgenstein and Teichmann do, is not consistent with the kind of thought we human beings have. Imagining such thought is empirically uninformed and based on ignorance, abstraction, projection, or idealization as is Frege’s “third realm.”

In order to understand and read mental “calculation” and “thought,” one must know the processes and practices in which they are used. One must know mathematics, poetry, language, chess, or Go. One must have some ideas about the general history of the seed, so to speak, what went on before and what will come out of it. One must know the general pattern. But one does not need to know all historical details of the given seed. This is again like reading in a book. One must know the language, but not all the details of the author and his or her environment. I do not want to say there is Fodorian “language of thought.” Nor do I think that all brains are alike. Brains have their idiosyncrasies. A brain-reading device must be trained and fine-tuned to the brain it is supposed to read from. Thus, brain reading is not as universal as is reading English or German.

Wittgenstein’s seed analogy is misleading. As a matter of fact, biologists do know how to read off from the seed the kind of plant that will grow from it. Biologists usually observe the “history” of seeds and plants to arrive at a better understanding of the causal processes and to find suitable classifications. But once this is done, they can read off from a given seed the kind of plant that will grow from it. What is more, even without having observed the cycles of seeds and plants in any great detail, knowing that there are such cycles and having a very close look at the seed with a microscope will allow them to tell what the effects will be when the seed is put into the ground and given enough water and sunlight. It will allow them to say what kind of plant will come from it. Thus, the analogy is not a helpful one. Seeds and brains are not “amorphous,” and there is no “chaos” in the brain. Imagining this ignores the facts. Wittgenstein imagines that as far as the structure of a plant is concerned there is chaos in the seed, and he says that “there is no reason” why this should not equally apply to thoughts in relation to the brain. But imagining this and saying so is not an argument. One would have to show that there is no structural connection.

On the one hand, it takes months and years for a plant to grow from its seed, and the process is subject to many external influences, but from thought to speech it is not that far. It takes a few seconds and all relevant causes are in the head. Hence, reading thoughts from brains should be easier than knowing from a seed how exactly the plant will look (and not just what kind of plant it is). On the other hand, brains are more complex than seeds and it will therefore be more difficult to read. But none of this contradicts the possibility of reading thoughts as causes of my utterances from the brain. There are calculations and pre-linguistic “thoughts” that never reach the level of consciousness. Making sense of them and deciding what to count as “thought” in a wider sense will be difficult, but not impossible.

Glock has already criticized the seed analogy. He follows Peirce’s theory of signs and says that indices are “connected to what they ‘represent’ by causal dependencies or by other natural relations such as spatial or temporal proximity.” More specifically, he says: “Patterns of neural firings are certainly indices of external phenomena, but only for observers with neurophysiological measuring equipment, not for ordinary subjects of thought.”²⁶ This is the line of thought I am pursuing. What is going on unconsciously in our minds is so far mostly terra incognita, and how to best classify what we discover in the brain is something we will decide about later when we discover and understand it. With the aid of computer-supported technology, we will develop criteria along the way. We will “make up the rules [of brain reading] as we go along,” we might say, quoting and extending Wittgenstein. This will be a new language game in the natural sciences and to some extend it will become everyday. It is a game we are beginning to create.

Unconscious “thought” does not need to be ordinarily expressed. It does not need to be thought of the form “that p,” where “p” is a proposition for which we have everyday words. Locke and Penrose were right when they said we sometimes think in images. This of course might vary from person to person.²⁷ What is more, the vehicle of thought does not need to be images or words at all. It can be structures and patterns of neural activities that develop in interplay with the environment. Such patterns can be discerned when observing the brain in action, and they can be classified (comparable to the classification of seeds). Wittgenstein, Hacker, Schröder, Glock, and even Penrose are looking at the structure of thought when it surfaces and becomes conscious. Then we naturally say that we think in words or images, because that is all we are aware of. Similar to Wittgenstein, they rely on ordinary expression and introspection, while I suggest that we also look at the activity of “thought” below the surface and threshold of consciousness with the aid of machine learning. We can look outside-in. This requires allowing for a wider notion of “thought.” When tracking the causes and using such technology, we leave the ordinary linguistic domain. We move from conscious thought to unconscious “thought” in a wider sense.²⁸

From a linguistic perspective, Ray Jackendoff suggests that “meanings are mostly unconscious.”²⁹ He points out that there are many linguistic rules that speaker and listener follow but are not aware of, and he extends this from the phonetic and syntactic to the semantic. He allows for “unconscious thought”³⁰ and argues for its richness: “In addition, unconscious representations if anything must be more highly articulated than those that appear in consciousness, not more ‘primitive’.”³¹ It is the unconscious thought that is rich and does the real work.

Expecting “thoughts” to be everyday linguistic phenomena is too narrow. It is like trying to understand another culture by limiting oneself to asking whether they have what we have. This would make one blind to what they have that we do not. It is also like trying to understand animals by asking whether they can do what we can do instead of closely observing what they actually do. We mostly do not have words (yet) for the kind of “thought” computer technology will allow us to discover. If one day thought-reading devices are available for everyday use and not only in laboratories, our forms of life will change. What this will be like is another story.

3 Neurological evidence

Hurlburt et al., (2017) offer a brief sketch of the history of psychology, saying that the first period (1879–1925), was marked by an emphasis on introspection (Wundt, Titchener, Külpe); the second (1925–1979), by behaviorism and the suppression of introspection; and the third (1979 to the present), by a resurgence of interest in introspection, mental content, and mental processes. William James was part of the first period, Wittgenstein part of the first and second and later turned against introspection, as did the behaviorist Skinner. But this second period has passed and ceded to a third that looks more favorably at introspection again. Hurlburt et al. argue against the idea that “because inner experience is private, it and descriptions thereof cannot possibly be of high fidelity.”³² Today, methods of “Descriptive Experience Sampling” (DES) have been developed with the aim of apprehending and describing “pristine” (natural) inner experiences with high fidelity. DES relies on “examining pairings of moments of experience and brain activations in multiple ways.”³³ For instance, “Susan’s apprehensions of her pristine experiences include a range of completeness in the inner expression of words, ranging from quite completely expressed with explicitly apprehended prosody to innerly speaking with implied words to thinking without words at all.”³⁴ Hacker, Teichmann, and others might object that saying so (“thinking without words”) does not make it so. But saying so has become commonplace in neuroscience, and distinctions between different mental phenomena are being made and correlated with differences in brain states.³⁵ Furthermore, tracking the causes leads to pre-linguistic “thought.”

Smallwood et al., (2012) take up ideas from William James. They approvingly quote him saying: “consciousness often neglects immediate perceptual input to focus on an internal train of thought.”³⁶ They show how internal thought that occurs at the cost of outward perception corresponds to a certain “cooperation between autobiographical information provided by the default mode network [DMN] and a frontal parietal control network [FPN] which helps sustain and buffer internal trains of thought against disruption by the external world.”³⁷ It is hypothesized that “the FPN and the DMN cooperate to produce the internal train of thought.”³⁸ Thus, models are being built and tested to show that well-known phenomena of competition between perception and inner thought correspond to a certain neurological operations.

In general, a basic goal in neuroscience is to show what (content) is represented how and where (in the brain). Sometimes the emphasis is on the what, sometimes on the how or where. I will now give examples.

Tusche et al., (2014) distinguish between positive and negative values of affective content and they predict the difference from brain activities: “We demonstrated that the valence of unconstrained affective thoughts can be reliably predicted based on neural activation patterns.”³⁹ This involves a decoding and identification of such content.⁴⁰ Here, a valence of thought, positive versus negative, is read from the brain.

Karapanagiotidis et al., (2016) focus on the imagination of past and future situations (mental time travel). They investigate “the neural landscape that allows us to imagine distant times and places.”⁴¹ This is not yet reading, but focusing on a specific content and tracking it, as the title of the paper promises: “Tracking thoughts.”

Morton et al., (2021) distinguish between knowledge of places and knowledge of people. They track where such knowledge is represented and how the distinction can be read from the brain: “We found that the semantic knowledge about people and places is represented in distinct anterior temporal and posterior medial brain networks, respectively.”⁴² Here, a distinction in thought, between people and places, can be read from the brain.

Huth et al., (2016) distinguish between twelve semantic categories and track corresponding brain activities. They “visualized where each of the 12 categories appeared in the shared semantic space [in the brain].”⁴³ It is possible to “determine which specific domains are represented in each voxel.”⁴⁴ They even show that “the organization of the semantically selective brain areas seems to be highly consistent across individuals.”⁴⁵ Thus a distinction in thought, between twelve different kinds of content (categories, semantic fields, concept vectors), can be read from the brain.

Wang et al., (2017) choose 240 propositions that are event or state descriptions. They arrive at 42 “neurally plausible semantic features.” Their aim is “to develop a mapping between a semantic characterization of a sentence […] and the resulting brain activation pattern that occurs when the sentence is read.”⁴⁶ They develop a predictive model and computational theory focusing on complex thoughts. Here, the aim is to develop a predictive bidirectional mapping between neural patterns and semantic contents.

Reber et al., (2019) report that “cells have been found to increase their firing rate in a response to, for example, the famous tennis player ‘Roger Federer’, whether his name is spoken by a computer voice or a picture of him is presented on a computer screen.”⁴⁷ They show that this is not an all-or-nothing phenomenon, and they track semantic categories on multiple levels of abstraction (apple, fruit, food, natural thing): “We found that neural activation patterns contain information on higher levels of categorical abstraction rather than just the level of individual exemplars.”⁴⁸ Here it is shown that “superordinate categorical level seems predominant”⁴⁹ and semantic categories are being tracked.

Horikawa et al., (2013) “read” visual imagery from the brain during sleep. They first rely on subjective reports of patients after awakening and use machine learning to correlate brain activities with these reports as well as with images in normal perception. They then read written outputs from computers that have learned how to “read” from the brain. Thus, they provide “a means to uncover subjective contents of dreaming using objective neural measurement.”⁵⁰

Quiroga et al., (2014) use ambiguous images to predict how we understand them. The idea is that we always see something X as something Y. If there is ambiguity, it is up to us whether we see it as Y₁ or Y₂. The study shows that the choice can be read from the brain: “Neurons […] signal the subjects’ perceptual decision.”⁵¹ Here, a distinction in perceptual-conceptual content, Y₁ versus Y₂, is read from the brain.

Makin et al., (2020) are inspired by the very idea of machine learning. They observe that “the task of decoding speech from neural activity” is similar to “the task of machine translation.”⁵² It is all pattern recognition. They “train […] on neural signals obtained from the electrocorticogram (ECoG) during speech production, and the transcription of the corresponding spoken sentence.”⁵³ The result is that “spoken speech can be decoded reliably from ECoG data […] with 250 word vocabularies.”⁵⁴ Words and not just sentences can be “identified” and therefore “generalization to decoding of novel sentences is possible.”⁵⁵

Moses et al., (2019) show that “speech content can be decoded directly” from the brain, if a limited vocabulary of answers and questions is used and the system is trained on the individual brain. Questions such as “Which musical instruments do you like to listen to?” and “How is your room currently?” had as possible answers: “Piano,” “Violin,” “Electric guitar,” Drums,” “Synthesizer,” “None of these,” “Bright,” “Dark,” “Hot,” “Cold,” “Fine.” They could “demonstrate that high-resolution recordings directly from the surface can be used to decode both perceived and produced speech in real-time.”⁵⁶ That is: “Using only neural signals, we detect when participants are listening and speaking and predict the identity of each detected utterance.”⁵⁷

Joaquín Fuster (2013) writes of “cognits,” which are “networks of cortical cell assemblies or smaller nets representing, as a unit, an item of memory or knowledge.”⁵⁸ He even identifies them with such items: “The cognit […] is a piece of knowledge or memory in the form of a distributed network of neurons of the cerebral cortex.”⁵⁹ Thus “the cognit ‘apple,’ for instance, consists of a network that associates by prior experience the neuronal representations of certain sensory or semantic qualities, such as the color red (or green or yellow), the spherical shape, a given flavor (taste and smell), and a word or symbol.”⁶⁰ “Linguistic cognits” are “nested” and “hierarchically organized like perceptual and executive cognits.”⁶¹ Here, neural features are organized similarly to the way ordinary concepts are organized regarding extension and intension. This is not yet reading. But models are developed and tested.

Monti et al., (2012) offer neurological evidence showing that “the role of language in deductive reasoning is confined to an initial stage” and further development is “not based on the neural mechanisms of natural language.”⁶² Thus, in the brain logical arguments are not carried out as linguistically structured as one might think.

Weineck et al., (2020) investigate how bats use voice for two different purposes: echolocation and communication.⁶³ Neural circuits fire rhythmically preceding the vocalization, and a difference in rhythm correlates with the difference between echolocation and communication. A computer can be trained to do the calculation in real time and make the prediction. They express the belief that “brain rhythms could be linked to sound production in mammals in general.”⁶⁴ Here, a difference between “speech” (echolocation versus communication) is shown to correspond to a difference in rhythms in neural firings. Echolocation is not speech, nor is communication between bats human communication. But this result shows that we are getting closer.

Theories such as Fuster’s develop models, as do theories of connectionism, parallel distributed processing, and neural networks. Machine learning makes it possible for us to track concept vectors and semantic fields that go beyond ordinary language. Thus, such models and vectors support the idea of thought being an iceberg. The models are tested, and neuroscience consistently pursues the idea that the “environment is largely within us”⁶⁵ because we are embedded in the environment through phylogenetic and ontogenetic cycles and loops. On the one hand, embeddedness goes well with Wittgenstein’s emphasis on practice and use in the outside world. On the other, it leads to reading from the brain, because the brain is a particularly rich part of such loops.

Saying in defense of Wittgenstein that he was concerned with thought more abstractly and in somehow conceptually pure and a priori ways (as I think Teichmann does), would, I think, not be a good move. Instead of holding on to abstract ideas, we should be more open to empirical discoveries about the nature of thought.

Insisting on manifestation is not wrong. But there is more than inside-out manifestation (IOM) in the form of ordinary expression or introspection. There is also manifestation via brain reading, which I call “outside-in manifestation” (OIM). It is something that happens to the subject from the outside, while expression is something done intentionally by the subject. Expression is in the loop with the outside world. It is naturally imbedded in patterns of life through social interaction, while brain reading is not natural. Brain reading is without such interaction and not in the loop (at least so far). Another difference is that this OIM tracks the neural causes of expression and thereby reveals mental “calculation” (as in perception) and unconscious and pre-linguistic “thought” in a wider sense. Neuroscience relies on concept vectors and machine learning to do the tracking and reveal deeper structures.

4 Conclusion

Conscious expression in speech and gesture is only a small part of mental activity. It is the tip of an iceberg. Much “thinking” goes on unconsciously before it surfaces. Unconscious and pre-linguistic processes have their own richness for which we do not have ordinary words. Machine learning can be sensitive to it. It will be interesting to arrive at an understanding of the neural activities that are relevant for the development of ordinary thought and that we will be willing to call “thought” in a wider sense. Thought is anchored outside and inside, in social networks and neural networks. Observing the brain of a bat in isolation does not tell us what it is like to be a bat. But observing it in combination with the lives of bats helps us make sense of what bats “think.”

Wittgenstein rejects the idea of psycho-physical parallelism. It is true that the physical brain is only a part of the larger loop of action that connects brain, body, and the outside world. It is also true that the brain reader must be a human being who understands our forms of life. In that sense, thoughts are not in the brain and make sense only in relation to the outside world. But this does not make reading thoughts from the brain impossible. We can read a book even though there is only ink and paper. In the brain, we find icons and indices instead of symbols. They are physical and related to the environment in ways that allow us to say something about them in terms of that relationship, i.e. “reading” them. Neuroscience gives evidence for this. This is how one moves from the psychological (forms of life) to the physical (the brain) and vice versa. I am not sure whether we want to call this a “parallelism,” but it explains interconnections and the possibility of reading thoughts from the brain.

Thinking does not need to be like “having an opinion” as Hacker says,⁶⁶ although sometimes it is. Thinking is not like “inheriting a fortune” as Schröder says.⁶⁷ Nor is it like “inheriting a house” as Teichmann claims.⁶⁸ Rather, it is an activity, as Hanfling puts it.⁶⁹ It is not true that “nothing” corresponding to thought need be going on in Le Penseur, as Hacker says. Nor is it true that there is nothing “conceptually amiss” in the idea that normal people might have no brains, as Teichmann suggests. Such claims ignore empirical facts about the thoughts we have. It is not true that thought “lacks the ‘micro-structure’ of a typical activity,” as Schröder claims,⁷⁰ although it lacks it via introspection. Thought has its own “micro-structure,” namely, patterns of neural activities. It is not true that thought must have ordinary linguistic content and be of or about something ordinarily expressed in language. There are concept vectors and neural patterns. The idea that language is the “vehicle” of thought is too narrow. The limits of ordinary expression are not the limits of thought. Rather, there is unconscious thought that we do not have everyday words for. Such thought has its richness in patterns of neural activities that are not ordinarily expressed. Thought of this kind can also be found in non-linguistic animals. We are rational animals, and primarily animals. Brains, bodies, and the environment are connected in causal loops, and it is a matter of empirical investigation how much can be traced, and where. Reading patterns requires knowledge about forms of life, the environment, and the way the brain functions in this environment—just as reading from a seed requires knowledge about the environment and knowledge of biology. But it does not need to be knowledge of the history of that particular seed.

Kant said that concepts are predicates of possible judgments. Frege emphasized that words have meaning in the context of sentences. From Wittgenstein we can learn that meaning is use and realized in practices and forms of life. I suggest we further expand these views to include unconscious “thoughts” and their manifestation in the brain. Such “thoughts” have meaning in their environment: the brain, the body, and the outside social and physical world. But given some basic knowledge about this environment and using machine learning to see how neural activity is in tune with it, we can learn how to read thoughts from the brain. How detailed that background knowledge of the environment and the individual brain with its individual history has to be, and what exactly we are ready to call “thought,” we will find out. But language is not the only vehicle of thought. Neural activity in its natural environment is another.