1 Introduction

“To say that an animal responds to sensory stimuli may not be the most natural and efficient way to describe behaviour. Rather, it appears that animals most of the time react to situations, to opponents or things which they actively isolate from their environment. Situations, things, partners or opponents are, in a way, the terms of behaviour. It is legitimate, therefore, to ask what phenomena correspond to them in the internal activity of the brain, or, in other words: how are the meaningful chunks of experience ‘represented’ in the brain?

A crude version of this question takes the form: is the presence of a relevant happening signaled by the activity of just one neuron, which otherwise is always silent, or is it represented by an irreducibly complex description of the activity of the brain?

Following an old idea (Hebb 1949 and older) we shall explore the possibility that it is neither single neurons nor abstract diffuse properties of the state of the brain which correspond to the relevant events of behaviour, but something in between, identifiable sets of neurons.

These ‘cell assemblies’ have recently gained support from neurophysiology in two ways. First, many years of recording responses of single neurons to sensory stimuli have shown that no very complicated or very unique input is needed to activate a neuron. The most efficient stimuli for cortical neurons are rather elementary configurations of the sensory input, such as moving lines in narrow regions of the visual field (Hubel and Wiesel 1959) or changing frequencies in certain delimited regions of the acoustic spectrum (Evans 1968). These ‘features’ cannot independently carry meaning but must be in the same relation to meaningful events as the phonemes of linguistics are to words or sentences. The whole meaningful event must be signaled in the brain by a set of neurons, each contributing a particular aspect which that event may have in common with many other events.

The second line of evidence is derived from the neurophysiology of learning. It was one of Hebb’s points that cell assemblies representing things in the brain are held together by excitatory connections between the neurons of which they are composed, and that these connections are established through a learning process. The most natural way in which such learning could take place is if a statistical correlation, say, a frequent coincidence of a certain set of elementary features in the input were transformed into synaptic connections between the corresponding neurons. Some recent observations on the plasticity of the connections of single neurons (Hubel and Wiesel 1965; Wiesel and Hubel 1965; Blakemore and Cooper 1971; Hirsch and Spinelli 1971) can indeed be explained by invoking such a mechanism.

It seems timely, therefore, to reconsider cell assemblies as a possible substrate of behaviour, and, particularly, to review the cerebral cortex with the idea in mind that it might be the place where cell assemblies are formed and sustained.

There remains one nagging thought, however, when we dismiss single neurons as the elements onto which complex situations, things, etc. are mapped in the brain. Suppose we were recording with a microelectrode from a neuron whose activity corresponds exactly to the presence of a close relative or to a particular bird’s song, or to the memory of a particular scent. The probability that, while we are recording we stumble accidentally upon such very rare stimulus configurations is negligibly small. Thus, strictly speaking, the idea of single neurons as classificatory elements for complex situations is beyond proof or disproof. Of course for similar reasons and for added technical difficulties it is even more hopeless to expect experimental proof of the correspondence of a cell assembly to a certain event in the input. In this field we must rely on theoretical plausibility and on considerations about the structure of networks in the brain that may seem more suited for one or the other scheme.”

Valentino Braitenberg wrote this beautiful introduction to his article on this subject in 1978. Here we want to spell out some of these old ideas about cell assemblies in the cortex in more detail in the light of the neuroscientific advances in the last 30 years.

In a nutshell, it turns out that most of the ideas presented in Valentino’s article have been substantiated and sometimes specified more clearly by this research, and only very few have to be revised.

In a very general formulation these are the main points:

  1. 1.

    Most of the input to cortical neurons comes from other cortical neurons, i.e. the cortex works mostly on its own output.

  2. 2.

    There is an informational imbalance between excitation and inhibition since excitatory connections greatly outnumber inhibitory ones.

  3. 3.

    The global cortico-cortical feedback is positive, since cortico-cortical fibers are excitatory.

  4. 4.

    There is a great dispersion of information in the cortical connectivity system .

  5. 5.

    Cortical connectivity can be described as the cooperation of an A- and a B-system, where the A-system distributes information globally by cortico-cortical fibers and the B-system distributes information locally by intracortical axonal arborizations.

  6. 6.

    There is a predominance of inborn specificity in the wiring at the macro-level (e.g. in the connection of areas) and of randomness (or individuality) at the micro-level.

  7. 7.

    Synaptic plasticity in the cortex is predominantly of the Hebbian type, i.e. excitatory synapses are strengthened by coincident firing of the pre- and postsynaptic neurons.

Now we will spell out these points in more detail also considering some of their further developments and functional consequences.

Building on Hebb’s original, mostly representational arguments for cell assemblies, the further development of cell assembly theory was mainly driven by neurophysiological and biophysical findings concerning the basic neuronal mechanisms and the detailed temporal processes of neuronal activation and interaction on one hand and by computational arguments and requirements on the other.

The neuroanatomical aspects are discussed in the next Sect. 2. Functional consequences of Braitenberg’s points in terms of a theory of cortical cell assemblies and its development from 1977 to the present time are detailed in the following sections. Section 3 describes the underlying network structure and Sect. 4 the ongoing computational process in a simplified abstract way. Section 5 considers the further development of Hebbian synaptic plasticity, i.e. its fine temporal aspects.

2 Results from neuroanatomy

As mentioned in Braitenberg’s introduction, Hebb based his theory on excitatory synaptic connections: he assumed that neurons which are repeatedly active together become more strongly connected, an idea which Hebb himself mentions to be “an old idea”. In this way, correlations in the outside world, such as the different properties of an object, can be inscribed into the brain. As a result, cell assemblies will be formed which are more strongly connected among each other than with other neurons. Braitenberg then elaborates in more detail on the consequences of mutual excitation, such as the ignition of an assembly when only some of its members are activated, or the possibility of associations between different cell assemblies.

An important argument in Braitenberg’s 1978-paper in support of the Hebbian theory was therefore the assumption that pyramidal cells, being the main cell type in the cortex, are excitatory neurons (points 2 and 3 above). He presents the arguments in favour of this, but it is interesting to see that this was not yet entirely certain at the time. In the meantime this is a fact beyond doubt, shown directly in many electrophysiological studies (e.g. Miles and Wong 1986; Thomson and Deuschars 1994; Markram et al. 1997).

The prevalence of excitatory connections has also been confirmed directly since then by showing that not only in the cerebellar (Uchizono 1965), but also in the cerebral cortex (e.g. Houser et al. 1984), synapses with a symmetrical appearance in the electron microscope contain the inhibitory transmitter GABA. These so-called Type-II synapses were known to constitute only a minority of the synapses in the cerebral cortex (e.g. Gray 1959; Colonnier 1968; Peters and Feldman 1976; Wolff 1976; Bär 1977).

These and many other structural properties have been quantified in the following years in the mouse cortex, as well as in other species (For reviews see White 1989; Braitenberg and Schüz 1991, 1998; DeFelipe et al. 2002). These studies provided further support for the theory of cell assemblies; they showed that the structure of the cortex (including the hippocampus) fully satisfies the requirements for this theory, in contrast to the structure of other main parts of the brain (cerebellar cortex, basal ganglia, thalamus). The cerebral cortex is the only large network in the brain which consists mainly of excitatory connections within itself (points 1–3). In addition, these connections are highly divergent and convergent (point 4) and thus allow for a rich repertoire of correlations and associations,as required for this theory.

Braitenberg’s point 4 has been substantiated during the last decades by the large amount of studies with anterograde and retrograde tracers in rodents, cats and primates, showing the connectivity between cortical areas. Many of the data in cats and/or monkeys have been summarized by Young et al. (1995) and in the database CoCoMac initiated by Rolf Kötter (Stephan et al. 2001; Kötter 2004; Bakker et al. 2012). It can be seen, for example, that many cortical areas in the monkey are connected with more than 10 other cortical areas.

The high degree of divergence and convergence becomes even more evident if one quantifies the density of fibers in terminal fields coming from a given injection site, as has been done in a tracer study in the mouse cortex. It could be shown that the axonal branches from an injection site comprising many thousand neurons contribute only a few percent to the total axonal density in the surrounding local terminal field, and they contribute even less than one percent to the axonal density in a distant projection field (Schüz et al. 2006). Thus, many terminal fields from both local and distant regions have to converge at any given place in order to arrive at the total density of axons in the cortex of about 4 km/mm\(^{3}\) (Braitenberg and Schüz 1998).

Another interesting fact shown by these numbers in the mouse is the quantitative dominance of shorter connections (i.e. within an area and between neighbouring areas) over longer ones. This seems to be a general feature in both, small and large brains. Although areas far away from each other can be connected, nearly half of the interareal connections are nearest-neighbour or next-door-but-one connections as has been shown by Scannell et al. (1995) for the cat. A finding in human brains goes into the same direction (Schüz and Braitenberg 2002): there is an inverse relationship between number and length of fibers connecting the cortex in its horizontal plane (Fig. 1).

Fig. 1
figure 1

Estimate of the number of fibers \(N\) (in both hemispheres together) connecting the human neocortex within itself in its horizontal plane (log–log plot). Compartment A population of axon collaterals of pyramidal cells running tangentially within the grey matter over a few millimeters, based on an estimate of five horizontal collaterals per pyramidal cell. Compartment B U-fiber system, connecting the cortex within itself over a few centimeters. Compartment \(C\) all cortico-cortical fibers of longer range, running in the depth of the white matter, i.e. fibers which make short-cuts between folds within the same lobe, fibers which connect the different lobes with each other and fibers of the Corpus callosum. From: Schüz and Braitenberg (2002), with permission

Full size image

For the division into a global and a local connectivity system (point 5), Braitenberg used the term A (apical) and B (basal) system, respectively, in view of the fact that often cortico-cortical inputs from other areas project into upper layers and reach there also the neurons from the lower layers via their apical dendrites. Nowadays one knows that this projection pattern is not general for all cortical areas; for example, in sensory areas it concerns only the feedback connections from hierarchically higher to lower areas (e.g. Rockland and Pandya 1979; Rockland and Virga 1989; Cauller 1995; Rockland 2004).

In addition to a local and a global connectivity system one needs also to include a middle range system in order to do justice to the connectivity in large brains. An indication for this are the horizontal stripes of myelinated fibers within the gray matter which were known in the human cortex since the times of Gennari (1782) and Baillarger (1840) and which contributed to the myeloarchitectonic differentiation of cortical areas (e.g. Vogt and Vogt 1919; for review see Nieuwenhuys 2013). Braitenberg himself spent some time in the Vogts’ laboratory in the 1950s and was attracted by myeloarchitectonics as an expression of differences in the wiring diagrams of the various cortical areas. He provided evidence for the assumption that these stripes correspond to the axon collaterals of pyramidal cells located in the layers above the stripes (Braitenberg 1962). This was later substantiated by Hellwig (1993) who showed how myeloarchitectonics can be generated from cyto-architectonic pictures. It is interesting to see the recent relevance of this old research in connection with current approaches to in-vivo-imaging of cortical myeloarchitectonics (Bock et al. 2013; Nieuwenhuys 2013).

Knowledge about this middle range system has increased considerably during the last two decades. It could be shown that the long horizontal collaterals in the gray matter give off patchy ramifications over a region of several millimeters, usually within the same area, and that the lateral spread increases with hierarchy of sensory cortical areas (e.g. Amir et al. 1993; Levitt and Lund 2002; Voges et al. 2010).

Quite a number of studies have dealt with the question of randomness of connectivity on the micro level (point 6), constrained by the geometry of axonal and dendritic arborizations. This assumption is still a successfully used first approach to understand connectivity (e.g. Binzegger et al. 2004) or—expressed more cautiously by Stepanyants and Chklovskii 2005)—potential connectivity in the cortex. Knowledge about the particular geometry of neurons in different layers gained by Golgi studies over the last century has further increased during the last decades by using intracellular injections of neurons (e.g. Binzegger et al. 2004; Stepanyants et al. 2008) and with it knowledge about the connectivity between layers and cell types (e.g. Binzegger et al. 2007). Electron microscopical studies have confirmed randomness of connectivity in some pyramidal cell systems, but have also shown biases away from randomness in some others (for an overview see White 1989; Braitenberg and Schüz 1998). Physiological investigations have also shown some interesting deviations from the anatomically expected connectivity scheme in case of subtypes of inhibitory neurons (Dantzker and Callaway 2000). A comprehensive review of such investigations and a heroic effort to understand the complex relation between anatomical and physiologically explored data on connectivity has been presented by Potjans and Diesmann (2012).

At the end of this section we want to come back to the “nagging thought” in Braitenberg’s introduction, i.e. the difficulty or even the potential impossibility to prove or disprove the representation of events in terms of cell assemblies as opposed to single neurons empirically. Indeed, today there may be hope: the fascinating technical developments in the field of multiphoton-microscopy and genetically encoded calcium indicators make it now possible to visualize the activity of neuronal populations on the single cell level and over long periods of time. They have opened up the possibility to relate the activity of neuronal populations to behaviour (for reviews see Kerr and Denk 2008; Wallace and Kerr 2010). Furthermore, in a recent study on the hippocampus Liu et al. (2012) were able to induce fear behaviour by optogenetic stimulation of neurons which had been activated during learning of this behaviour, indicating that these neurons were part of the engram leading to this behaviour. Thus, new insights into the representation of events will be possible which could hardly be foreseen at the time when Valentino wrote his article. In addition, new computational methods have been developed that alleviate the combinatorial problem of identifying correlation structures involving several neurons (i.e. more than two) in multi-unit recordings (see Picado-Muino et al. 2013).

3 Global cortical asssemblies

The view of the cerebral cortex as a substrate for Hebbian cell assemblies formed by associative memory mechanisms, i.e. Hebbian synaptic plasticity, was certainly the first point of Valentino’s article. It is consistent with the anatomical observations summarized in the introduction. In particular, the stabilization of cell assemblies by auto-association, a cornerstone of cell assembly theory, requires a prominence of excitatory connections between cortical neurons that are further strengthened by Hebbian synaptic plasticity (points 2, 3, 7). However, there is an additional component in Valentino’s assembly concept, namely that at each moment in time there is only one assembly active in the whole cortex. This view is motivated by the subjective unity of conscious experience and by the idea that the cortex combines all the incoming information from different sensory systems and proprioceptors with the reafference of the motor commands into one coherent representation of the present situation. This idea is substantiated by the anatomical evidence for rich excitatory interconnectivity by the combination of intra-cortical and cortico-cortical connections (points 4, 5).

In his article Valentino goes on to describe the advantages of a cell assembly representation consisting of many neurons distributed all over the cortex versus the naive one “grandmother” neuron representation, and he also speculates about the dynamics of the activation of assemblies in terms of “threshold control”, i.e. of global activation or inhibition of all pyramidal cells in larger regions of the cortex. In this context the formation, detection and production of sequences of assemblies (corresponding to Hebb’s “phase sequences”) was also discussed and even the possibility of a fast temporal structure inside the activation of one assembly was mentioned, but not spelled out in detail. These two levels of temporal structure can be understood as the first stages of a hierarchical temporal structure, which was considered to be functionally meaningful for example in the representation and parsing of language, as mentioned already in Valentino’s 1978 article. The idea of sequencing and/or fine temporal structures of assemblies has been elaborated in particular by (Abeles 1982, 1991), who presented evidence for precise temporal events in multi-unit spike trains which could be explained by hetero-associative temporal structure in assemblies, also called “synfire-chains”. The early ideas concerning assembly dynamics, fine temporal structure, and temporal hierarchies probably needed the most extensive elaboration from a neurophysiological point of view. This has indeed been developed by several authors until the present time (e.g.Bienenstock 1995; Thorpe et al. 1996; Wennekers 1998, 2006; Miller 2000; Rullen and Thorpe 2001; Rao 2005; Izhikevich and Hoppensteadt 2009; Szatmáry and Izhikevich 2010; Humble et al. 2012). Also a more detailed modeling of the 6-layered intra-cortical connectivity structure has been incorporated into assembly theory (e.g. Krone et al. 1986; Hawkins and Blakeslee 2004, see also Potjans and Diesmann 2012). In his book “Neural Assemblies. An Alternative Approach to Artificial Intelligence” Palm (1982a) developed a more detailed computational view of the practical use of cell assemblies in the cortex for solving various problems that may help the animal to survive. This involves speculations about prediction, generating sequences, planning actions, delayed reward and reinforcement learning (Sutton and Barto 1998; Russel and Narvig 2003), hierarchies and many other ideas from the then relatively new field of artificial intelligence (Winston 1984).

These early ideas have been extended in many directions by Valentino’s scholars, including the authors of this article (see Palm 1987b, 1990a, 1993b; Krone et al. 1986; Abeles 1988; Schüz and Palm 1989; Braitenberg and Schüz 1991; Wennekers et al. 1995,Wennekers and Palm 19986 Knoblauch and Palm 2001, 2002a, b; Pulvermüller 2002; Schüz and Braitenberg 2002; Knoblauch et al. 2005a, b; Wennekers 2006; Markert et al. 2007; Wennekers and Palm 2007; Kiefer and Pulvermüller 2012; Shtyrov et al. 2012) Similar ideas have of course also been presented by many other authors during the last 30 years (e.g. Miller 1996; Hawkins and Blakeslee 2004; Rao 2005; Hecht-Nielsen 2007; Lansner 2009 and authors in Aertsen 1993). One important conceptual point in our further development of assembly theory is to view the cortex not just as one big associative memory, but as a network of interconnected local memories. According to the view already sketched by Palm (1990a, 1993a) and elaborated for example by Knoblauch et al. (2005a, b) and Bouchain and Palm (2012), each local “node” in this network can be considered as a piece of cortex ranging in size from a “hypercolumn” (Hubel and Wiesel 1974, 1977, i.e. about 1 mm\(^2\) surface area) to a small “cortical area” (perhaps up to 50  mm\(^{2})\). In reality such a node or “module” cannot be regarded as a well-separated piece of cortex, but is locally connected to the neighboring nodes. Basically each local cortical “column” is connected to its neighborhood with a connection density that decreases continuously with distance perhaps up to a millimeter (in addition to a few millimeters of patchy intra-areal connections that have been observed in some species such as cats and primates).

In our view such a node consists of two associative memories: an auto-associative one, where the local activity patterns are stored or stabilized by Hebbian learning (Braitenberg’s B-system) and a hetero-associative one (Braitenberg’s A-system) which learns associations between activation patterns of some other cortical areas that are transmitted to the local node by cortico-cortical fiber connections and the currently active local pattern. These hetero-associative memories can be used to match the local activation with the incoming activation from other areas yielding strong local activation when everything fits together and weaker activation down to local inactivation if the patterns don’t fit so well. Qualitatively, such a view of the prevalent cortical computation is shared by most of the popular “brain theories” that have been proposed over the years (e.g. Legéndy 1967, 1975; Marr 1969, 1970, 1971; Palm 1982a; Palm and Aertsen 1986; Amit 1989; Bienenstock 1995; Grossberg 1976a, b, 1982, 1999; Rao and Ballard 1999; Edelman and Tononi 2000; Hawkins and Blakeslee 2004; Anderson 2004; Hecht-Nielsen 2007; George and Hawkins 2009; Eliasmith 2013), although the verbal descriptions all use different terminology. An interesting exception seems to be the view proposed by Dana Ballard and some of his pupils. Here again the local activity is matched against the incoming signals from other areas, but the resulting local activity and also the output activity from the local node measures the difference (or dissonance) between the local and global activities, which may also be viewed as the “prediction error”, instead of the degree of agreement (or consonance). In spite of this clear conceptual difference it is not at all easy to discriminate these two views experimentally. Perhaps some of the cortical areas adhere to Dana’s principle and most others to ours, or perhaps many areas can somehow even do both? This last view is detailed in a paper by Knoblauch et al. (2007), where the idea is, roughly speaking, that consonance is the dominant mode of operation and only cortical layer III works according to the difference principle. This model also offers the possibility to reconcile apparently conflicting experimental evidence on this issue. Our global assembly theory also seems to be generally consistent with some very detailed cortical models of the visual system (e.g. George and Hawkins 2009; Weidenbacher and Neumann 2009; Raudies and Neumann 2010; Poort et al. 2012) and therefore can serve as a framework for embedding these models into a unified global brain theory.

One specific point distinguishing our view from most of the other theories is derived from a mathematical analysis of the efficiency of associative memories that are based on Hebbian synaptic plasticity: the stored activity patterns, i.e. the local assemblies in each of the local nodes have to be sparse, i.e. only a very low fraction of the local neurons should be active (Palm 1980, 1987a, 1990b, 2013; Palm and Sommer 1992; Palm et al. 1994; Markert et al. 2008; Knoblauch et al. 2010; Knoblauch 2011). This sparseness requirement seems to fit well to neurophysiological observations (e.g. Waydo et al. 2006).

Since we don’t know which of the current “brain theories” are true, and they are not easy to distinguish, anyway, one reasonable strategy that we are following now is to see how far we can get with the machinery proposed here, i.e. with a network of associative memories.

4 Computation in associative networks

In order to understand the computational potential of the cortical associative memory network it is necessary to consider the temporal dynamics of neural activation in more detail. Of course, this has been intensely investigated in the last 30 years, mostly in behaviorally stimulated or activated cortical areas, and there is also considerable work on the statistics of cortical spike trains (e.g. Abeles et al. 1995; Softky 1995; Bair and Koch 1996; Kenet et al. 2003; Knoblauch and Palm 2005; Renart et al. 2010). For our purpose it suffices to provide a rough qualitative picture of the processes involved herein. The activity in the axonal fibers of this network consists of spikes and sequences of spikes .The average spike-frequency will not be very high (e.g. 5 spikes/s; cf. for example Herz et al. 1964; Burns and Webb 1976 or Renart et al. 2010), even in an activated area, but this means that still the number of spikes occurring in the 10,000 afferent fibers to a typical pyramidal neuron in this network will be 50,000/s, i.e. 50/ms on average. The corresponding postsynaptic potentials (PSP) will be low-pass filtered and averaged with other PSPs on their way to the cell-body resulting in a fluctuating signal (with a variance of perhaps 10–20 % of the mean, depending on the correlation between different spike trains among other things). Even if the local associative memory dynamics (of “iterative retrieval”, see Wennekers et al. 1995; Schwenker et al. 1996) has reached a fixed point in one local node of the network and the activity would be “constant” from a computational point of view, there still would be these fluctuations, and corresponding fluctuations in the timing of the outgoing spikes generated by these neurons. Our simulations of two cortico-cortically connected areas (Knoblauch and Palm 2002a, b or Wennekers et al. 1995) may also give an impression of this “quasi-constant” fluctuating activity. Such a relatively constant local activation pattern can persist for a relatively long time in spite of changes in some of the inputs from other cortical areas or from outside the cortex. This may be useful since it is uncertain for how long the cortico-cortical (or even extra-cortical) inputs activating the local module can be assumed to be “constant” in this sense.

Despite of these fluctuations we will now make use of a common theoretical simplification which results in a discretization of neural activity and time: we will consider the activity reached in one module during this “local settling time” or LST as one “state” in a simplified computational description and the LST as the elementary computational time-step. It is plausible to assume that this “settling” of the local modules can occur several times per second and this settling time may well be different for different modules. To fix ideas we assume here that the LST may be somewhere between 10 and 50 msec. If we assume that the total travel time of neural activity from one neuron to its neighbours is between 1 and 5 msec (e.g., Girard et al. 2001), there can be at least 3 up to perhaps 30 associative iteration steps in one LST. We found (Schwenker et al. 1996) that in auto-associative memory 3 steps are usually enough to reach a “fixed point”, but of course, the local stabilization of activity could also lead into a small cycle instead of a fixed point giving room to some temporal fine structure in the sequence of activation in a local assembly. In this case, of course, the local associations learned in the Hebbian synapses would be hetero-associative between the subsequent patterns of the cycle instead of auto-associative between the neurons in just one pattern (Wennekers and Palm 1996). In each LST step one such pattern (or small cycle) will move into a subsequent one driven by the input coming from the cortical input nodes to our local node. Formally, this can obviously be described as an input-driven state transition in the local node. So each local node can be roughly described as a finite automaton (McCulloch and Pitts 1943; Kleene 1956; Hopcroft and Ullman 1979; Knoblauch et al 2004; Markert et al. 2005) and the whole cortex as a network of these automata. We should add here that the distributed assembly representation of “automata-states” in the local cortical modules provides an important additional property which intuitively is not machine-like at all: due to the overlap between different assemblies there is a notion of similarity between states which may also be used to influence the state transitions (Palm 1981; Knoblauch et al. 2005a, b; Markert et al. 2007) and there is also the possibility to represent a superposition of a few states by subthreshold activity (Knoblauch and Palm 2002a, b).

Theoretically, one reasonably large automaton (provided with unlimited external memory) can do any kind of computation (Turing 1936; Hopcroft and Ullman 1979), so an associative memory network of sufficient size (like a human or monkey or cat cortex) certainly also has this potential. More importantly, the typical behavioral or cognitive tasks of an animal or human agent can well be broken down into processes of matching sensory inputs to learned patterns, thereby inducing state-transitions that may include outputs to other areas or even to motor centers of the brain to produce the appropriate reactions (Palm 1982a; Wennekers and Palm 1996, 2007). Even delayed reactions or longer-term predictions leading to goal-directed behavior can be learned in such structures as shown, for example, by the theory of reinforcement learning, which just requires a simple, biologically plausible learning scheme for the predicted reward (Sutton and Barto 1998) probably realized in the basal ganglia (Schultz et al. 1997; Schultz 2002), and associative structures that relate the current input to the overall situation, the expected reward, and a proposed action (Palm 1982a). Recently we have demonstrated that this associative network structure is also well-suited for simple language understanding and production (Markert and Palm 2006; Markert et al. 2007, 2008). These more elaborated processing schemes require more complex temporal considerations which are paralleled by refined experimental investigations showing corresponding complex temporal biophysical properties and mechanisms in the cortex. These more recent developments are discussed in the next two sections.

From a simplified computational point of view the cortex can now be regarded as a network of local cortical modules where the activation in each module is regarded as “approximately constant” i.e. as in one “activity state” (described by a corresponding pattern of active neurons) for a certain period of time, which we have called the LST.

Since we cannot assume a global clock which determines the duration of all LSTs, we need for each local module a mechanism that determines the end of the current LST and the switching to the next “state”. In Wennekers and Palm (2009) we have discussed such mechanisms leading to the suggestion that states can remain stable for long durations due to the strong auto-associative internal connectivity in the local module and that state-transitions can be triggered simply by increasing background excitation (most likely from the thalamic region that corresponds to the cortical area (e.g. Jones 1985; Brodal 2010). Again, this state switching dynamics can also be directed into different branches, i.e. branch to different next states, depending on the specific excitation from related areas through the A-system. This mechanism can be used to create variable LSTs in different cortical areas and to organize their interplay among the nodes in the cortical network according to the current computational demands. For example, in hierarchical processing of incoming sensory signals or motor behaviors it may be useful to have longer LSTs at higher hierarchical levels combining several shorter LSTs at lower levels. There is actually physiological evidence that short LSTs are typical for early visual or auditory areas whereas longer LSTs, which may also be considered as a realization of working memory, are more pronounced in the frontal parts of the cortex, where sensory-motor patterns can perhaps be integrated over larger periods of time (see for example Huyck and Passmore 2013).

It is hard to create a generic scheme for such a cortico-thalamo-cortical control system. However, the anatomical and physiological basis for such a system exists (Miller 1996; Sherman 2007) and concrete realizations for the task of sentence understanding and also continuous speech recognition (in the Wall Street Journal corpus) can be found in Markert et al. (2007, 2009) and Kara and Palm (2008). For this kind of organization we found it useful and actually sufficient to have a measure for the total or average local activity in each module and to use it for thalamic control by other cortico-cortically related modules (see Bouchain and Palm 2012). But of course the question, how such a delicate control structure could have developed by a combination of genetic evolution and individual learning still remains a mystery to us.

5 Synaptic plasticity and learning

One basic assumption of our view of the cortex as an organized network of local cortical modules is the use of auto- and hetero-associative learning for the stabilization of local patterns in single modules and the establishment of cortico-cortical relations between patterns in different modules. The synaptic mechanism for the formation of these auto- and hetero-associations was supposed to be Hebbian synaptic plasticity (point 7, see also Palm 1982a, b).

Over the last twenty years, synaptic plasticity has been investigated intensely, in particular with respect to the timing of the “coincidence” of pre- and postsynaptic activity. The most prominent view today concerning synaptic plasticity in the cerebral cortex (including the hippocampus) is that it is an ubiquitous phenomenon that seems to be triggered by the timing of the presynaptic spike and the postsynaptic spike that is apparently backpropagated through the dendritic tree of the postsynaptic neuron. This kind of plasticity is called “spike-timing dependent plasticity” (STDP). The by now classical figure 7 of Bi and Poo (1998) shows their experimental findings concerning the increase or decrease of synaptic effectivity as a function of the time difference between the occurrences of a pre- and a postsynaptic spike where the presynaptic spike arrives at the synapse on the axon of the presynaptic neuron as usual and the spike of the postsynaptic neuron is backpropagated through the dendritic tree to the synapse (compare F in Fig. 3, corresponding to similar data from Froemke and Dan 2002). Interpreting this finding as evidence for a local mechanism at each individual synapse implies that the two times in question mark the passing-by of the backpropagated postsynaptic and the presynaptic spike at the synapse. The figure of Bi and Poo shows that roughly coincident firing of pre- and postsynaptic spikes will generally tend to increase synaptic strength, but in more detail, the presynaptic spike should occur a few msec before the postsynaptic one at the synapse and the reverse temporal relation occurring repeatedly will lead to a decrease of synaptic strength (Carporale and Dan 2008). Backpropagation of spikes through the dendritic tree and STDP mechanisms of this type are today believed to be ubiquitous not only in the hippocampus, but throughout the cortex (Kampa and Stuart 2006; Spruston 2008, 2009).

The finding of Bi and Poo is clearly consistent with the use of STDP for hetero-association, because the incoming (cortico-cortical) presynaptic spike pattern will help to activate the postsynaptic local population subsequently. This has also been shown in simulation studies of STDP (Gerstner et al. 1993; Levy et al. 2001; Izhikevich 2006; Morrison et al. 2007). Its use for auto-association, however, seems problematic, because auto-associative connections are between neurons of the same population which are presumably firing (roughly) in coincidence, and the axonal delays leading from the presynaptic cell body to the synapse are probably longer than the dendritic delays of the postsynaptic spike back to the synapse. Therefore STDP was generally believed to strongly depress synaptic strengths of auto-associative connections which has also been shown in several computer simulations (Knoblauch and Sommer 2003, 2004; Kozloski and Cecchi 2008, 2010; Lubenov and Siapas 2008; Clopath et al. 2010; Hauser et al. 2009; Hauser et al. 2010).

Recently, we have repeated these experiments for various STDP models with biologically realistic parameter ranges (Knoblauch et al. 2012; Hauser 2012). We found that a reasonable jitter in the occurrence times of the spikes in the active population can indeed lead to enhancement of the auto-associative connections even if the axonal delay to the synapse is a bit longer than the dendritic delay. This is shown in Figs. 2 and 3.

Fig. 2
figure 2

Network simulation showing that STDP (spike-timing dependent plasticity) can couple neurons that fire in synchrony. a Summed spike activity of a simulated neuron population as a function of time (10 ms sliding time window). Initially (t \(\,<\,\)3 s), synchronization is weak. However, synaptic modification by STDP (switched on at t = 3 s) increases synaptic strength for connections with small axonal transmission delays (panel c) leading to strong synchronized oscillations of spike activity. Correspondingly, the distribution of time lags between pre- and postsynaptic spikes gets increasingly peaked (b). Note that the effect of STDP increases strongly with the presence of synchronization (e.g., see c: there is only little change in the strength distribution between t = 0 and t = 10, but a strong change between t = 10 and t = 20, corresponding to the strong increase of synchronization around t = 13). Still, the “decoupling effect”, for example, as described by Lubenov and Siapas (2008) for very precise (stimulation-induced) synchronization, occurs only for synapses with large delays, whereas synapses with small delays (e.g., d \(<\) 4 ms) continue to increase their strengths. Figure modified from Knoblauch et al. (2012) (color figure online)

Full size image
Fig. 3
figure 3

STDP can either couple or decouple synchronized neurons depending on the distribution of spike time lags G (at the synaptic site) and the shape of the STDP curve F (from Froemke and Dan 2002). In general, the resulting expected weight change (per spike) is the integral of the product of F and G. a Precise synchronization (time window T = 1 ms) results in depression of synaptic strength even for small effective transmission delays (d = 1 ms) because the lag distribution G has overlap only with the LTD (long term depression) branch of the STDP curve F.b More realistic coarse synchronization (T = 10 ms) results in potentiation of synaptic strength because G overlaps also with the LTP (long term potentiation) branch of F (that dominates over LTD). c Contour plot of expected weight change (i.e., the integral of FG) as a function of T and d showing a large parameter region where coarsely synchronized neurons get coupled by LTP. d Average weight change as function of average spike rate for uncorrelated firing (“rate coding”, Black curve), precise synchronization (T = 1  ms, blue), and coarse synchronization (T = 10 ms, green) assuming d = 1 ms. Note that coarse synchrony is most effective in coupling neurons even at low firing rates. Effective delay d = d0+dax-dbap includes time shift of the STDP curve (d0), axonal delay (dax), and dendritic delay of the backpropagating spike (dbap). Figure modified from Knoblauch et al. (2012) (color figure online)

Full size image

Basically, our simulations and analyses reveal that STDP strengthens the synaptic connections between synchronized neurons if the jitter of spike times (parameter T in Fig. 3) is larger than the effective (axonal minus dendritic) transmission delays (parameter d in Fig. 3). This effect can easily be understood by inspecting shapes of experimental STDP curves, for example, as measured by Bi and Poo. There, the discontinuity near zero time lag is typically such that the synaptic potentiation for small positive time lags is much larger than the synaptic depression for small negative time lags (see STDP function F around \(\Delta \) t=0 in Fig. 3). Thus, averaging over a mixture of potentiation and depression events as expected for synchronization with realistic spike jitter [e.g., 10msec range, for more discussion see for example Wennekers and Palm (1999)] will lead to a net potentiation of synaptic strength even for significant axonal delays and low average firing rates. We have extended this reasoning also to more realistic models of STDP, for example based on spike triplets instead of spike pairs (Pfister and Gerstner 2006; Clopath et al. 2010). Our results reconcile STDP with the idea that spike synchronization has a constructive rather than destructive role in the formation of auto-associative memories and cell assemblies.

6 Conclusion

The 1978-paper by Braitenberg has provided strong arguments for a theory of cell assemblies in the spirit of Donald Hebb by showing how well the particular structure of the cerebral cortex fits the requirements of such a theory. This has been substantiated by the neuroanatomical research over the last 30 years. As discussed in Sect. 2, a lot of quantitative data from neuroanatomy have accumulated which—together with data from physiology (for review see Huyck and Passmore 2013)—make it possible to deal with computational and dynamical aspects of cell assemblies in an associative network of cortical areas. This theoretical scheme also fits well with most of the functional “cortex theories” that have been developed from Donald Hebb’s early book (Hebb 1949) up to the present day. Such a theory of the cortex as a network of local networks generated by a combination of genetic wiring principles and individual associative learning has been sketched here (and earlier for example by Markert et al. 2007) exhibiting the enormous computational capabilities of the cortex for quite general cognitive demands.