Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Overview

Mental function is usually divided into three parts: perception, cognition, and action – the so-called sense-think-act cycle. Though this view is no longer held dogmatically, it is useful as a structuring framework for discussing mental processes. Several decades of theory and experiment have elucidated an intricate, multiconnected functional architecture for the brain [1, 2] – a simplified version of which is shown in Fig. 35.1. While all regions and functions shown – and many not shown – are important, this figure provides a summary of the main brain regions involved in perception, cognition, and action. The highlighted blocks in Fig. 35.1 are discussed in this chapter, which focuses mainly on the higher level mechanisms for the control of behavior.

Fig. 35.1
figure 1figure 1

A general schematic of primary signal flow in the nervous system. Many modulatory regions and connections, as well as several known connections, are not shown. The shaded areas indicate the components covered in this chapter

Full size image

The control of action (or behavior) is, in a real sense, the primary function of the nervous system. While such actions may be voluntary or involuntary, most of the interest in modeling has understandably focused on voluntary action. This chapter will follow this precedent.

It is conventional to divide the neural substrates of behavior into higher and lower levels. The latter involves the musculoskeletal apparatus of action (muscles, joints, etc.) and the neural networks of the spinal cord and brainstem. These systems are seen as representing the actuation component of the action system, which is controlled by the higher level system comprising cortical and subcortical structures. This division between a controller (the brain) and the plant (the body and spinal networks), which parallels the models used in robotics, has been criticized as arbitrary and unhelpful [3, 4], and there has recently been a shift of interest toward more embodied views of cognition [5, 6]. However, the conventional division is useful for organizing material covered in this chapter, which focuses primarily on the higher level systems, i. e., those above the spinal cord and the brainstem.

The higher level system can be divided further into a cognitive control component involving action selection, configuration of complex actions, and the learning of appropriate behaviors through experience, and a motor control component that generates the control signals for the lower level system to execute the selected action. The latter is usually identified with the motor cortex (GlossaryTerm

M1

), premotor cortex (GlossaryTerm

PMC

), and the supplementary motor area (GlossaryTerm

SMA

), while the former is seen as involving the prefrontal cortex (GlossaryTerm

PFC

), basal ganglia (GlossaryTerm

BG

), the anterior cingulate cortex (GlossaryTerm

ACC

) and other cortical and subcortical regions [7]. With regard to the generation of actions per se, an influential viewpoint for the higher level system is summarized by Doya [8]. It proposes that higher level control of action has three major loci: the cortex, the cerebellum, and the GlossaryTerm

BG

. Of these, the cortex – primarily the GlossaryTerm

M1

 – provides a self-organized repertoire of possible actions that, when triggered, generate movement by activating muscles via spinal networks, the cerebellum implements fine motor control configured through error-based supervised learning [9], and the GlossaryTerm

BG

provide the mechanisms for selecting among actions and learning appropriate ones through reinforcement learning [10, 11, 12, 13]. The motor cortex and cerebellum can be seen primarily as motor control (though see [14]), whereas the GlossaryTerm

BG

falls into the domain of cognitive control and working memory (GlossaryTerm

WM

). The GlossaryTerm

PFC

is usually regarded as the locus for higher order choice representations, plans, goals, etc. [15, 16, 17, 18], while the GlossaryTerm

ACC

is thought to be involved in conflict monitoring [19, 20, 21].

2 Motor Control

Given its experimental accessibility and direct relevance to robotics, motor control has been a primary area of interest for computational modeling [22, 23, 24]. Mathematical, albeit non-neural, theories of motor control were developed initially within the framework of dynamical systems. One of these directions led to models of action as an emergent phenomenon [25, 26, 27, 28, 29, 3, 30, 31, 32, 33] arising from interactions among preferred coordination modes [34]. This approach has continued to yield insights [29] and has been extended to multiactor situations as well [33, 35, 36, 37]. Another approach within the same framework is the equilibrium point hypothesis  [38, 39], which explains motor control through the change in the equilibrium points of the musculoskeletal system in response to neural commands. Both these dynamical approaches have paid relatively less attention to the neural basis of motor control and focused more on the phenomenology of action in its context. Nevertheless, insights from these models are fundamental to the emerging synthesis of action as an embodied cognitive function [5, 6].

A closely related investigative tradition has been developed from the early studies of gaits and other rhythmic movements in cats, fish, and other animals [40, 41, 42, 43, 44, 45], leading to computational models for central pattern generators (GlossaryTerm

CPG

s), which are neural networks that generate characteristic periodic activity patterns autonomously or in response to control signals [46]. It has been found that rhythmic movements can be explained well in terms of GlossaryTerm

CPG

s – located mainly in the spinal cord – acting upon the coordination modes inherent in the musculoskeletal system. The key insight to emerge from this work is that a wide range of useful movements can be generated by modulation of these GlossaryTerm

CPG

s by rather simple motor control signals from the brain, and feedback from sensory receptors can shape these movements further [43]. This idea was demonstrated in recent work by Ijspeert etal [47] showing how the same simple GlossaryTerm

CPG

network could produce both swimming and walking movements in a robotic salamander model using a simple scalar control signal.

While rhythmic movements are obviously important, computational models of motor control are often motivated by the desire to build humanoid or biomorphic robots, and thus need to address a broader range of actions – especially aperiodic and/or voluntary movements. Most experimental work on aperiodic movement has focused on the paradigm of manual reaching [30, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64]. However, seminal work has also been done with complex reflexes in frogs and cats [65, 66, 67, 68, 69, 70, 71, 72], isometric tasks [73, 74], ball-catching [75], drawing and writing [60, 76, 77, 78, 79, 80, 81], and postural control [71, 72, 82, 83].

A central issue in understanding motor control is the degrees of freedom problem  [84] which arises from the immense redundancy of the system – especially in the context of multijoint control. For any desired movement – such as reaching for an object – there are an infinite number of control signal combinations from the brain to the muscles that will accomplish the task (see [85] for an excellent discussion). From a control viewpoint, this has usually been seen as a problem because it precludes the clear specification of an objective function for the controller. To the extent that they consider the generation of specific control signals for each action, most computational models of motor control can be seen as direct or indirect ways to address the degrees of freedom problem.

2.1 Cortical Representation of Movement

It has been known since the seminal work by Penfield and Boldrey [86] that the stimulation of specific locations in the GlossaryTerm

M1

elicit motor responses in particular locations on the body. This has led to the notion of a motor homunculus – a map of the body on the GlossaryTerm

M1

. However, the issue of exactly what aspect of movement is encoded in response to individual neurons is far from settled. A crucial breakthrough came with the discovery of population coding by Georgopoulos etal [49]. It was found that the activity of specific neurons in the hand area of the GlossaryTerm

M1

corresponded to reaching movements in particular directions. While the tuning of individual cells was found to be rather broad (and had a sinusoidal profile), the joint activity of many such cells with different tuning directions coded the direction of movement with great precision, and could be decoded through neurally plausible estimation mechanisms. Since the initial discovery, population codes have been found in other regions of the cortex that are involved in movement [49, 53, 54, 60, 77, 78, 79, 80, 87]. Population coding is now regarded as the primary basis of directional coding in the brain, and is the basis of most brain–machine interfaces (GlossaryTerm

BMI

) and brain-controlled prosthetics [88, 89]. Neural network models for population coding have been developed by several researchers [90, 91, 92, 93], and population coding has come to be seen as a general neural representational strategy with application far beyond motor control [94]. Excellent reviews are provided in [95, 96]. Mathematical and computational models for Bayesian inference with population codes are discussed in [97, 98].

An active research issue in the cortical coding of movement is whether it occurs at the level of kinematic variables, such as direction and velocity, or in terms of kinetic variables, such as muscle forces and joint torques. From a cognitive viewpoint, a kinematic representation is obviously more useful, and population codes suggest that such representations are indeed present in the motor cortex [100, 48, 53, 54, 60, 77, 78, 79, 80, 99] and GlossaryTerm

PFC

 [101, 15]. However, movement must ultimately be constructed from the appropriate kinetic variables, i. e., by controlling the forces generated by specific muscles and the resulting joint torques. Studies have indicated that some neurons in the GlossaryTerm

M1

are indeed tuned to muscle forces and joint torques [100, 102, 103, 58, 59, 73, 99]. This apparent multiplicity of cortical representations has generated significant debate among researchers [74]. One way to resolve this issue is to consider the kinetic and kinematic representations as dual representations related through the constraints of the musculoskeletal system. However, Shah et al. [104] have used a simple computational model to show that neural populations tuned to kinetic or kinematic variables can act jointly in motor control without the need for explicit coordinate transformations.

Graziano etal [105] studied movements elicited by the sustained electrode stimulation of specific sites in the motor cortex of monkeys. They found that different sites led to specific complex, multijoint movements such as bringing the hand to the mouth or lifting the hand above the head regardless of the initial position. This raises the intriguing possibility that individual cells or groups of cells in the GlossaryTerm

M1

encode goal-directed movements that can be triggered as units. The study also indicated that this encoding is not open-loop, but can compensate – at least to some degree – for variation or extraneous perturbations. The GlossaryTerm

M1

and other related regions (e. g., the supplementary motor area and the GlossaryTerm

PMC

) appear to encode spatially organized maps of a few canonical complex movements that can be used as basis functions to construct other actions [105, 106, 107]. A neurocomputational model using self-organized feature maps has been proposed in [108] for the representation of such canonical movements.

In addition to rhythmic and reaching movements, there has also been significant work on the neural basis of sequential movements, with the finding that such neural codes for movement sequences exist in the supplementary motor area [109, 110, 111], cerebellum [112, 113], GlossaryTerm

BG

 [112], and the GlossaryTerm

PFC

 [101]. Coding for multiple goals in sequential reaching has been observed in the parietal cortex [114].

2.2 Synergy-based Representations

A rather different approach to studying the construction of movement uses the notion of motor primitives, often termed synergies [115, 116, 63]. Typically, these synergies are manifested in coordinated patterns of spatiotemporal activation over groups of muscles, implying a force field over posture space [117, 118]. Studies in frogs, cats, and humans have shown that a wide range of complex movements in an individual subject can be explained as the modulated superposition of a few synergies [115, 119, 120, 63, 65, 66, 67, 68, 69, 70, 71, 72]. Given a set of n muscles, the n-dimensional time-varying vector of activities for the muscles during an action can be written as

m q ( t ) = ∑ k = 1 N c k q g k ( t - t k q ) ,
(35.1)

where g k ( t ) is a time-varying synergy function that takes only nonnegative values, c k q is the gain of the kth synergy used for action q, and t k q is the temporal offset with which the kth synergy is triggered for action q [69]. The key point is that a broad range of actions can be constructed by choosing different gains and offsets over the same set of synergies, which represent a set of hard-coded basis functions for the construction of movements [120, 121]. Even more interestingly, it appears that the synergies found empirically across different subjects of the same species are rather consistent [67, 72], possibly reflecting the inherent constraints of musculoskeletal anatomy. Various neural loci have been suggested for synergies, including the spinal cord [107, 122, 67], the motor cortex [123], and combinations of regions [124, 85].

Though synergies are found consistently in the analysis of experimental data, their actual existence in the neural substrate remains a topic for debate [125, 126]. However, the idea of constructing complex movements from motor primitives has found ready application in robotics [127, 128, 129, 130, 131, 132], as discussed later in this chapter. A hierarchical neurocomputational model of motor synergies based on attractor networks has recently been proposed in [133, 134].

2.3 Computational Models of Motor Control

Motor control has been modeled computationally at many levels and in many ways, ranging from explicitly control-theoretic models through reinforcement-based models to models based on emergent dynamical patterns. This section provides a brief overview of these models.

As discussed above the GlossaryTerm

M1

, premotor cortex (GlossaryTerm

PMC

) and the supplementary motor area (GlossaryTerm

SMA

) are seen as providing self-organized codes for specific actions, including information on direction, velocity, force, low-level sequencing, etc., while the GlossaryTerm

PFC

provides higher level codes needed to construct more complex actions. These codes, comprising a repertoire of actions [10, 106], arise through self-organized learning of activity patterns in these cortical systems. The GlossaryTerm

BG

system is seen as the primary locus of selection among the actions in the cortical repertoire. The architecture of the system involving the cortex, GlossaryTerm

BG

, and the thalamus, and in particular the internal architecture of the GlossaryTerm

BG

 [135], makes this system ideally suited to selectively disinhibiting specific cortical regions, presumably activating codes for specific actions [10, 136, 137]. The GlossaryTerm

BG

system also provides an ideal substrate for learning appropriate actions through a dopamine-mediated reinforcement learning mechanism [138, 139, 140, 141].

Many of the influential early models of motor control were based on control-theoretic principles [142, 143, 144], using forward and inverse kinematic and dynamic models to generate control signals [145, 146, 147, 148, 149, 150, 55, 57] – see [146] for an excellent introduction. These models have led to more sophisticated ones, such as GlossaryTerm

MOSAIC

(modular selection and identification for control) [151] and GlossaryTerm

AVITEWRITE

(adaptive vector integration to endpoint handwriting) [81]. The GlossaryTerm

MOSAIC

model is a mixture of experts, consisting of many parallel modules, each comprising three subsystems. These are: A forward model relating motor commands to predicted position, a responsibility predictor that estimates the applicability of the current module, and an inverse model that learns to generate control signals for desired movements. The system generates motor commands by combining the recommendations of the inverse models of all modules weighted by their applicability. Learning in the model is based on a variant of the EM algorithm. The model in [57] is a comprehensive neural model with both cortical and spinal components, and builds upon the earlier VITE model in [55]. The GlossaryTerm

AVITEWRITE

model [81], which is a further extension of the VITE model, can generate the complex movement trajectories needed for writing by using a combination of pre-specified phenomenological motor primitives (synergies). A cerebellar model for the control of timing during reaches has been presented by Barto etal [152].

The use of neural maps in models of motor control was pioneered in [153, 154]. These models used self-organized feature maps (GlossaryTerm

SOFM

s) [155] to learn visuomotor coordination. Baraduc etal [156] presented a more detailed model that used multiple maps to first integrate posture and desired movement direction and then to transform this internal representation into a motor command. The maps in this and most subsequent models were based on earlier work by [90, 91, 92, 93]. An excellent review of this approach is given in [94]. A more recent and comprehensive example of the map-based approach is the GlossaryTerm

SURE-REACH

(sensorimotor, unsupervised, redundancy-resolving control architecture) model in [157] which focuses on exploiting the redundancy inherent in motor control [84]. Unlike many of the other models, which use neutrally implausible error-based learning, GlossaryTerm

SURE-REACH

relies only on unsupervised and reinforcement learning. Maps are also the central feature of a general cognitive architecture called GlossaryTerm

ERA

(epigenetic robotics architecture) by Morse etal [158].

Another successful approach to motor control models is based on the use of motor primitives, which are used as basis functions in the construction of diverse actions. This approach is inspired by the experimental observation of motor synergies as described above. However, most models based on primitives implement them nonneurally, as in the case of GlossaryTerm

AVITEWRITE

 [81]. The most systematic model of motor primitives has been developed by Schaal etal [129, 130, 131, 132]. In this model, motor primitives are specified using differential equations, and are combined after weighting to produce different movements. Recently, Matsubara et al. [159] have shown how the primitives in this model can be learned systematically from demonstrations. Drew et al. [123] proposed a conceptual model for the construction of locomotion using motor primitives (synergies) and identified the characteristics of such primitives experimentally. A neural model of motor primitives based on hierarchical attractor networks has been proposed recently in [133, 134, 160], while Neilson and Neilson [124, 85] have proposed a model based on coordination among adaptive neural filters.

Motor control models based on primitives can be simpler than those based on trajectory tracking because the controller typically needs to choose only the weights (and possibly delays) for the primitives rather than specifying details of the trajectory (or forces). Among other things, this promises a potential solution to the degrees of freedom problem [84] since the coordination inherent in the definition of motor primitives reduces the effective degrees of freedom in the system. Another way to address the degrees of freedom problem is to use an optimal control approach with a specific objective function. Researchers have proposed objective functions such as minimum jerk [161], minimum torque [162], minimum acceleration [163], or minimum energy [85], but an especially interesting idea is to optimize the distribution of variability across the degrees of freedom in a task-dependent way [144, 164, 165, 166, 167]. From this perspective, motor control trades off variability in task-irrelevant dimensions for greater accuracy in task-relevant ones. Thus, rather than specifying a trajectory, the controller focuses only on correcting consequential errors. This also explains the experimental observation that motor tasks achieve their goals with remarkable accuracy while using highly variable trajectories to achieve the same goal. Trainin et al. [168] have shown that the optimal control principle can be used to explain the observed neural coding of movements in the cortex. Biess et al. [169] have proposed a detailed computational model for controlling an arm in three-dimensional space by separating the spatial and temporal components of control. This model is based on optimizing energy usage and jerk [161], but is not implemented at the neural level.

An alternative to these prescriptive and constructivist approaches to motor control is provided by models based on dynamical systems [25, 26, 27, 29, 3, 31, 32, 33]. The most important way in which these models diverge from the others is in their use of emergence as the central organizational principle of control. In this formulation, control programs, structures, primitives, etc., are not preconfigured in the brain–body system, but emerge under the influence of task and environmental constraints on the affordances of the system [33]. Thus, the dynamical systems view of motor control is fundamentally ecological [170], and like most ecological models, is specified in terms of low-dimensional state dynamics rather than high-dimensional neural processes. Interestingly, a correspondence can be made between the dynamical and optimal control models through the so-called uncontrolled manifold concept [171, 31, 33, 39]. In both models, the dimensions to be controlled and those that are left uncontrolled are decided by external constraints rather than internal prescription, as in classical models.

3 Cognitive Control and Working Memory

A lot of behavior – even in primates – is automatic, or almost so. This corresponds to actions (or internal behaviors) so thoroughly embedded in the sensorimotor substrate that they emerge effortlessly from it. In contrast, some tasks require significant cognitive effort for one or more reason, including:

  1. 1.

    An automatic behavior must be suppressed to allow the correct response to emerge, e. g., in the Stroop task [172].

  2. 2.

    Conflicts between incoming information and/or recalled behaviors must be resolved [19, 20].

  3. 3.

    More contextual information – e. g., social context – must be taken into account before acting.

  4. 4.

    Intermediate pieces of information need to be stored and recalled during the performance of the task, e. g., in sequential problem solving.

  5. 5.

    The timing of subtasks within the overall task is complex, e. g., in delayed-response tasks or other sequential tasks [173].

Roughly speaking, the first three fall under the heading of cognitive control, and the latter two of working memory. However, because of the functions are intimately linked, the terms are often subsumed into each other.

3.1 Action Selection and Reinforcement Learning

Action selection is arguably the central component of the cognitive control process. As the name implies, it involves selectively triggering an action from a repertoire of available ones. While action selection is a complex process involving many brain regions, a consensus has emerged that the GlossaryTerm

BG

system plays a central role in its mechanism [10, 12, 14]. The architecture of the GlossaryTerm

BG

system and the organization of its projections to and from the cortex [135, 174, 175] make it ideally suited to function as a state-dependent gating system for specific functional networks in the cortex. As shown in Fig. 35.2, the hypothesis is that the striatal layer of the GlossaryTerm

BG

system, receiving input from the cortex, acts as a pattern recognizer for the current cognitive state. Its activity inhibits specific parts of the globus pallidus (GlossaryTerm

GPi

), leading to disinhibition of specific neural assemblies in the cortex – presumably allowing the behavior/action encoded by those assemblies to proceed [10]. The associations between cortical activity patterns and behaviors are key to the functioning of the GlossaryTerm

BG

as an action selection system, and the configuration and modulation of these associations are thought to lie at the core of cognitive control. The neurotransmitter dopamine (GlossaryTerm

DA

) plays a key role here by serving as a reward signal [138, 139, 140] and modulating reinforcement learning [176, 177] in both the GlossaryTerm

BG

and the cortex [141, 178, 179, 180].

3.2 Working Memory

All nontrivial behaviors require task-specific information, including relevant domain knowledge and the relative timing of subtasks. These are usually grouped under the function of GlossaryTerm

WM

. An influential model of GlossaryTerm

WM

in [181] identifies three components in GlossaryTerm

WM

: (1) a central executive, responsible for attention, decision making, and timing; (2) a phonological loop, responsible for processing incoming auditory information, maintaining it in short-term memory, and rehearsing utterances; and (3) a visuospatial sketchpad, responsible for processing and remembering visual information, keeping track of what and where information, etc. An episodic buffer to manage relationships between the other three components is sometimes included [182]. Though already rather abstract, this model needs even more generalized interpretation in the context of many cognitive tasks that do not directly involve visual or auditory data. Working memory function is most closely identified with the GlossaryTerm

PFC

 [183, 184, 185].

Almost all studies of GlossaryTerm

WM

consider only short-term memory, typically on the scale of a few seconds [186]. Indeed, one of the most significant – though lately controversial – results in GlossaryTerm

WM

research is the finding that only a small number of items can be kept in mind at any one time [187, 188]. However, most cognitive tasks require context-dependent repertoires of knowledge and behaviors to be enabled collectively over longer periods. For example, a player must continually think of chess moves and strategies over the course of a match lasting several hours. The configuration of context-dependent repertoires for extended periods has been termed long-term working memory [189].

3.3 Computational Models of Cognitive Control and Working Memory

Several computational models have been proposed for cognitive control, and most of them share common features. The issues addressed by the models include action selection, reinforcement learning of appropriate actions, decision making in choice tasks, task sequencing and timing, persistence and capacity in GlossaryTerm

WM

, task switching, sequence learning, and the configuration of context-appropriate workspaces. Most of the models discussed below are neural with a range of biological plausibility. A few important nonneural models are also mentioned.

Fig. 35.2
figure 2figure 2

The action selection and reinforcement learning substrate in the BG. Wide filled arrows indicate excitatory projections while wide unfilled arrows represent inhibitory projections. Linear arrows indicate generic excitatory and inhibitory connectivity between regions. The inverted D-shaped contacts indicate modulatory dopamine connections that are crucial to reinforcement learning. Abbreviations: SMA=supplementary motor area; SNc=substantia nigra pars compacta; VTA=ventral tegmental area; OFC=orbitofrontal cortex; GPe=globus pallidus (external nuclei); GPi=globus pallidus (internal nuclei); STN=subthalamic nucleus; D 1 = excitatory dopamine receptors; D 2 = inhibitory dopamine receptors. The primary neurons of GPi are inhibitory and active by default, thus keeping all motor plans in the motor and premotor cortices in check. The neurons of the striatum are also inhibitory but usually in an inactive down state (after [190]). Particular subgroups of striatal neurons are activated by specific patterns of cortical activity (after [136]), leading first to disinhibition of specific actions via the direct input from striatum to GPi, and then by re-inhibition via the input through STN. Thus the system gates the triggering of actions appropriate to current cognitive contexts in the cortex. The dopamine input from SNc projects a reward signal based on limbic system state, allowing desirable context-action pairs to be reinforced (after [191, 192]) – though other hypotheses also exist (after [14]). The dopamine input to PFC from the VTA also signals reward and other task-related contingencies

Full size image

A comprehensive model using spiking neurons and incorporating many biological features of the GlossaryTerm

BG

system has been presented in [13, 193]. This model focuses only on the GlossaryTerm

BG

and explicitly on the dynamics of dopamine modulation. A more abstract but broader model of cognitive control is the agents of the mind model in [14], which incorporates the cerebellum as well as the GlossaryTerm

BG

. In this model, the GlossaryTerm

BG

provide the action selection function while the cerebellum acts to refine and amplify the choices. A series of interrelated models have been developed by O’Reilly, Frank etal [17, 179, 194, 195, 196, 197, 198, 199]. All these models use the adaptive gating function of the GlossaryTerm

BG

in combination with the GlossaryTerm

WM

function of the prefrontal cortex to explain how executive function can arise without explicit top-down control – the so-called homunculus [196, 197]. A comprehensive review of these and other models of cognitive control is given in [200]. Models of goal-directed action mediated by the GlossaryTerm

PFC

have also been presented in [201] and [202]. Reynolds and O’Reilly etal [203] have proposed a model for configuring hierarchically organized representations in the GlossaryTerm

PFC

via reinforcement learning. Computational models of cognitive control and working have also been used to explain mental pathologies such as schizophrenia [204].

An important aspect of cognitive control is switching between tasks at various time-scales [205, 206]. Imamizu etal [207] compared two computational models of task switching – a mixture-of-experts (GlossaryTerm

MoE

) model and GlossaryTerm

MOSAIC

 – using brain imaging. They concluded that task switching in the GlossaryTerm

PFC

was more consistent with the GlossaryTerm

MoE

model and that in the parietal cortex and cerebellum with the GlossaryTerm

MOSAIC

model.

An influential abstract model of cognitive control is the interactive activation model in [208, 209]. In this model, learned behavioral schemata contend for activation based on task context and cognitive state. While this model captures many phenomenological aspects of behavior, it is not explicitly neural. Botvinick and Plaut [173] present an alternative neural model that relies on distributed neural representations and the dynamics of recurrent neural networks rather than explicit schemata and contention. Dayan etal [210, 211] have proposed a neural model for implementing complex rule-based decision making where decisions are based on sequentially unfolding contexts. A partially neural model of behavior based on the CLARION cognitive model has been developed in [212].

Recently, Grossberg and Pearson [213] have presented a comprehensive model of GlossaryTerm

WM

called LIST PARSE. In this model, the term working memory is applied narrowly to the storage of temporally ordered items, i. e., lists, rather than more broadly to all short-term memory. Experimentally observed effects such as recency (better recall of late items in the list) and primacy (better recall of early items in the list) are explained by this model, which uses the concept of competitive queuing for sequences. This is based on the observation [101, 214] that multiple elements of a behavioral sequence are represented in the GlossaryTerm

PFC

as simultaneously active codes with activation levels representing the temporal order. Unlike the GlossaryTerm

WM

models discussed in the previous paragraph, the GlossaryTerm

WM

in LIST PARSE is embedded within a full cognitive control model with action selection, trajectory generation, etc. Many other neural models for chains of actions have also been proposed [214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224].

Higher level cognitive control is characterized by the need to fuse information from multiple sensory modalities and memory to make complex decisions. This has led to the idea of a cognitive workspace. In the global workspace theory (GlossaryTerm

GWT

) developed in [225], information from various sensory, episodic, semantic, and motivational sources comes together in a global workspace that forms brief, task-specific integrated representations that are broadcast to all subsystems for use in GlossaryTerm

WM

. This model has been implemented computationally in the intelligent distribution agent (GlossaryTerm

IDA

) model by Franklin etal [226, 227]. A neurally implemented workspace model has been developed by Dehaene etal [172, 228, 229] to explain human subjects’ performance on effortful cognitive tasks (i. e., tasks that require suppression of automatic responses), and the basis of consciousness. The construction of cognitive workspaces is closely related to the idea of long-term working memory [189]. Unlike short-term working memory, there are few computational models for long-term working memory. Neural models seldom cover long periods, and implicitly assume that a chaining process through recurrent networks (e. g., [173]) can maintain internal attention. Iyer etal [230, 231] have proposed an explicitly neurodynamical model of this function, where a stable but modulatable pattern of activity called a graded attractor is used to selectively bias parts of the cortex in the context-dependent fashion. An earlier model was proposed in [232] to serve a similar function in the hippocampal system.

Another class of models focuses primarily on single decisions within a task, and assume an underlying stochastic process [186, 233, 234, 235]. Typically, these models address two-choice short-term decisions made over a second or two [186]. The decision process begins with a starting point and accumulates information over time resulting in a diffusive (random walk) process. When the diffusion reaches one of two boundaries on either side of the starting point, the corresponding decision is made. This elegant approach can model such concrete issues as decision accuracy, decision time, and the distribution of decisions without any reference to the underlying neural mechanisms, which is both its chief strength and its primary weakness. Several connectionist models have also been developed based on paradigms similar to the diffusion approach [236, 237, 238]. The neural basis of such models has been discussed in detail in [239]. A population-coding neural model that makes Bayesian decisions based on cumulative evidence has been described by Beck etal [98].

Reinforcement learning [176] is widely used in many engineering applications, but several models go beyond purely computational use and include details of the underlying brain regions and neurophysiology [141, 240]. Excellent reviews of such models are provided in [241, 242, 243]. Recently, models have also been proposed to show how dopamine-mediated learning could work with spiking neurons [244] and population codes [245].

Computational models that focus on working memory per se (i. e., not on the entire problem of cognitive control) have mainly considered how the requirement of selective temporal persistence can be met by biologically plausible neural networks [246, 247]. Since working memories must bridge over temporal durations (e. g., in remembering a cue over a delay period), there must be some neural mechanism to allow activity patterns to persist selectively in time. A natural candidate for this is attractor dynamics in recurrent neural networks [248, 249], where the recurrence allows some activity patterns to be stabilized by reverberation [250]. The neurophysiological basis of such persistent activity has been studied in [251]. A central feature in many models of GlossaryTerm

WM

is the role of dopamine in the GlossaryTerm

PFC

 [252, 253, 254]. In particular, it is believed that dopamine sharpens the response of GlossaryTerm

PFC

neurons involved in GlossaryTerm

WM

 [255] and allows for reliable storage of timing information in the presence of distractors [246]. The model in [246, 252] includes several biophysical details such as the effect of dopamine on different ion channels and its differential modulation of various receptors. More abstract neural models for GlossaryTerm

WM

have been proposed in [256] and [257].

A especially interesting type of attractor network uses the so-called bump attractors – spatially localized patterns of activity stabilized by local network connectivity and global competition [258]. Such a network has been used in a biologically plausible model of GlossaryTerm

WM

in the GlossaryTerm

PFC

in [259], which demonstrates that the memory is robust against distracting stimuli. A similar conclusion is drawn in [180] based on another bump attractor model of working memory. It shows that dopamine in the GlossaryTerm

PFC

can provide robustness against distractors, but robustness against internal noise is achieved only when dopamine in the GlossaryTerm

BG

locks the state of the striatum. Recently, Mongillo etal [260] have proposed the novel hypothesis that the persistence of neural activity in GlossaryTerm

WM

may be due to calcium-mediated facilitation rather than reverberation through recurrent connectivity.

4 Conclusion

This chapter has attempted to provide an overview of neurocomputational models for cognitive control, GlossaryTerm

WM

, and motor control. Given the vast body of both experimental and computational research in these areas, the review is necessarily incomplete, though every attempt has been made to highlights the major issues, and to provide the reader with a rich array of references covering the breadth of each area.

The models described in this chapter relate to several other mental functions including sensorimotor integration, memory, semantic cognition, etc., as well as to areas of engineering such as robotics and agent systems. However, these links are largely excluded from the chapter – in part for brevity, but mainly because most of them are covered elsewhere in this Handbook.