A Cortico-Basal Ganglia Model to Understand the Neural Dynamics of Targeted Reaching in Normal and Parkinson’s Conditions

Abstract

We present a cortico-basal ganglia model to study the neural mechanisms behind reaching movements in normal and in Parkinson’s disease conditions. The model consists of the following components: a two-joint arm model (AM), a layer of motor neurons in the spinal cord (MN), the proprioceptive cortex (PC), the motor cortex (MC), the prefrontal cortex (PFC), and the basal ganglia (BG). The model thus has an outer sensory-motor cortical loop and an inner cortico-basal ganglia loop to drive learning of reaching behavior. Sensory and motor maps are formed by the PC and MC which represent the space of arm configurations. The BG sends control signals to the MC following a stochastic gradient ascent policy applied to the value function defined over the arm configuration space. The trainable connections from PFC to MC can directly activate the motor cortex, thereby producing rapid movement avoiding the slow search conducted by the BG. The model captures the two main stages of motor learning, i.e., slow movements dominated by the BG during early stages and cortically driven fast movements with smoother trajectories at later stages. The model explains PD performance in stationary and pursuit reaching tasks. The model also shows that PD symptoms like tremor and rigidity could be attributed to synchronized oscillations in STN–GPe. The model is in line with closed-loop control and with neural representations for all the nuclei which explains Parkinsonian reaching. By virtue of its ability to capture the role of cortico-basal ganglia systems in controlling a wide range of features of reaching, the proposed model can potentially serve as a benchmark to test various motor pathologies of the BG.

10.1 Introduction

Reaching movements are for movement science, what the simple pendulum is for classical mechanics. Reaching movements reveal a lot about how the brain plans and executes movement kinematics and dynamics, in normal and pathological conditions. Early experiments by Fitts and Morrasso on reaching movements showed that the hand velocity profile has a bell-shaped distribution providing a glimpse into the planning of motor trajectories (Fitts, 1954; Morasso, 1981). However, it was observed that such planning required adaptive feedback mechanisms which could relay the current state of the motor effector and the learning framework for optimal control (Todorov, 2004). As a result, internal models were introduced which minimize the error between the target and current arm position by including the factor of variability which accounts for the noise in movement (Shadmehr & Krakauer, 2008). According to the optimal feedback control framework, the current state plays a crucial role in determining the future state and eventually the trajectory and thus probed investigators to modeling reaching with Baye’s approach as it allows integration of previous knowledge with current sensory information (Schaal & Schweighofer, 2005). Kording and Wolpert showed that the experimental results of visually guided reaching task in the presence of noisy feedback and explained using the Bayesian approach how subjects represented both the statistics of the sensorimotor task and the uncertainty in the task (Körding & Wolpert, 2004). A neural correlate to Bayesian processing by neurons was suggested by Knill and Pouget using a computational model by introducing Poisson noise in the neural activity (Knill & Pouget, 2004).

Though the above computational models provided insights into movement planning and execution, they do not specify the corresponding neural correlates. During the same period, experimental groups were studying the roles of various cortical and subcortical areas in motor learning and execution (Doya, 1999). Particularly, the basal ganglia (BG) are involved in the learning of new actions and sequences from cortical projections which are modulated by the midbrain dopaminergic system (Hikosaka, Nakamura, Sakai, & Nakahara, 2002). Using a computational model, Nakahara, Doya, & Hikosaka, (2002) showed that parallel learning occurs in BG-cortical systems through the visual and the motor loop. The final output is selected by the presupplementary area which acts as a coordinator for optimal acquisition and execution of well-learned sequences (Hikosaka et al., 2002; Nakahara, Doya, & Hikosaka, 2001). Chen and colleagues developed a model of closed-loop control of hand movement, in which a sensory module receives input from visual/proprioceptive areas and a Motor Module drives a mechanical two-link arm (Chen & Reggia, 1996). By randomly activating various points in motor cortex, the arm is driven to various points in the workspace; feedback from the hand is used to train the proprioceptive cortex, the motor cortex, and the motor neurons of the spinal cord module. The model is trained by unsupervised learning and was unable to describe goal-oriented reaching which requires either supervised or reinforcement learning. Izawa, Kondo, and Ito (2004) modeled a two-link arm model with detailed arm kinematics and included learning using reinforcement learning (Izawa et al., 2004).

We present a model of reaching that describes the contributions of basal ganglia (BG) and the sensory-motor cortical pathway to reaching. The model particularly highlights the role of BG in motor learning. In the model, the BG system discovers the desired motor cortical output by processing the reaching error which, we propose, is coded by nigrostriatal dopamine signals. This desired output is used by the motor cortex for training. Thus in the model, the BG leads the motor cortex in learning. The relative contributions of the cortical areas and BG evolve with learning, with the contribution of BG dwindling with learning. The model explains reaching movements in normal and Parkinsonian conditions and explores the causes of the distinct paths of evolution of PD symptoms into tremor-dominant and rigidity-dominant. The present model is a detailed network version of a simple lumped model of reaching that we proposed earlier (Magdoom et al., 2011).

10.2 Methods

The cortico-basal ganglia model consists of two major components: the outer loop which is the sensory-motor cortical loop and an inner loop which is the cortico-basal ganglia loop (Fig. 10.1). These loops are an integral part for the execution of controlled movements.

Fig. 10.1

Cortico-basal ganglia model used for simulating the reaching movements. The architecture is designed to have two loops, a sensory-motor ‘outer’ loop (shown by solid black arrows) and the cortico-basal ganglia ‘inner’ loop (shown by dashed black arrows). The basal ganglia is shown to have projections from midbrain dopaminergic (DA) neurons. The motor cortex receives projections from higher frontal areas which in the model is the prefrontal cortex. The (m × n) shows the size of the neuronal sheet used for each area in the model (in the basal ganglia all the nuclei are 15 × 15 in size)

10.2.1 Arm Model

A simple two-joint kinematic model of an arm is used in the model. Each joint is controlled by an agonist (Ag) and an antagonist (An) muscle pair innervated by a pair of motor neurons; the muscles in turn control the position of the arm in the 2D space. The input to the arm is a four-dimensional vector ϕ ^MN(t) which represents the muscle innervations for the agonist–antagonist pair for both the joints. The activation is then transformed to obtain the joint angles (θ ^JA _S/E (t)) for shoulder and the elbow joint using Eqs. (10.1) and (10.2), respectively.

$$ \theta_{S}^{\text{JA}} \left( t \right) = \left( {\phi_{\text{Ag}}^{\text{MN}} \left( t \right) - \phi_{\text{An}}^{\text{MN}} \left( t \right)} \right)\frac{\pi }{2} + \frac{\pi }{2} $$

(10.1)

$$ \theta_{E}^{\text{JA}} \left( t \right) = \left( {\phi_{\text{Ag}}^{\text{MN}} \left( t \right) - \phi_{\text{An}}^{\text{MN}} \left( t \right)} \right)\frac{\pi }{2} + \frac{\pi }{2} $$

(10.2)

The arm covers a given set of targets in the workspace, restricted by the range of movements of the joints. The joint angle measures are subsequently used to determine the lengths (μ ^E and μ ^S) of each muscle [Eqs. (10.3), (10.4), (10.5), and (10.6)].

$$ \mu_{\text{Ag}}^{S} \left( t \right) = \sqrt {a_{S}^{2} + b_{S}^{2} + 2a_{S} b_{S} \,\cos \left( {\theta_{S}^{\text{JA}} } \right)} $$

(10.3)

$$ \mu_{\text{An}}^{S} \left( t \right) = \sqrt {a_{S}^{2} + b_{S}^{2} - 2a_{S} b_{S} \,\cos \left( {\theta_{S}^{\text{JA}} } \right)} $$

(10.4)

$$ \mu_{\text{Ag}}^{E} \left( t \right) = \sqrt {a_{E}^{2} + b_{E}^{2} + 2a_{E} b_{E} \,\cos \left( {\theta_{E}^{\text{JA}} } \right)} $$

(10.5)

$$ \mu_{\text{An}}^{E} \left( t \right) = \sqrt {a_{E}^{2} + b_{E}^{2} - 2a_{E} b_{E} \,\cos \left( {\theta_{E}^{\text{JA}} } \right)} $$

(10.6)

These muscle lengths form the four-dimensional vector (M _L  = [μ ^S_Ag μ ^S_An μ ^E_Ag μ ^E_An ]) which is used to develop a sensory (proprioceptive) map of the arm. Furthermore, the end effector position (X ^arm = [x ^arm₁ x ^arm₂ ]) is also estimated in Eqs. (10.7) and (10.8).

$$ x_{1}^{\text{arm}} = \left( {l_{S} - a_{S} } \right)\,\cos \left( {\theta_{S}^{\text{JA}} } \right) + l_{E} \,\cos \left( {\theta_{S}^{\text{JA}} + \theta_{E}^{\text{JA}} } \right) $$

(10.7)

$$ x_{2}^{\text{arm}} = \left( {l_{S} - a_{S} } \right)\,\sin \left( {\theta_{S}^{\text{JA}} } \right) + l_{E} \,\sin \left( {\theta_{S}^{\text{JA}} + \theta_{E}^{\text{JA}} } \right) $$

(10.8)

10.2.2 The Sensory-Motor Cortical Loop

• Sensory and Motor Maps

The sensory-motor cortical loop comprises of the arm, the proprioceptive cortex (PC), i.e., the proprioceptive area of the primary somatosensory cortex, the motor cortex, and the spinal motor neurons. PC is modeled as a self-organizing map (SOM) of size N _PC × N _PC (Kohonen, 1990). In order to develop a sensory map of the arm which we will from now on refer to as the proprioceptive map/cortex (PC), the muscle length vector (M _L (t)) received from the arm is used as feature vector to train the PC. The activation of a single node i in the PC is given by Eq. (10.9).

$$ P_{i} \left( t \right) = \exp \left( {\frac{{ - \left\| {M_{L} \left( t \right) - \left. {W_{{{\text{PC}},i}} } \right\|} \right.^{2} }}{{\sigma_{\text{PC}}^{2} }}} \right) $$

(10.9)

where W _PC,i is the weight connection between the muscle length vector of the arm and the ith node of the PC, and σ _PC is the width of the Gaussian response.

The motor cortex (MC) is modeled as a combination of a continuous attractor neural network (CANN) (Trappenberg, 2003) and a SOM of size N _MC × N _MC. This represents two distinct characteristics of cortical areas which are known to have low-dimensional representation of the input space and dynamics based on the connectivity in these areas. The CANN architecture is characterized by short-range excitation and long-range inhibition. Its weight kernel (W ^C_MC ) is parameterized by the strength of the excitatory connections (A ^C_lat ), the radius of the excitatory connections (σ ^C_lat ), and the global inhibition constant (K ^C). A dynamic model like the CANN is used to model MC, instead of a static model like SOM, so as to be able to dynamically integrate the afferent inputs coming from the PC, BG, and the prefrontal cortex (PFC). The PC activity is used as the input to generate the low-dimensional feature maps at the level of the MC. The MC uses this sensory map information to develop a motor map of the arm. This is done by giving the output of PC, a matrix of size N _PC × N _PC, converted into a vector of size N ²_PC  × 1, as input vector to the SOM part of MC. Training of this SOM is performed by the standard SOM algorithm (Kohonen, 1990). Output of the PC (G _PC), in addition to two other inputs, is presented as input to the CANN part of the MC (I _MC). The network is fully connected from the arm to the PC; similarly, every PC neuron projects to every neuron in MC. The activation of a node i in the SOM part of the MC is given by Eq. (10.10).

$$ G_{{{\text{PC}},i}} \left( t \right) = \exp \left( {\frac{{ - \left\| {P\left( t \right) - W_{{{\text{MC}},i}} } \right\|^{2} }}{{\sigma_{\text{MC}}^{2} }}} \right) $$

(10.10)

Here, W _MC,i is the weight connection between the PC and the ith node of the SOM part of MC and σ _MC is the width of the Gaussian response. The MC activation via the attractor dynamics is driven by the PC, the BG, and the PFC. Therefore, the total input coming into the MC is $I_{\text{MC}} \left( t \right) = A_{\text{PC}} G_{\text{PC}} \left( t \right) + A_{\text{BG}} G_{\text{BG}} \left( t \right) + A_{\text{PFC}} G_{\text{PFC}} \left( t \right)$, where A _PC, A _BG, and A _PFC are the respective gains of the PC, BG, and PFC networks. With these inputs, the activation dynamics of the MC is given in (10.11).

$$ \tau_{\text{MC}} \frac{{{\text{d}}g_{\text{MC}} }}{{{\text{d}}t}} = - g_{\text{MC}} + W_{\text{MC}}^{C} \otimes G + I_{\text{MC}} $$

(10.11)

where g _MC is the internal state of the MC neurons, W ^C_MC is the weight kernel given by $W_{{{\text{MC}},i,j}}^{C} = A_{\text{lat}}^{C} \,\exp \left( {\frac{{ - \left\| {(i_{\text{MC}} - i_{h} ) + (j_{MC} - j_{h} )} \right\|^{2} }}{{2\left( {\sigma_{\text{lat}}^{C} } \right)^{2} }}} \right) - K^{C}$ which determines the local excitation/global inhibition dynamics, [i _MC, j _MC] are the locations of the nodes in MC, and [i _h, j _h ] corresponds to the central node. The output activity of the MC (G(t)) is obtained by performing a divisive normalization [Eq. (10.12)], which is done often to produce biologically realistic activity bumps (Pouget & Latham, 1999).

$$ G\left( t \right) = \frac{{g_{\text{MC}}^{2} }}{{1 + \left( {\frac{2\pi }{{N_{\text{MC}}^{2} }}} \right)b_{\text{MC}} \sum {g_{\text{MC}}^{2} } }} $$

(10.12)

Neurons of the motor cortex project to the motor neuronal layer (MN). The motor neurons—there are just four of them—in turn project one each to the four muscles of the arm as described by Eq. (10.13).

$$ \phi^{\text{MN}} \left( t \right) = A_{\text{MN}} W_{{{\text{MC}} \to {\text{MN}}}} G\left( t \right) $$

(10.13)

In order to close this loop, i.e., to train the connections between the MC and the MN (W _MC→MN) layers [Eq. (10.14)], we initially provide the input at the MN layer as the desired activation for the arm (φ ^MN _D (t)). This produces a sensory activity in the PC which in turn generates a motor activity in the MC (G(t)). The weights between the MC and the MN layers are trained in a supervised manner by comparing the network-derived MN activation φ ^MN(t) to the desired activation φ ^MN _D (t) (Eq. 10.14). This gives a loop which is consistent in mapping the external arm space to the neuronal space and vice versa.

$$ \Delta W_{{{\text{MC}} \to {\text{MN}}}} = \eta_{{{\text{MC}} \to {\text{MN}}}} \left( {\phi_{D}^{\text{MN}} \left( t \right) - \phi^{\text{MN}} \left( t \right)} \right)G\left( t \right) $$

(10.14)

10.2.3 Training the Cortical Loop

The training schema for the entire model is shown in Fig. 10.2. The steps for training the sensory-motor loop are as follows.

Fig. 10.2

Training schema in the cortico-basal ganglia model. It initially starts with (a) training the arm to PC connections, followed by (b) training the PC-to-MC connections, and finally closing the loop by (c) training the MC-to-MN weights. Then, the BG module is introduced and the PFC-to-MC connections are trained (d). In every figure, the dashed arrows indicate the connections that are being trained

• 1. Randomly generate n different muscle activations of the arm, which result in n-arm configurations. Each configuration of the arm provides a feature vector of muscle lengths, M _L .

• 2. The feature vector of muscle lengths, M _L , is presented as input to the PC layer, which is trained using the SOM algorithm.

• 3. The output state of PC layer is then presented as input to MC. Output of MC is presented as input to MN layer via a weight stage (W _MC→MN). W _MC→MN are trained by the following procedure. A random activation vector (φ ^MN) is given to the MN layer. The output of the MN layer then activates the arm and puts it in an equilibrium configuration. Starting from the muscle lengths from the arm, we track the signal flow via the PC, MC, and back to MN layers. Output of the MN layer, ideally, must be equal to the random activation vector (φ ^MN) given to the arm. The matrix of the sensory activations is then passed on to the MC layer (SOM) to evolve the motor map of the arm.

• 4. Finally, the loop is closed by training the weight connection between the MC and the MN layers by generating a desired MN activity (φ ^MN _D (t)) and approximating the network-derived MN activity φ ^MN(t) as in Eq. (10.14).

10.2.4 The Basal Ganglia

The BG module has the following components: the striatum, the Globus Pallidus internal and external segments (GPi and GPe), the subthalamic nucleus (STN), and the thalamus (Fig. 10.3a). The output of the BG modulates the MC activity to provide the appropriate control signal for the arm to reach the target. We begin with an outline of learning and operation of the BG module. In line with our earlier models of BG, in the proposed model, the BG module is trained by reinforcement learning to choose the optimal actions (Balasubramani, Chakravarthy, Ravindran, & Moustafa, 2014; Chakravarthy & Balasubramani, 2015; Chakravarthy, Joseph, & Bapi, 2010; Gupta, Balasubramani, & Chakravarthy, 2013; Magdoom et al., 2011; Muralidharan, Balasubramani, Chakravarthy, Lewis, & Moustafa, 2013). The BG, acting via the cortical loop, drives the arm so that the hand reaches the desired target (Fig. 10.3b). Prefrontal inputs which represent the target or the goal position and the current hand position information from the sensory cortex are thought to be combined in the BG to compute a value function that codes for the error between the desired and the actual hand position (Fig. 10.3c). The output of the BG performs a form of stochastic hill-climbing over the value function (Magdoom et al., 2011). Thus, by way of searching for the maxima in the value function, in the early stages of learning, the BG module drives the motor cortex to make reaching movements to the target. In the model however this value computation (V ^arm (t)) is not a result of training; value is presented as an explicit function of distance between the end effector (X ^arm) and the goal position (X ^targ) in Eq. (10.15). The σ _V term defines the spatial range over which the value function is sensitive for that particular target.

Fig. 10.3

Basal ganglia and value function. The network of the basal ganglia (a) which receives cortical input, ΔG(t), and via the direct and indirect pathways computes ΔG(t + 1). The indirect pathway has a 2D sheet of reciprocally connected STN–GPe neurons whose dynamics is governed by the lateral (neighborhood Gaussian connectivity, green STN, red GPe) and interconnectivity in these layers. The two-link arm (b) represented by blue lines for the links and green circle for the end effector, approaching a goal position (red circle) and computing the value function which peaks at the target location (c) in this case is a target at [0 0]. The BG dynamics essentially constitutes a stochastic hill-climbing mechanism that seeks to maximize the value function in order to reach the target

$$ V^{\text{arm}} \left( t \right) = \exp \left( {\frac{{ - \left\| {X^{\text{targ}} - X^{\text{arm}} } \right\|^{2} }}{{\sigma_{V}^{2} }}} \right) $$

(10.15)

• Stochastic Hill-Climbing

The input to the striatum comes from the motor cortex in the form of a difference vector. We hypothesize that this difference vector which is the change in the motor cortical activity (ΔG(t)) is the drive for the sustenance of motor activity. This information is then modulated via the direct (projections from D1-expressing neurons in the striatum) and the indirect (projections from D2-expressing neurons in the striatum) pathways as a function of the dopamine signal [Eqs. (10.17) and (10.18)]. From previous studies (Chakravarthy & Balasubramani, 2015; Magdoom et al., 2011), we have shown that this switching between direct and indirect pathways can be carried out using a form of the temporal difference signal called the value difference [Eq. (10.16)].

$$ \delta_{V} = V^{\text{arm}} \left( t \right) - V^{\text{arm}} \left( {t - 1} \right) $$

(10.16)

The quantity δ _v in Eq. (10.16) is called the value difference, which is subtly different from the temporal difference error. We proposed earlier that value difference also correlates with dopamine signals just as TD error has been suggested to be represented by dopamine signals (Chakravarthy & Balasubramani, 2015; Magdoom et al., 2011; Muralidharan et al., 2013). Value difference signals are thought to be carried by nigrostriatal connections to the striatum, where they modulate the responses of striatal projection neurons to cortical inputs as follows:

$$ y_{{{\text{D}}1}} = \frac{1}{{1 + \exp \left( { - \lambda_{{{\text{D}}1}} \left( {\delta_{V} - t_{{{\text{D}}1}} } \right)} \right)}}\Delta G\left( t \right) $$

(10.17)

$$ y_{{{\text{D}}2}} = \frac{1}{{1 + \exp \left( { - \lambda_{{{\text{D}}2}} \left( {\delta_{V} - t_{{{\text{D}}2}} } \right)} \right)}}\Delta G\left( t \right) $$

(10.18)

where y _D1 and y _D2 represent the outputs of D1R- and D2R-expressing medium spiny neurons (MSNs), respectively. In the nonlinearity, λ _D1 and t _D1 and λ _D2 and t _D2 are the gains and the thresholds of the direct and indirect pathways, respectively. Also λ _D1 = −λ _D2 which suggests that when δ _v is positive (negative), the direct (indirect) pathway is selected. Since D2R-expressing MSNs of the striatum project to the GPe, y _D2 influences GPe neural dynamics, which in turn influences STN neural dynamics, as shown below.

$$ \tau_{\text{GPe}} \frac{{{\text{d}}x_{\text{GPe}} }}{{{\text{d}}t}} = - x_{\text{GPe}} + \varepsilon_{g} \sum {\sum {W^{\text{glat}} } } x_{\text{GPe}} + w_{sg} y_{\text{STN}} + y_{{{\text{D}}2}} $$

(10.19)

$$ \tau_{\text{STN}} \frac{{{\text{d}}x_{\text{STN}} }}{{{\text{d}}t}} = - x_{\text{STN}} + \varepsilon_{s} \sum {\sum {W^{\text{slat}} } } y_{\text{STN}} - w_{gs} x_{\text{GPe}} $$

(10.20)

$$ y_{\text{STN}} = \tan \,h\left( {\lambda_{\text{STN}} x_{\text{STN}} } \right) $$

(10.21)

λ _STN controls the slope of the sigmoid, thus the STN output. τ _STN and τ _GPe are the respective timescales of STN and GPe. The weight parameters that control the connection strengths between the STN and GPe are w _sg and w _gs , and the weights that control lateral connections within both the STN and the GPe layer are W ^slat and W ^glat with connection strengths ϵ _s and ϵ _g , respectively, which have a Gaussian neighborhood as defined in (10.22).

$$ W_{i,j,k,l}^{{{\text{glat}}/{\text{slat}}}} = \exp \left( { - \frac{{\left( {i_{g/s} - k_{g/s} } \right)^{2} + \left( {j_{g/s} - \text{l}_{g/s} } \right)^{2} }}{{\left( {\sigma_{\text{lat}}^{g/s} } \right)^{2} }}} \right) $$

(10.22)

The indirect pathway consisting of the STN and GPe forms a coupled excitatory–inhibitory pair of neuronal pools [Eqs. (10.19) and (10.20)]. Such excitatory–inhibitory pairs of neuron pools are known to exhibit complex oscillations (Kalva, Rengaswamy, Chakravarthy, & Gupte, 2012). The dynamics of these oscillators is highly dependent on the input, which constitutes the projections from the D2-expressing neurons of the striatum. The STN layer in the model exhibits correlated activity for high striatal input, and uncorrelated oscillatory activity for low striatal inputs (see Appendix in Chap. 5). The uncorrelated oscillations of the STN are a key source of exploratory drive that randomly pushes the arm around in the workspace.

Here, σ ^g/s_lat is the spread of the lateral connections, respectively, for the STN–GPe network. So for a given neuron i, j the weights represent a 2D Gaussian whose maximum is centered on (i, j). The output of the STN is combined in the GPi with the signal arriving via the direct pathway from the D1R-expressing MSNs in the striatum as follows:

$$ y_{\text{GPi}} = A_{{{\text{D}}1}} y_{{{\text{D}}1}} - A_{{{\text{D}}2}} y_{\text{STN}} $$

(10.23)

At the level of the GPi, the DP output, i.e., y _D1 and the STN output (y _STN) are combined (Eq. 10.23) and then passed on to the thalamus. The thalamus is modeled as a continuous attractor network which is necessary to integrate as well as filter information from the GPi output.

10.2.5 Prefrontal Cortex—Information of Goal Position

The motor command is thought to arise from the PFC, in the sense that the goal of the movement is represented in the PFC (Asplund, Todd, Snyder, & Marois, 2010; Matsumoto, Suzuki, & Tanaka, 2003). The PFC specifies information regarding the position of the goal to be reached. Similar to the PC and the MC layers, the PFC layer is trained like a SOM with weights W _PFC, but the input features are the spatial locations that the arm could reach in the model. Training in the model occurs in weights linking the PFC and the MC. The weights between the PFC and the MC (W _PFC→MC) are trained as follows. A target, X ^targ = [x ^targ₁ x ^targ₂ ] activates corresponding neurons in the PFC with activation U(t) using Eq. (10.24).

$$ U_{i} \left( t \right) = \exp \left( {\frac{{ - \left\| {X^{\text{targ}} \left( t \right) - \left. {W_{{{\text{PFC}},i}} } \right\|} \right.^{2} }}{{\sigma_{\text{PFC}}^{2} }}} \right) $$

(10.24)

The arm initially makes reaching movements which are driven by dynamics aided by a stochastic hill-climbing procedure called ‘Go-Explore-NoGo (GEN)’ applied to the value function (Chakravarthy & Balasubramani, 2015). Furthermore, whenever the arm reaches the target position, the connections from PFC and MC are also trained, so that the motor command can directly activate the motor cortex, thereby producing rapid movement, without the slow search conducted by the BG. In this case, the training is initiated only when the arm reaches the target, i.e., the end effector is within a small radius, ξ (=0.1 units) of the target location. Similar to the MC and MN layer weights, the W _PFC→MC are trained in a supervised fashion (Eq. 10.25). Let G _PFC be the activity that PFC activation induces in MC; let G _targ be the activity in MC that drives the arm to the target location. Therefore, G _targ serves as a target vector for G _PFC. The weights from PFC to MC are therefore trained as follows:

$$ \Delta W_{{{\text{PFC}} \to {\text{MC}}}} = \eta_{{{\text{PFC}} \to {\text{MC}}}} \left( {G^{\text{targ}} \left( t \right) - G^{\text{PFC}} \left( t \right)} \right)U\left( t \right) $$

(10.25)

Here, G ^PFC(t) is the PFC-driven MC activity and as this learning progresses, the arm reaches the goal position faster and faster. Therefore, the model exhibits two stages of motor learning: Slow movements dominated by the BG are seen in the early stages, while the cortically driven fast movements dominate the later stages. The PFC contribution increases as a function of the reaching error as in Eq. (10.26).

$$ A_{\text{PFC}} = A_{\text{PFC}} + \frac{{f_{t} }}{{r_{e} }} $$

(10.26)

Here, f _t is a factor for controlling the speed of growth of A _PFC and r _e is the reaching error estimated as average distance to the target.

10.2.6 Timescales of Motor Movement in the Cortex and the BG

Reaching movements, like several other behavioral events, involve dynamics at multiple timescales: the neuronal activity which is generally in milliseconds, and the actual movement which unfolds over the order of seconds. In the model, the cortical loop is assumed to run slightly slower than the BG module. The integration time step used is 1 ms. As the dynamics of the STN–GPe loop in the indirect pathway needs some time to settle, we run this loop for 50 iterations, before sending the output to the MC. Thus, a single update of the MC activity happens after every 50 ms during which the BG dynamics run. All the results presented are at the timescale of the MC.

10.2.7 Simulating Pathology—Parkinsonian Condition

The value difference term ‘δ _v ’, as we have mentioned earlier, is a correlate of the dopamine signal. To simulate the dopamine-deficient state of PD in the model, the ‘δ _v ’ term is clamped to a lower value. Thus if [d _low d _high] represents the normal range of the dopamine signal exhibited by the control subjects, then PD OFF conditions are simulated having a smaller range [d _low δ ^* _v ], where δ ^* _v denotes the clamped limit that is lesser than d _high [Eq. (10.27)]. In addition to this, PD ON conditions could also be simulated [Eq. (10.28)] in the model by adding a constant additive term which we call the medication factor to the value difference (δ ^med _v ).

$$ \text{PD}\;\text{OFF} :\;\begin{array}{*{20}c} {{\text{If}}\;\delta_{V} > \delta_{V}^{*} } \\ {\quad \delta_{V} = \delta_{V}^{*} } \\ \end{array} $$

(10.27)

$$ \text{PD}\;\text{ON} :\; \begin{array}{*{20}l} {{\text{If}}\;\delta_{V} > \delta_{V}^{*} } \hfill \\ {\quad \delta_{V} = \delta_{V}^{*} + \delta_{V}^{{\text{med}}} } \hfill \\ {\text{else}} \hfill \\ {\quad \delta_{V} = \delta_{V} + \delta_{V}^{{\text{med}}} } \hfill \\ \end{array} $$

(10.28)

Furthermore, the degeneration of the SNc neurons is not the only pathology linked with PD. Others areas of the BG such as the STN and GPe are shown to have pathological synchronized oscillations in the PD patients (Weinberger, Hutchison, & Dostrovsky, 2009). Pathological β-band oscillations in these loops have also been linked to PD tremor and rigidity (Mallet et al., 2008; Weinberger et al., 2009). Therefore, in addition to clamping δ _v , we also investigated other parameters such as the lateral connection strengths in STN and GPe neurons (W _slat & W _glat), the interconnection strengths w _sg and w _gs , and the relative contributions of the direct and the indirect pathways on the final motor action. Finally, there is definitely an influence of dopamine on the excitability of the cortical neurons. As a result to study these effects, we also introduced a variable for tonic dopamine levels (δ _ton), which is updated using the value difference using Eq. (10.29), to understand the effect of dopamine depletion in the higher cortical areas.

$$ \tau_{\text{ton}} \frac{{{\text{d}}\delta_{\text{ton}} }}{{{\text{d}}t}} = - \delta_{\text{ton}} + A_{\text{ton}} \delta_{V} $$

(10.29)

This gives an estimate of the averaged gradient information or the value function, which controls the dynamics of the MC. We made the tonic dopamine variable δ _ton control the strength of connectivity in the MC, i.e., A ^C_lat which controls the strength of lateral connectivity within the attractor network of MC using Eq. (10.30).

$$ \tau_{\text{MC}} \frac{{{\text{d}}A_{\text{lat}}^{C} }}{{{\text{d}}t}} = - A_{\text{lat}}^{C} + f\left( {\lambda_{\text{ton}} \left( {\delta_{\text{ton}} - \theta_{\text{ton}} } \right)} \right) + k $$

(10.30)

where f is a sigmoid function $\left( {f\left( \cdot \right) = \frac{1}{{1 + \exp \left( { - \cdot } \right)}}} \right)$, with slope λ _ton and threshold θ _ton and k is a constant to maintain baseline values of A ^C_lat .

10.3 Results

10.3.1 Mapping of the Joint Configurations in the PC and MC

The sensory-motor cortical loop is initially tested (Fig. 10.4a) as a stand-alone network, and the MC is activated to investigate the range of movements of the arm in the workspace. Activation to the MC is given as I _MC = I _PFC + I _BG + I _PC + I _app where I _PFC = I _BG = I _PC = 0 and I _app is a Gaussian current and a matrix of size N _MC × N _MC in which is centered on random nodes to activate different regions of the MC. In Eq. (10.31), i _MC and j _MC represent the nodes in the MC and i _r and j _r are random nodes over which the Gaussian current is centered.

Fig. 10.4

Sensory and motor maps. The sensory-motor loop is probed at the level of the MC (a) and the mapping of the end effector positions approximated by the network is compared to all possible positions in the arm workspace (b). The joint configuration maps formed for both the PC (c) and the MC (d) layers, where blue lines indicate the two links and the green dot denotes the end effector position

$$ I_{\text{app}} = \exp \left( { - \frac{{\left( {i_{\text{MC}} - i_{r} } \right)^{2} + \left( {j_{\text{MC}} - j_{r} } \right)^{2} }}{{\left( {\sigma_{\text{lat}}^{g/s} } \right)^{2} }}} \right) $$

(10.31)

We observe that the arm is capable of reaching most of the positions in the output space (Fig. 10.4b), suggesting a consistent mapping of arm configurations in the feature space. Furthermore, to understand the loop’s ability to represent arm positions uniquely, the activity generated upon probing the MC and the activity generated via the loop, i.e., arm → PC → MC are compared and are found to be the same. In addition, the activity of the PC and MC is mapped back onto the joint configuration space which produces map structures shown in Fig. 10.4c, d. In case of the PC map, the joint configuration space is fairly uniform and topography is well maintained, whereas just one level above in the MC the map starts to become more complex with both regions of continuous change in the configuration space and areas of fractures or discontinuities. The regions of overlapping representations seem to have increased from the PC map to the MC map.

10.3.2 Reaching Movements of the Arm

We initially tested the model by providing multiple targets to reach and to test if the arm reaches these areas. Figure 10.5a (1–6) shows a snapshot of the network in action as the arm reaches the target. The MC activity corresponds to the arm configuration that has successfully reached the target. The PFC activity codes for the goal position (represented by the red star). Initial movements of the arm are solely driven by the gradient information present in the value function (Eq. 10.15). The indirect pathway of the BG provides activity with low correlation under certain parametric conditions of STN–GPe connections (see BG column, STN–GPe in Table 10.1) to enable sufficient exploration of the arm in the workspace (Chakravarthy & Balasubramani, 2015). This in turn leads to training the connections between the PFC and the MC (Eq. 10.25). The PFC input to the MC specifies the activity that the motor cortex should evolve in order to reach the target. The GPi activity, which forms the input to the thalamus from the BG, is integrated in the thalamus. It is important to note that, when the MC activity and the PFC input into MC are the same, it means that the network has learned to approximate the activity needed to reach the target (Eq. 10.25). There are 50 trials in total in the simulation, where the initial 20 trials are used for learning the target location and the trajectory to follow for a successful reach, during which the amplitude of PFC input is increased as per Eq. (10.26) and the PFC-to-MC connections (W _PFC→MC) are trained. In the next 30 trials, the arm is tested for its performance. For each trial, the arm is initialized to a starting position and provided with a specific target (the target position is kept constant for all trials) to reach. A successful reach is signified by the arm coming within at least ξ units of distance from the target. The trial is then terminated in two cases: (a) when the arm reaches the target successfully or (b) when the target is not reached and the simulation crosses the maximum time limit.

Fig. 10.5

Reaching behavior in controls. The simulation snapshot of the model while performing the reaching task (a) and the activities of multiples areas in the model (a.1–6). The end effector trajectories (b) obtained in the case of controls for reaching three different targets (represented by three different colors—blue, green, and red) across trials as the learning of the PFC-to-MC connections (W_PFC→MC) takes place. The ellipses show the spatial ± SD as the model performs reaching across 50 trials. The velocity profiles during a reach (blue line) compared to the lognormal distribution (LN Fit, green dotted line) (c). Performance of the model in control conditions as a function of the time to reach target location (d) and the variability in reach times (e) through trials (here each block refers to 10 trials)

Table 10.1 Parameter values used for simulating the cortico-basal ganglia model

The end effector trajectories become smoother as learning progressed in controls and, furthermore, there is decrease in hand path variability as learning progresses (Fig. 10.5b). Here, the spatial variance is represented as ellipses (Georgopoulos, Kalaska, & Massey, 1981) and we see that the variance decreases with trials. We investigated the velocity profiles of the arm while performing the reach, and the characteristic of bell-shaped curve is observed in the profile (Fig. 10.5c). Additionally, it is known from previous work that these velocity profiles fit well to a delta-lognormal distribution (Plamondon, 1998), with the two lognormal components corresponding to the agonist and the antagonist bursts, respectively. We found that the reaching profiles obtained in the model fit well to this distribution (Fig. 10.5c). The performance of the arm also improves with trials seen as a decrease in the time taken to reach the target (Fig. 10.5d). There is a reduction in the variance of the time to reach across blocks of trials suggesting lower motor variability upon learning (Fig. 10.5d).

10.3.3 Velocity Profiles of Controls and PD Patients

Majsak and colleagues (Majsak, Kaminski, Gentile, & Flanagan, 1998) performed reaching experiments in both healthy controls and PD subjects at self-determined speeds to estimate changes in kinematics of the subjects under conditions where the objects are stationary and moving. The PD subjects were on their dopamine medication (Sinemet^®) and performed six trials in each case (a) when object was stationary, (b) moving, and (c) stationary again. In the model however we compared the performance of the arm in the stationary case to compare the basal-level activities. The PD ON condition in the model is simulated using Eqs. (10.27) and (10.28) where the clamped dopamine limit, δ ^* _V , is set to 0.1 and the medication factor (δ ^med _V ) to 0.5. In order to account for the slowness and reduced velocity in PD movement, the tonic dopamine variable (δ _ton) is introduced via Eq. (10.29) and affects the MC dynamics using Eq. (10.30).

The δ _ton values in controls and PD condition reveal that the controls show increase in the levels of tonic dopamine as the task progresses, whereas it is much smaller in case of PD (Fig. 10.6a). This affects the attractor dynamics of the MC, where the values of A ^C_lat are always high for the PD case compared to the control case where after some time the value falls due to the increase in δ _ton (Fig. 10.6b). Lower values of A ^C_lat lead to easy translation of the neural activity bump over the neural space, and high values make it difficult for the BG to trigger movements as a result of the local excitation and global inhibition dynamics. The velocity profiles of the subject groups for a self-determined speed are shown in Fig. 10.6c. The simulated controls reach the target faster than in the PD case, and their peak velocities are also higher. The kinematic variables of the reach task are shown in Fig. 10.6d–f. A significant difference is seen in the movement time between the controls and the PD subjects, suggesting slow or bradykinetic movements in the PD case.

Fig. 10.6

Stationary target reaching task. The evolution of the tonic dopamine variable (δ _ton) (a) and the MC dynamics variable (A ^C_lat ) (b) for the entire task duration (50 trials) for controls, PD1 and PD2. Here, PD1 and PD2 refer to different clamped dopamine levels (δ ^* _V ), which is 0.1 (blue line) and 0.01 (red line), respectively. Kinematics of reaching movements with velocity profiles of controls and PD (c), time to peak velocity (d), peak velocity (e), and movement time (f) from experiment (adapted from Majsak et al., 1998) and the corresponding model performance

10.3.4 Model Performance on the Pursuit Task

Another set of experiments captured by the model included the pursuit task conducted by Soliveri and group where they tracked the ability of PD subjects to pursue a moving target using a manipulandum (Soliveri, Brown, Jahanshahi, Caraceni, & Marsden, 1997). In the model, this task is abstracted to the arm trying to reach for a series of continuously changing target positions. The target moves back and forth in a straight line in a sinusoidal fashion with a frequency of 0.25 Hz, thereby making it predictable. There were three blocks in the experiment each including 10 trials. The behavioral variable measured both in the experiment and model is percentage of time on target which is defined as the time spent on the target using the manipulandum (experiment) or arm (model). As previously described, in the model the arm is thought to be on the target if the distance to the target is within ξ (=0.1) units. The PD subjects included in the study were on their DA medication. In the model, we designed the target to move similarly as in the experiment, in a sinusoidal fashion. As the target shifted, the activation in the PFC also changed at every instant of time as it codes for the target location. This meant that the peak of the value function (Eq. 10.15) also changes continuously with time giving the arm the necessary information to track the moving target. The δ ^* _V is set to −0.2 and the δ ^med _V to 0.01. From Fig. 10.7a, it is evident that the controls are capable of pursuing the moving target more efficiently than PD ON subjects which the model captures (Fig. 10.7b), even though both subject groups showed learning as the trials progressed. This phenomenon is also observed in the experiment where the PD subjects also learn the task across blocks (Fig. 10.7c, d). The last block in the experiment suggests a performance drop in both the control and PD subjects which the authors attribute to fatigue. This is not captured in the model as we did not take into account the factor of fatigue in the muscle model. See Table 10.1 for parameter values used for simulating the experimental conditions.

Fig. 10.7

Pursuit reaching task. The performance of subjects on the pursuit task (adapted from Soliveri et al., 1997) and the differences observed in control and PD behavior in experiment (a and c) and the model (b and d)

10.3.5 Motor Initiation with the Cortico-BG Loop

The relative strengths of the BG along with the PFC inputs into the MC are analyzed to understand the dynamics of the cortico-basal ganglia loop on movement initiation. The arm was initialized to a starting configuration, and the PFC input is provided as pulses of duration (50 ms) with varying amplitudes. The displacement of the arm from its starting position is tracked. In the presence of only PFC (i.e., A _BG = 0), the amplitude of PFC input to initiate sufficient movement has to be high (A _PFC > 0.9). The introduction of the BG with varying degree of strengths (A _BG = 0.01, 0.05, 0.1, and 0.2) leads to movement at lower amplitude of PFC input, thus making motor initiation easier, though with higher contribution of BG the movement variability increases (Fig. 10.8a). With the introduction of PD condition (δ ^* _V  = 0, δ ^* _V  = −1 and w _sg  = w _gs  = 3, which causes synchronized oscillations of the STN–GPe loop), keeping A _BG constant, we again see a tendency toward higher input strengths of PFC required to initiate movement (Fig. 10.8b). We observe that there needs to be compensation from higher cortical areas in disease conditions for reaching movement and could be interpreted as a deficiency in voluntary movement initiation due to the impaired BG.

Fig. 10.8

Motor initiation in the cortico-BG loop. The displacement of the arm from a starting position with varying degree of the PFC and BG input strengths (A _PFC, A _BG) (a) and the effect of PD condition (b) on motor initiation

10.3.6 PD Symptoms

The model is further extended to understand several motor symptoms commonly seen in PD. In the model, three cardinal symptoms of PD movement are simulated: tremor, rigidity, and bradykinesia. Initially to simulate PD condition, the value difference is clamped, but that alone does not reproduce all the above symptoms in the model. Other parameters also must be varied as shown below.

• Tremor, Rigidity, and Bradykinesia

PD symptoms start appearing in the model when the dopamine signal (δ ^* _V  = −1) is clamped and the connection strengths between the STN and GPe (w _sg and w _gs ) and the lateral connection strengths in the STN (ϵ _s ) are manipulated. In all the symptom cases, the interconnections, i.e., w _sg and w _gs are increased from the control levels (w _sg and w _gs  = 1 in controls, w _sg and w _gs  = 3 in PD). Controls have a smooth trajectory to the target (Fig. 10.9a), and tremor starts to appear initially with just the increase in the connection strength between STN and GPe, i.e., w _sg and w _gs (Fig. 10.9b). Rigidity and bradykinesia seem to coexist in the model and start to emerge upon decreasing the contribution of the lateral connections within the STN (ϵ _s ) compared to the tremor case (Fig. 10.9c) (for rigidity ϵ _s  = 0.7). We estimated the frequency spectrum of the velocity of the arm during the reaching task and in the tremor case (Fig. 10.9e). There is increased power in the 4–10 Hz range which is seen clinically in PD patients as well (Jankovic, 2008). This also brings about differences in the spectrogram of the average STN activity in the tremor and rigidity scenarios compared to control scenario. In the control case, the STN activity remains sufficiently decorrelated (Fig. 10.9g). However in both the symptomatic cases, power of the spectrum seems to be concentrated within a narrow frequency range: tremor (frequency range = 10–40 Hz) and rigidity (frequency range = 20–55 Hz) (Fig. 10.9h, i). Various studies have observed pathological oscillations in the indirect pathway especially in the STN–GPe loop and are generally related to the symptoms observed in the PD patients (Hammond, Bergman, & Brown, 2007; Mallet et al., 2008; Weinberger et al., 2009). These pathological oscillations are in the β-band (13–30 Hz), and this is seen in the model where the frequency spectrogram of the average STN activity shows increased power in the β-range. In the rigidity case, the spectrum shifts to higher frequency band compared to Parkinsonian tremor. This shift in the spectrum seems to result in the arm restricted to a very small part of the state space, thereby reflecting movements that are slower and more rigid.

Fig. 10.9

Emergence of PD symptoms in the model. Movement trajectories (dark green trace) of the arm during reaching a target (red dot) in controls (no δ ^* _V , w _sg and w _gs  = 1, ϵ _s  = 1) (a), PD tremor (δ ^* _V  = −1, w _sg and w _gs  = 3, ϵ _s  = 1) (b), and PD rigidity (δ ^* _V  = −1, w _sg and w _gs  = 3, ϵ _s  = 0.7) (c) conditions. The frequency spectrograms of movement in controls (d), PD tremor (e), and PD rigidity (f), where PD tremor shows increased power in the 4–10 Hz regions. The spectrogram of the averaged STN activity in controls (g), during PD tremor (h), and rigidity (i). The spectrum shows a shift into higher frequencies in case of the PD symptoms (rigidity > tremor)

The three symptomatic conditions are also presented as a function of distance to the target position (Fig. 10.10a). On further exploration of the entire range of values for the STN–GPe interconnections (w _sg and w _gs ) and the intraconnections (ϵ _s ), the regimes of disease states seem to appear (Fig. 10.10b). In Fig. 10.10b, the values are obtained as mean of the Fourier spectrum of the arm velocity for each condition. Therefore, higher mean values of the Fourier spectrum suggest more tremor-like behavior and the lower values suggest more rigid and bradykinetic movements. Intermediate values could be control-like behavior with more a balanced frequency spectrum as shown in Fig. 10.9d. It suggests that at a given w _sg the range of ϵ _s at which tremor or rigidity appears is different. This could be a reason for high symptom variability among PD patients and the bias toward development of certain symptoms earlier in the disease compared to later. However, the general trend suggests that decrease in the lower lateral strengths in the STN may be a major causative reason for rigidity and higher interconnection strength for tremor.

Fig. 10.10

Influence on network parameters on PD symptoms. PD symptoms viewed as a function of the distance to the target position (a). The analysis of the strength of interconnections within STN–GPe (w _sg and w _gs ) and the lateral connection strength within the STN (ϵ _s ) and their effect on the type of symptoms manifested in PD is represented as the expected value of the Fourier spectrum of velocity (b). The blue and red range represents rigidity/bradykinetic, and tremor movements, respectively. The control regime lies in the green range

10.4 Discussion

We present a cortico-basal ganglia model that performed reaching tasks. By inducing PD conditions in the model, we are able to simulate the impairments seen in PD reaching movements. There are two different loops in the model: the sensory-motor cortical loop and the cortico-basal ganglia loop. So the model is on the lines of optimal control theory and closed-loop control where the sensory-motor integration provides the necessary feedback mechanisms for control; the cortico-basal ganglia loop sets the optimality criterion to maximize performance (Todorov, 2004). In this case, the criterion becomes the value function: Stochastic gradient descent dynamics executed by the BG model essentially drives the reaching movements of the arm toward a target. One of the assumptions in the model is that this value function may be readily available to the BG module by the top-down information from higher cortical areas. A plausible mechanism could be the prefrontal cortical connections to the ventral striatum (Alexander, Crutcher, & DeLong, 1991; Botvinick, 2008), which code the goal information in the form of value function at the level of ventral striatum. Alternatively, it is not essential to assume that such a value function can be constructed readily from the goal information; the value can also be constructed in the ventral striatum by the plasticity of cortico-striatal connections that combine the goal information from the prefrontal cortex with the sensory-motor information from the sensory-motor cortical projections to the ventral striatum.

The acquisition of motor skill requires learning at several levels (Hikosaka et al., 2002). One of the key points of the model is during the early phase of learning, when the movements are slower and more dynamic, the BG is dominant. As the cortical learning progresses, specifically PFC-to-MC training in the model, the movements become quicker and more directed toward the target location. This phenomenon is actually seen in monkeys performing associative learning tasks, where initially the neuronal activity is higher in the striatal areas, suggesting responses to rewards, and a slow increase in the activity of the prefrontal cortex (Pasupathy & Miller, 2005). These studies also suggest that the output of BG trains the higher cortical areas and this is precisely captured in the model. There is a significant contribution of striatum during initial stages of learning, and as the cortical systems start to take over, actions become habitual and automatic (Ashby, Turner, & Horvitz, 2010). The initial slower movements in the model are driven by climbing the value function (i.e., dopamine-dependent), and as the PFC contribution increases, it can activate other cortical areas like MC which can directly influence spinal motor neurons for faster movements.

10.4.1 Cortico-Basal Ganglia Loop as an Attractor Network

The proposed model highlights the idea that the final hand position due to reaching is an attracting state of the cortico-basal ganglia dynamics. Therefore, the attractor dynamics of the cortico-basal ganglia loop must be well understood in order to understand reaching dynamics in normal and Parkinsonian conditions. The attractor dynamics that drives the reaching movements in the model arises from three sources: (1) the lateral connections in the CANN model of MC, (2) the sensory-motor cortical loop dynamics, and (3) the cortico-basal ganglia loop dynamics. The attractor dynamics of the CANN model has been explored extensively in other studies (Gupta et al., 2013; Jankovic, 2008). The attractor dynamics of the sensory-motor cortical loop, even in the absence of the cortico-basal ganglia loop, is demonstrated in Fig. 10.8a, where it is shown that the PFC input to MC must exceed a threshold to move the hand from its current state. Addition of the cortico-basal ganglia loop seems to lower the threshold; it is easier initiate the hand movement for a given PFC activation, if there is assistance from the cortico-basal ganglia loop. This result explains the relative difficulty observed in PD patients in initiating hand movements (Chen & Reggia, 1996). In Fig. 10.8b, the PD pathology is investigated slightly differently. Instead of removing the BG input to MC, the δ _V term, which represents dopamine projections to the striatum, is clamped at two levels (clamp value, δ ^* _V  = 0, and δ ^* _V  = −1; w _sg  = w _gs  = 3). These parameter settings suppress the dopamine signal (δ _V ) and also change the dynamics of STN–GPe loop to synchronized dynamics, emulating PD conditions (Chakravarthy & Balasubramani, 2015). Under these conditions also, it can be seen that it is harder to initiate movement, even though the strength of BG input to MC is unaffected. These results show that the dynamics of the cortico-basal ganglia loop amplifies the output of MC, thereby facilitating movement in normal condition. This amplification is suppressed in PD conditions, due to reduced dopamine levels and increased synchronization of the STN–GPe loop. These results resonate well with the model of the role of BG in willed action proposed in (Chakravarthy, 2013).

10.4.2 Indirect Pathway for Exploration and Emergence of PD Symptoms

The STN–GPe loop modeled as a network of coupled oscillators induces exploratory dynamics in the model [Eqs. (10.19), (10.20), and (10.21)]. We have in previous studies substantiated the role of the indirect pathway as an Explorer that performs random search over the action space which is necessary when viewing basal ganglia as a reinforcement learning engine (Chakravarthy & Balasubramani, 2015). The exploration in the model comes from the fact that there is a stochastic drift in the activity of MC, influenced by the complex dynamics of the STN–GPe, thereby driving the arm to visit all possible arm configurations. As a result, the indirect pathway becomes very important in the initial trials to drive arm movements where the movement variability is also high (Fig. 10.5d, e). In the initial part of a reaching trial, value difference (δ _V ) is small which makes the striatal output to GPe and GPi low. Therefore, in the initial part of a trial, the output of BG is dominated by the output of the STN–GPe loop, which facilitates movement initiation. However, once movement begins, δ _V changes significantly strongly reflecting the gradient of the value function. In the PD case, the clamping of the value difference (δv) in the model enhances the outputs of D2R neurons in the striatum and amplifies the contributions of the indirect pathway to BG output. Thus, in PD conditions, the BG output depends on the dynamics of the STN–GPe loop, with altered dynamics of STN–GPe manifesting as impaired movement (Table 10.2).

Table 10.2 STN–GPe parameters and relation to PD symptoms (rigidity and tremor)

Table 10.2, which summarizes the results from Fig. 10.9, shows the parameters of STN–GPe loop under control and PD conditions. Particularly, it shows that the internucleus connections (w _gs and w _sg ) are high compared to the control for both the symptom categories (rigidity and tremor). Furthermore, in case of rigidity, the STN lateral connectivity strength (ϵ _s ) is lower than in case of tremor. It is known that symptoms in PD could be correlated with synchronized oscillations in the STN, which is often seen in the β-range (Mallet et al., 2008; Weinberger et al., 2009). Studies show that the oscillatory activity in the STN ranges from the low frequency 3–7 Hz to beta (13–30 Hz) in the more dorsal regions to even gamma (30–100 Hz) in the ventral areas (Zaidel, Spivak, Grieb, Bergman, & Israel, 2010). The model concurs with this where we see the STN activity ranging from desynchronized in control case (Fig. 10.9g) to synchronized beta in tremor (Fig. 10.9h) condition and high beta (20.5–28 Hz), bordering on gamma (25–100 Hz) during rigidity (Fig. 10.9i). There is not much evidence on how such pathological oscillations give rise to both tremor and rigidity. An interesting observation from the model was that both tremor and rigidity were associated with different frequency bands of the STN activity (as shown in Fig. 10.9h, i), with rigidity associated closer to the gamma range compared to the tremor. It remains to be verified whether such firing patterns exist in the basal ganglia under conditions of rigidity and tremor.

10.4.3 Effect of Dopamine on Motor Performance

In the model, the dopamine signal (δ _v ) aids in switching between the direct and the indirect pathways of the BG. In order to model the effect of dopamine on the motor cortex, we define the tonic dopamine variable, δ _ton, [Eq. (10.29)], which controls the lateral inhibition in the CANN component of the MC. This tonic dopamine variable is a local-time-averaged version of phasic dopamine (Eq. 10.29). There seems to be a higher degree of intracortical inhibition with the application of dopamine agonists to the motor cortex and a significant decrease in this inhibition upon the administration of dopamine antagonist (Ziemann, Tergau, Bruns, Baudewig, & Paulus, 1997). In PD ON subjects, the peak velocity of reaching and the acceleration of movement are higher and the time spent in deceleration is lower compared to the OFF case suggesting its benefit in reducing bradykinesia (Castiello, Bennett, Bonfiglioli, & Peppard, 2000). In the model, these effects are reproduced by making the A ^C_lat parameter in MC a function of the tonic dopamine variable δ _ton. The A ^C_lat parameter modulated by the tonic dopamine affects the intrinsic excitability of the CANN component of MC (Fig. 10.6b), where larger values of A ^C_lat make the CANN dynamics more stable and resistant to any changes in the input that comes from areas, viz. BG, PC and PFC.

10.4.4 Limitations and Future Directions

The immediate limitation of the model is the lack of a distinct striatal module; instead we have used striatal activation functions to modulate the cortical input entering the BG. Since the arm used is a 2D kinematic model (it could be extended to 3D naturally), the introduction of nonlinear muscle model with force dynamics would aid in understanding the agonist–antagonist interaction during movement. In future, we would like to extend the model as a test bench to analyze reaching movement impairments in other basal ganglia pathologies like Huntington’s chorea, ballismus, dystonia, and even drug-induced dyskinesias (Jankovic, 2008). Since the model is generalized in its approach, by involving different end effectors, like locomotor apparatus or articulators, we can understand motor behavior such as gait and speech, respectively (Canter, 1963; Hausdorff, Cudkowicz, Firtion, Wei, & Goldberger, 1998). One of the interesting results from the model is that with BG impairment, movement initiation becomes difficult and would require more voluntary effort to do so. This could be tested by stimulating motor areas using techniques like TMS (transcranial magnetic stimulation) in PD patients and see whether movement initiation requires more amplitude of stimulation than controls. This would also enhance the theory of the BG as an active player in regulating willed action (Chakravarthy, 2013). We show that shift in PD symptoms from tremor to rigidity could be caused by an increase in the correlated activity of the STN neurons. Experiments could target the changing activity in the STN and look for similar changes in movement behavior. Finally, the attractor dynamics, local excitation and global inhibition, of the MC in the model could be manipulated by using DA agonist and antagonists to see which aspects of these dynamics does dopamine have an influence on.