Abstract
Many brain areas are activated by the possibility and receipt of reward. Are all of these brain areas reporting the same information about reward? Or are these signals related to other functions that accompany reward-guided learning and decision-making? Through carefully controlled behavioral studies, it has been shown that reward-related activity can represent reward expectations related to future outcomes, errors in those expectations, motivation, and signals related to goal- and habit-driven behaviors. These dissociations have been accomplished by manipulating the predictability of positively and negatively valued events. Here, we review single neuron recordings in behaving animals that have addressed this issue. We describe data showing that several brain areas, including orbitofrontal cortex, anterior cingulate, and basolateral amygdala signal reward prediction. In addition, anterior cingulate, basolateral amygdala, and dopamine neurons also signal errors in reward prediction, but in different ways. For these areas, we will describe how unexpected manipulations of positive and negative value can dissociate signed from unsigned reward prediction errors. All of these signals feed into striatum to modify signals that motivate behavior in ventral striatum and guide responding via associative encoding in dorsolateral striatum.
Access provided by Autonomous University of Puebla. Download chapter PDF
Similar content being viewed by others
Keywords
1 Introduction
Imagine that you ritualistically purchase your morning coffee from the same place every day, but one day, you have a bad experience and the coffee is subpar. What will you do? Will you continue to habitually follow your routine or will you take your business elsewhere? It seems like a relatively simple computation, but it is remarkable to think about how many different brain signals come into play in situations like these. At the very least, you need to break the habit and be motivated to pursue new goals. This involves detection of errors and allocation of attention so that new associations can be formed. Subsequently, you must consider potential options and their economic value, as well as the probability of achieving those options. In this chapter, we examine neural correlates of these functions in animals performing tasks during which past and potential experiences—both positive and negative—modify behavior.
Over the last two decades, the number of brain areas in which activity has been shown to be modulated by rewards and cues that predict reward has escalated dramatically. In fact, all the brain areas illustrated in Fig. 1a contain neurons that increase firing to cues that predict reward and to reward themselves (and even this is not an all-inclusive list). This raises an important question: Are all these brain areas encoding the exact same information or are they subserving different functional aspects of reward processing? Recent work has tried to tease apart these functions to determine what exactly is being encoded by nodes in this circuit. Below, we describe single neuron recordings in behaving animals that have addressed this issue.
This chapter is broken down into four sections. Each section describes experiments that have tried to parse “reward -related” neural activity into the functions illustrated in Fig. 1b. This figure tries to encapsulate all possible functions that go into making simple decisions based on anticipated reward and punishment. It is clear that this is an extremely complex computation! Several of the arrows linking proposed functions are bidirectional, illustrating that most mechanisms involved are highly interrelated and influence each other. When examining neural correlates of these functions, great care needs to be taken when linking firing patterns of single units to functions related to decision-making.
In the first section, we examine studies that distinguish value from other signals that covary with value, such as motivation and salience. Learned value can be defined as the relative anticipated worth that some cue predicts, either positive or negative. Motivation is the enhancement or decrement of motor output based on an increased or decreased level of arousal. More specifically, a stimulus is “salient” if it leads to a general increase in arousal or is attention grabbing, whereas something is “motivational” if it enhances motor behaviors. In the first section, we will focus on dissociating “value” signals from signals related to motivation and salience. Notably, the dissociation between the latter two is much more difficult to study and requires further investigation; however, signals directly related to motivated motor output should be observed in the period leading up the behavioral response, whereas signals related to attention or salience might solely be during the presentation of cues (i.e., salient events), but not necessarily the actions associated with them.
In the second section, we examine neurons that increase firing to the delivery of unexpected outcomes, which are critical for reporting surprising events so that learning can occur. Several brain areas respond to reward delivery but few specifically report errors in reward prediction. To uncover these correlates, experimental paradigms must violate expectations in both positive (outcome better than expected) and negative (outcome worse than expected) directions. This allows researchers to dissociate signed from unsigned prediction error signals. Signed prediction error signals lead to changes in the associate strength of conditioned stimuli that predict anticipated value, whereas unsigned prediction error signals lead to increased attention so that learning can occur.
All of these signals (value, motivation, prediction errors, attention, etc.) must modulate systems that guide behavior. The striatum is thought to be one major interface that integrates this information with motor output. In the third section, we will describe studies that demonstrate a basic trend along the diagonal of striatum (ventral-medial to dorsal-lateral) by which correlates are more reward-related in ventral-medial regions, whereas correlates are more associative and motor-related in more dorsal-lateral sections. Finally, in the fourth section, we discuss where all of these “reward-related” signals might be integrated into a common signal.
2 Value Versus Motivation and Salience
Neurons that increase firing prior to a desirable reward might be encoding value or they might reflect the increased motivation or salience associated with receiving this reward. It is not trivial to dissociate value from motivation or salience because they covary in most situations; that is, the more you value something, the more motivated you are to work to obtain it (Solomon and Corbit 1974; Daw et al. 2002; Lang and Davis 2006; Phelps et al. 2006; Anderson 2005; Anderson et al. 2011). The literature has extensively examined neural systems involved in making decisions based on potential outcomes, both good and bad, but whether these signals reflect value or motivation/salience is still not entirely clear. As described above, we define motivation as a process that invigorates motor responding and salience as cues that are attention grabbing or arousing. The two are intertwined; salient cues might lead to increased motivation, both of which might be triggered by cues that predict valued reward and induce faster responding. Importantly, neural correlates related with value can be dissociated from motivation and salience by examining firing to aversive stimuli, which have low value but are highly salient and motivating (Fig. 2a). A classic example is that of the carrot and the stick. A donkey finds both salient and motivating, but the carrot has high value, whereas the stick has low value.
By manipulating both anticipated appetitive and aversive events, experimental procedures can dissociate value from motivation and salience. That is, cues that predict appetitive and aversive outcomes have opposite values, but both are highly salient and motivational. Several studies have done just that, dissociating these signals by motivating behavior with both the promise of reward and the threat of punishment. In these experiments, animals learn that conditioned stimuli (CS) predict potential rewards or the possibility of punishment. Typically, there are three trial types where the CS predicts: (1) a large reward (e.g., sucrose, juice); (2) a neutral condition or a small (or no) reward; and (3) a small (or no) reward with the threat of an aversive outcome, such as delivery of a bitter quinine solution, electric shock, or air puff to the eye (Rolls et al. 1989; Roesch and Olson 2004; Roesch et al. 2010a; Bissonette et al. 2013; Matsumoto and Hikosaka 2009; Brischoux et al. 2009; Anstrom et al. 2009; Lammel et al. 2011). If neurons encode value, neural activity should show a monotonic relationship during appetitive, neutral, and aversive trials (Fig. 2a; theoretical neural signals). If activity is modulated by factors like motivation or salience that vary with the strength of appetitive and aversive stimuli, neurons should respond with the same “sign” for appetitive and aversive trials compared to neutral trials. This approach has been applied to several brain areas thought to represent value in one form or another.
2.1 Orbitofrontal Cortex (OFC)
Most evidence prior to the described work clearly suggested that activity in OFC reflected value . OFC is strongly connected with the limbic system and has proven to be a key associative structure (Pickens et al. 2003; Burke et al. 2008; Ostlund and Balleine 2007; Izquierdo et al. 2004). Neurons in OFC respond to cues that predict differently valued rewards, as well as to the anticipation and delivery of rewards themselves. Activity in OFC is modulated by a number of economic factors (e.g., probability, effort, delay, and size) for both appetitive and aversive stimuli. Its firing is also influenced by the availability of alternative rewards (i.e., relative reward value) and how satiated the animal is. Still, all these neural representations might reflect how salient or motivating the predicted reward is, not its value.
To disentangle the two, Roesch and Olson recorded neural activity in OFC during performance of a task in which monkeys responded to cues that indicated the size of the reward the monkey would receive if correct (one or three drops of juice) and the size of a penalty that would be incurred upon failure (a 1 s or 8 s time-out). Figure 2a illustrates the 3 trial types. The first promises 3 drops of liquid reward with no threat of punishment. This trial type has high value and high motivation as demonstrated by high accuracy and fast reaction times. The second trial type (neutral) has low value and motivation because it only promises a small reward with no risk of punishment. The critical trial type is the third one. It also promises only a small reward but the threat of punishment is high; thus, it has low value but is highly motivational as evidenced by high accuracy and faster reaction times similar to those observed on high reward trials (Fig. 2b). Thus, choice rate and response latencies showed that monkeys were more motivated by large rewards and penalties as compared to smaller/neutral trials, allowing for the dissociation of value and motivation via simultaneous manipulation of appetitive and aversive outcomes (Roesch and Olson 2004).
In this study, OFC neural activity was found to encode the value associated with cues, as opposed to their motivational properties. That is, OFC neurons fired most strongly for cues that predicted large reward and least strongly for cues that predicted large penalty relative to neutral conditions (Fig. 2c). This was in stark contrast to neurons in an area of cortex more strongly associated with motor output, the premotor cortex, which fired at a higher rate during the preparation to move to achieve a large reward and to avoid the large penalty—i.e., they appeared to encode a factor related to the level of motivation to respond (Fig. 2d). Other studies have replicated these results in OFC and have further shown populations of neurons that encode potential rewards offers, the identity of specific rewards expected or obtained, and the option that is eventually chosen during performance of choice tasks (Hosokawa et al. 2007; Morrison et al. 2011; Padoa-Schioppa and Assad 2006, 2008).
From these and other studies in primates, it has been suggested that OFC encodes an abstract “common currency” (Padoa-Schioppa and Assad 2006; Padoa-Schioppa and Cai 2011; Rudebeck and Murray 2014); however, rodent studies have emphasized that OFC can also encode specific outcomes, representing the sensory qualities of available outcomes and potential maps of task/environmental space (Wilson et al. 2014; Schoenbaum et al. 2011a; Schoenbaum and Eichenbaum 1995). Some primate studies also support this notion, showing that OFC can convey sensory and informational aspects of reward in addition to their hedonic properties (Wallis and Miller 2003; Wallis et al. 2001; Tsujimoto et al. 2011, 2012). Nevertheless, all of these studies agree that OFC is critical for signaling predictions about future outcomes in the service of reward-guided decision-making .
2.2 Nucleus Accumbens
Next, we turn our discussion to the nucleus accumbens core (NAc), which receives strong glutamatergic projections from OFC pyramidal neurons. NAc has been described as the “critic” in actor-critic models of reinforcement learning, which model NAc as a generator of value predictions that are subsequently used by dopamine (DA) neurons to compute prediction errors necessary for updating actions polices in the “actor” (e.g., dorsal striatum) (Redish 2004; Joel et al. 2002; van der Meer and Redish 2011; Padoa-Schioppa 2011; Barto 1995; Niv and Schoenbaum 2008; Sutton and Barto 1998; Takahashi et al. 2008; Houk et al. 1995; Haber et al. 2000; Ikemoto 2007). In addition to this proposed role, NAc has been traditionally described as the “limbic-motor interface,” motivating behaviors in response to both appetitive and aversive stimuli, and not for representing value per se. Consistent with both of these theories, pharmacological manipulations of NAc impact motivated behaviors dependent on value expectations during a variety of tasks (Cardinal et al. 2002a, b; Berridge and Robinson 1998; Di Chiara 2002; Ikemoto and Panksepp 1999; Salamone and Correa 2002; Di Ciano et al. 2001; Wadenberg et al. 1990; Wakabayashi et al. 2004; Yun et al. 2004; Gruber et al. 2009; Stopper and Floresco 2011; Ghods-Sharifi and Floresco 2010; Floresco et al. 2008; Blokland 1998; Giertler et al. 2003), including reward seeking (Ikemoto and Panksepp 1999), cost-benefit analysis (Stopper and Floresco 2011; Floresco et al. 2008), and delay/effort discounting (Ghods-Sharifi and Floresco 2010; Cardinal et al. 2001). Furthermore, single-unit recordings have clearly demonstrated that neural activity in NAc is modulated by the value associated with cues that predict reward in rats (Setlow et al. 2003; Janak et al. 2004; Carelli and Deadwyler 1994; Day et al. 2011; Ito and Doya 2009; Goldstein et al. 2012; Nicola et al. 2004; van der Meer et al. 2010; van der Meer and Redish 2009; Lansink et al. 2010; Kalenscher et al. 2010) and monkeys (Cromwell et al. 2005; Shidara and Richmond 2004; Schultz et al. 1992; Kim et al. 2009; Nakamura et al. 2012) performing a variety of instrumental tasks, including go/no-go (Setlow et al. 2003; Schultz et al. 1992), lever pressing (Janak et al. 2004; Carelli and Deadwyler 1994; Day et al. 2011; Cromwell et al. 2005; Shidara and Richmond 2004), discrimination (Goldstein et al. 2012; Nicola et al. 2004; van der Meer et al. 2010), maze running (van der Meer and Redish 2009; Lansink et al. 2010; Kalenscher et al. 2010), and eye movement paradigms (Kim et al. 2009; Nakamura et al. 2012).
Thus, the basic finding across many of these studies is that single-unit activity in NAc is modulated by cues that predict reward after an instrumental response is performed. As above, this activity might reflect value or motivation. Therefore, application of a paradigm similar to the one described above allowed for an exploration of these different potential roles. In a study by Bissonette et al., rats performed a task where they were motivated by both reward (sucrose solution) and threat of punishment (quinine delivery). Rats found sucrose and quinine appetitive and aversive, respectively, as illustrated by increased and decreased licking, (Fig. 2e), but were most strongly motivated by cues that predicted either a large possible sucrose reward or a possible quinine punishment, as illustrated by better accuracy and faster reaction times for these cues, compared to a neutral cue (Neu) which only predicted a small sucrose reward, similar to the neutral cue in the aforementioned primate study (Fig. 2f, g). In addition, cues that predicted potential reward and punishment (odor cues) were presented before stimuli (lights) that instructed the instrumental response to dissociate value signals from specific motor planning.
Interestingly, activity of separate populations of single neurons in NAc encoded either value or motivation prior to the response instruction (i.e., light). This result suggests that NAc might represent the value of the goal the rat is working for as well as the motivational level associated with differently signed outcomes. Activity of some neurons was stronger for conditioned stimuli that predicted large reward and weaker for punishment trials relative to neutral trials (i.e., value representation; Fig. 2h). Other NAc neurons fired strongly for cues that predicted reward and punishment, respectively (i.e., motivation; Fig. 2i). These results suggest that NAc fulfills both motivational and evaluative functions via separate neuronal populations and might be critical for integrating both types of information as the “limbic-motor interface,” as well as the “critic” (Bissonette et al. 2013). Thus, NAc appears to be a common junction point for concurrently signaling value and motivation , leading to the invigoration of particular behavioral actions over others. This idea is consistent with pharmacological studies which suggest that DA in the NAc disrupts the ability to modify behavior based on the current value of predicted outcomes (Burton et al. 2013; Singh et al. 2010), as well as to interfere with behavioral measures of motivation or salience (Nunes et al. 2013; Salamone 1994; Salamone and Correa 2012; Salamone et al. 1991, 2012; Koch et al. 2000; Berridge 2007; Lex and Hauber 2010; Salamone 1986; McCullough and Salamone 1992).
2.3 Parietal Cortex
Finally, the most recent brain area to be scrutinized by separating value from salience is the parietal cortex. Parietal neural activity is dependent on the value of expected actions (action-value) (Platt and Glimcher 1999; Sugrue et al. 2004; Louie and Glimcher 2010) and is thought to be critical for making economic decisions (Louie and Glimcher 2010; Gold and Shadlen 2007; Sugrue et al. 2005; Rangel and Hare 2010). Others suggest that this signal reflects increased salience induced by the promise of a better reward, not the value associated with it. For example, Leathers and Olson reported that primate lateral intraparietal (LIP) neurons fire most strongly when a behavior is associated with a large versus small reward and more importantly, to cues that predicted a large versus small penalty (Leathers and Olson 2012). They argued that this signal reflected increased salience because activity was high for both large reward and penalty. They also suggest that this reflected salience (as opposed to motivation) because it did not span the delay between the cue and the behavioral response. This study has sparked considerable debate between influential leaders in the field (Leathers and Olson 2012, 2013; Newsome et al. 2013). Further work is necessary to determine whether salience versus value encoding in parietal cortex is task or procedurally specific. One possible middle ground in this debate might be that neurons in LIP, like NAc, encode both properties, and that one function is emphasized over the other depending on context.
3 Signed Prediction Error Versus Attention /Salience
The above discussion focused on neural firing leading to a decision and whether or not it reflects the value of potential goals or the motivation associated with obtaining those goals. Here, we consider what happens after the predicted outcome is or is not delivered, and whether it was better or worse than expected. These signals may be used to update previously described “value” and “motivation” signals when contingencies change.
Many brain areas increase activity in response to unexpected delivery of reward. Most commonly these signals are interpreted as “reward prediction signals.” However, this signal might also reflect other functions, such as changes in attention that result due to the delivery of unexpected reward. Note that this activity cannot represent motivation because motivation usually increases with training as one learns to expect the more valued reward. Furthermore, signals related to motivation are maintained as long as the animal is not satiated and/or is actively pursuing reward. The signals that we will describe in this section attenuate with learning, as rewards become anticipated (i.e., no longer unexpected).
We will discuss two types of prediction errors, signed and unsigned. Signed prediction error (PE) signals strengthen or weaken associations in downstream brain areas during learning and adaptive decision-making . In the coffee example, if the coffee is bad, positive associations with the coffee shop must be attenuated (negative PE) so that it is no longer sought after, but if it is really good, these associations must be strengthened (positive PE) to promote coffee-seeking at that specific coffee shop. In contrast, unsigned prediction errors modulate attention. In the example, attention increases both when the coffee is really great and when it is very bad. Increased attention is necessary to determine what in the environment caused the deviation from reward expectations (e.g., new coffee brand or barista). Attention and prediction error signals likely work together in an intricate manner that has not been fully realized in the literature. Prediction errors are necessary to increase attention, and attention is needed to detect and learn from prediction errors. How these two interact in the brain is an interesting question, but as a first step, we must dissociate these two highly interrelated functions. Below we describe studies that accomplish this by manipulation of both positive and negative events, in an unpredictable manner.
3.1 Midbrain Dopamine (DA) Neurons
Many midbrain dopamine (DA) neurons signal signed PEs that are essential for reinforcement learning (Steinberg et al. 2013). Phasic bursting of DA neurons intensifies when an outcome occurs that is “better” than predicted (positive PE), and firing of DA neurons decreases or is inhibited when an outcome is “worse” than anticipated (negative PE). When rewards are accurately predicted, outcomes elicit “no” change in DA neural activity (Schultz 1997, 1999). Over the course of learning, phasic DA bursting “shifts” from occurring at the time of reward delivery to the time of reward-predicting cues that precede outcomes. Dopaminergic reward prediction error signals are commonly referred to as a “teaching signals,” updating decision circuits about changes in contingencies so that behaviors can be modified when expectations are violated (Montague et al. 1996; Schultz 1998; Bromberg-Martin et al. 2010a).
Although PE signaling is frequently studied in the context of animals learning to approach cues that elicit an appetitive outcome, positive prediction errors appear to be elicited in any situation that is construed as being better than expected [e.g., short delay to reward, low effort to achieve reward, high probability (Roesch et al. 2010b)]. Positive PE signals are even induced when predicted negative events do not occur. This has been observed in primates, where DA neurons that increase firing to unexpected rewards also increase firing when an expected air puff is not delivered (Bromberg-Martin et al. 2010a), and when rats do not receive an anticipated foot shock during an avoidance procedure (Oleson et al. 2012).
The literature has recently turned its focus to how these DAergic prediction error signals may be derived. In its simplest form, reward prediction error computations require two pieces of information: the reward prediction and the reward that was actually received (actual minus predicted). As mentioned above, computational models refer to structures that signal predictions as the “critic.” They calculate the expected value based on the summed values of environmental cues. Structures proposed to fill this role include the nucleus accumbens, prefrontal cortex, and/or amygdala (O’Doherty et al. 2004; Belova et al. 2008; Balleine et al. 2008; Daw et al. 2005). Indeed, several studies have shown that information from the orbitofrontal and prefrontal cortex is necessary for expectancy-related changes in phasic firing of midbrain dopamine neurons (Takahashi et al. 2009, 2011; Jo et al. 2013).
Others have demonstrated that an error signal also occurs upstream of DA neurons. The lateral habenula (LHb) (Bromberg-Martin et al. 2010a; Hikosaka 2010; Matsumoto and Hikosaka 2007), which is thought to receive error information from globus pallidus border (GPb) (Hong and Hikosaka 2008), also signals signed reward prediction errors but in the opposite sign. Activity of these neurons is excited or inhibited by negative or positive prediction errors, respectively. Furthermore, prediction error signaling in LHb occurs earlier than those in DA neurons and stimulation of LHb inhibits DA firing (Matsumoto and Hikosaka 2007; Bromberg-Martin et al. 2010b). DA neurons are likely to receive this information via the adjacent rostromedial tegmental nucleus (RMTg) and indirect connections through midbrain GABA neurons in the ventral tegmental area (VTA) of the midbrain (Hong et al. 2011; Ji and Shepard 2007; Omelchenko et al. 2009; Jhou et al. 2009; Kaufling et al. 2009; Brinschwitz et al. 2010). Together these results suggest that DA neurons receive widespread information regarding reward predictions and prediction errors themselves.
Signed prediction errors are not the only information conveyed by midbrain DA neurons. Some midbrain DA neurons also signal motivational salience and are excited by both rewarding and aversive events. This signal is thought to support systems for orienting, cognitive processing, and motivational drive (Matsumoto and Hikosaka 2009; Bromberg-Martin et al. 2010a). These neurons are identified on a gradient, being more prominently found in ventromedial substantia nigra pars compacta (SNc) (Matsumoto and Hikosaka 2009) and tapering toward the VTA. These salience signaling DA neurons show increased activity for cues which signal either positive or negative potential events, such as foot shock and air puff (Brischoux et al. 2009; Anstrom et al. 2009; Lammel et al. 2011). PE- or value-related signals are more commonly found in the ventromedial SNc area of the midbrain and the lateral VTA. In contrast, motivational salience signals are more predominantly found in the dorsolateral SNc (Matsumoto and Hikosaka 2009). Interestingly, DA neurons in these different areas of the midbrain project to different downstream regions. DA neurons in the salience predominant regions project preferentially to areas of the prefrontal cortex such as dorsolateral prefrontal cortex (DLPFC) and to NAc, in particular the core of the nucleus accumbens, while DA neurons in the value- and PE-predominant region project mainly to ventromedial PFC and to the shell of the nucleus accumbens in the NAc (Bromberg-Martin et al. 2010a). In this way, DA signaling is able to both modify representations of expected outcomes, and the need for motivation and attention to salient features in the environment when contingencies are violated (Bromberg-Martin et al. 2010a).
3.2 Basolateral Amygdala (ABL)
Like midbrain DA neurons, several reports have now suggested that ABL also signals reward prediction errors. Traditionally, ABL was thought to serve many of the same functions as OFC consistent with their reciprocal connections (Bechara et al. 1999; Malkova et al. 1997; Kesner and Williams 1995; Hatfield et al. 1996; Cousens and Otto 2003; Parkinson et al. 2001; Jones and Mishkin 1972; Winstanley et al. 2004; Churchwell et al. 2009; Ghods-Sharifi et al. 2009; Cardinal 2006). Studies examining the encoding of both appetitive and aversive signals in OFC and ABL have shown that the two are dependent on each other for normal encoding when rats discriminate stimuli that predict appetitive (sucrose) and aversive (quinine) outcomes during reversal learning (Roesch et al. 2007a, 2010a, b; Haney et al. 2010; Saddoris et al. 2005; Schoenbaum 2004; Schoenbaum et al. 1998, 1999, 2000, 2003a, 2006, 2007, 2009, 2011a; Schoenbaum and Esber 2010; Schoenbaum and Roesch 2005; Stalnaker et al. 2007; Rudebeck and Murray 2008, 2011, 2014; Rudebeck et al. 2013a, b).
Although amygdala is important for signaling expected outcomes by acquiring and storing associative information related to both appetitive and aversive outcomes (LeDoux 2000; Murray 2007; Gallagher 2000; Schoenbaum et al. 2003b; Ambroggi et al. 2008), it also supports other functions related to associative learning, such as signaling attention to salient cues, uncertainty about outcome probability, and intensity of stimuli (Morrison et al. 2011; Saddoris et al. 2005; Tye and Janak 2007; Tye et al. 2010; Belova et al. 2007). These studies have shown that activity in ABL is modulated by the predictability of both appetitive and aversive events, specifically when predictions are violated (Roesch et al. 2010a, b; Tye et al. 2010; Belova et al. 2007). During events that are highly salient and attention grabbing, such as when an outcome is unexpectedly delivered or when an expected outcome is omitted, ABL neurons increase firing (Roesch et al. 2010a). In rats, it has been reported that activity in ABL increases when rats were expecting reward, but it was not delivered during extinction (Tye et al. 2010). Modulation of neural activity in lateral and basal nuclei of amygdala by expectation has also been described during fear conditioning in rats (Johansen et al. 2010). In primates, unexpected delivery of appetitive or aversive (air puff) outcomes caused amygdala neurons to fire more strongly than when they were completely predictable (Belova et al. 2007). Additionally, the same populations of ABL neurons which represent appetitive stimuli were also activated by aversive stimuli, regardless of the particular sensory modality from which the experience comes (Shabel and Janak 2009). These studies suggest a critical role for ABL in multiplexing signals related to attention, which may be modulated by salience or intensity, as well as signaling the associated value of cues and outcomes.
This hypothesis is consistent with data showing that outcome encoding ABL neurons exhibit increased firing to rewards that are unexpectedly delivered and omitted in a task in which reward expectations were violated by varying its size and the time to its delivery (Roesch et al. 2010a). Recall, these correlates are different than what we have described for DA neurons; DA neurons increase and decrease firing during unexpected delivery of reward, respectively. In this experiment, unexpected up-shifts in value occurred whenever a reward was larger than expected or arrived earlier than anticipated (Fig. 3a). On the other hand, down-shifts in value occurred whenever reward was unexpectedly smaller or was delayed (Fig. 3b). In this study, neurons in ABL tended to exhibit higher firing for both up- and down-shifts, whereas DA neurons exhibited increases and decreases in firing, respectively.
Two common models for interpreting neural signals relating to expected and actual outcomes are the Rescorla–Wagner (Rescorla and Wagner 1972) (R-W) model (Fig. 3c) and the Pierce–Hall (PH) (Pearce and Hall 1980) model (Fig. 3d). The R-W model uses errors to drive the change in associative strength, where larger errors cause larger changes in associative strength, smaller errors drive smaller changes, and if no error occurs, there is no change in associative strength. Since these error values are computed from the expected versus actual outcomes, the sign of errors may be positive (outcome is better than expected) or negative (outcome is worse than expected) as observed in the firing of DA neurons (Fig. 3e). An alternative model presented by PH uses the absolute (unsigned) value of a prediction error to determine the amount of attention needed on following trials. A large prediction error will warrant a large increase in attention as observed in the firing of ABL neurons (Fig. 3f), whereas a small prediction error will elicit a corresponding smaller change in attention. Notably, this signal would take several trials to develop because attention on previous trials must be taken into account. Importantly, both models involve changes that are proportional to the size of the prediction error, but because the R-W model uses signed PEs, the changes in strength of PEs over learning is opposite for better than expected versus worse than expected trials (Fig. 3c). In contrast, in the PH model the changes in strength over training are equivalent for both of those types of trials (Fig. 3d). Note, although there is strong evidence that ABL signals are unsigned, consistent with PH signals, there is also some evidence for a population of ABL neurons which instead encode signed prediction errors (Belova et al. 2007; Klavir et al. 2013).
3.3 Anterior Cingulate (ACC)
One brain area that likely receives and transmits prediction error-related information to ABL is the ACC (Klavir et al. 2013). ACC has strong reciprocal connections with ABL (Dziewiatkowski et al. 1998; Cassell and Wright 1986; Sripanidkulchai et al. 1984) and has been shown to be involved in a number of functions related to error processing, conflict monitoring, behavioral feedback, and attention (Totah et al. 2009; Walton et al. 2004; Rushworth et al. 2004, 2007; Wallis and Kennerley 2010; Rudebeck et al. 2008; Rushworth and Behrens 2008; Hillman and Bilkey 2010; Matsumoto et al. 2007; Amiez et al. 2005, 2006; Sallet et al. 2007; Hayden et al. 2011; Quilodran et al. 2008; Kennerley et al. 2006, 2009; Kennerley and Wallis 2009; Ito et al. 2003; Carter et al. 1998; Holroyd and Coles 2002; Oliveira et al. 2007; Paus 2001; Scheffers and Coles 2000; Rothe et al. 2011; Magno et al. 2006). During tasks that manipulate the predictability of reward, ACC neurons signal unsigned prediction errors (Hayden et al. 2011; Bryden et al. 2011a) consistent with theories put forth by Pearce and Hall as described above. Further, it has been shown that activity in ACC was elevated at the beginning of behavioral trials after reward contingences changed unexpectedly and that these changes in firing were correlated with behavioral measures of attention (Bryden et al. 2011a). This is different than what is typically described in ABL, where modulation after violations in reward prediction induces attention-related changes at the time of reward delivery, not during subsequent presentations of cues. While these studies are consistent with the role that ACC might play in allocating attention when prediction errors occur, other primate studies have suggested that ACC can also signal prediction errors of the signed variety (Klavir et al. 2013; Matsumoto et al. 2007).
4 Correlates of Motivation and Associative Encoding in Striatum
In the above sections, we dissociate value from motivation and salience and define signals specifically related to signed prediction errors, which increase and decrease associative strength, and unsigned prediction errors, which increase attention so that learning can occur. How do these signals impact behavior? One conduit is striatum; however, even in striatum signals are not simply integrated and transformed into motor output. Striatum is a structure with many overlapping neural correlates related to reward-guided and stimulus-driven behaviors. Neurons there have been shown to be responsive to cues that predict reward, during initiation of actions, the anticipation of reward, and the delivery of reward (Yin and Knowlton 2006). Although there are many overlapping correlates, there appears to be a basic trend along the diagonal of striatum (ventral-medial to dorsal-lateral) by which correlates are more reward-related in ventral-medial regions, whereas correlates are more associative and motor-related in more dorsal-lateral sections.
This has been demonstrated by neural recordings in primates that have divided the caudate (one of three main regions of the primate striatum, of which the dorsal-medial striatum in rats is the closest anatomical homologue) into three subdivisions (dorsal, central, and ventral) (Nakamura et al. 2012) based on anatomical connections with cortical and subcortical structures as reported by Haber and Knutson (2010), and in accordance with tripartite subdivisions observed in humans (Fig. 4b) (Karachi et al. 2002). In this study, monkeys performed a task in which they fixated at central spot and then responded to a target to the left or the right of fixation. During different blocks of trials, one of those produced a large reward, the other a small reward (Fig. 4a).
As expected from the anatomy, the functional segregation as the recording electrode passed through ventral-medial to dorsolateral striatum reflected the progression of limbic to associative to sensorimotor afferents (Haber and Knutson 2010; Lau and Glimcher 2007, 2008; Samejima et al. 2005; Stalnaker et al. 2010). Recordings obtained more ventral medially demonstrated that activity was driven by the expected size of the reward, with fewer representations related to the action necessary to achieve it. Ventral-medial portions of the caudate, including the central and ventral part, tended to fire more strongly for large versus small rewards than dorsal caudate (Fig. 4c) and fewer neurons were modulated by the direction of the behavioral response compared to dorsal caudate. This is in contrast to dorsal-lateral portions of caudate which exhibited high directionality and reward selectivity but showed no preference for large over small reward. This study suggests a continuum of correlates with ventral-medial aspects of striatum reflecting value, and more dorsal-lateral regions better reflecting associative and motor aspects of behavioral output.
Similar results have been obtained in studies which examine the difference across the ventral-medial/dorsal-lateral divide in monkeys (Hollerman et al. 1998; Apicella et al. 1991; Cai et al. 2011) and in rats (van der Meer et al. 2010; Takahashi et al. 2007; Wang et al. 2013). For example, in a task where odor cues predicted differently valued rewards and the direction necessary to achieve those rewards (Fig. 5a), the majority of NAc neurons fired significantly more strongly for cues that predicted high-value outcomes for actions made in one particular direction (into the cell’s response field) (Roesch et al. 2009; Burton et al. 2015). Further, faster response times (a behavioral measure of more motivated behavior) were correlated with higher firing rates. These data suggested that activity in NAc represents the motivational value associated with chosen actions necessary for translating cue-evoked value signals into motivated behavior as discussed above (Bissonette et al. 2013; Roesch et al. 2009; McGinty et al. 2013; Catanese and van der Meer 2013). These findings are consistent with those described in Sect. 1, showing that separate NAc neurons encode both value and motivation. To characterize neurons, we performed an ANOVA with value (high or low) and direction (contralateral or ipsilateral to the recording site) as factors on activity collected during the decision period (cue onset to response). In NAc, roughly equal proportions of neurons were selective for contralateral and ipsilateral response directions; however, of those selective for value, the large majority fired more strongly for high-value reward (Fig. 5b; NAc).
In contrast to NAc, encoding in dorsolateral striatum (DLS) was remarkably associative, representing all aspects of the task; outcome type, response direction, and the specific identity of conditioned stimuli (Burton et al. 2014, 2015). For example, the neuron in Fig. 5c responded the most strongly for odor cues that predicted a short delay after moving in the contralateral direction. Many DLS neurons were selective for very specific combinations of task events related to future outcomes, responses, and the stimuli that preceded them; however, there was not a preponderance of neurons that showed increased firing for any one combination over another, as was observed in NAc. If we characterize single neurons, the same as we did for NAc, we see that there is a preponderance of neurons showing a preference for contralateral movements, but of those that are value selective, equal proportions preferred high- and low-value reward (Fig. 5b; DLS). These results suggest that neural correlates in NAc are more closely tied to the motivational level associated with choosing high-value goals, whereas correlates in DLS are more associative, representing expected outcomes and response directions across a range of stimuli.
Although NAc and DLS are not directly connected, more ventral-medial regions like NAc likely impact more dorsal-lateral areas in striatum via the “spiraling” connectivity with dopamine (DA) neurons (Joel et al. 2002; van der Meer and Redish 2011; Niv and Schoenbaum 2008; Takahashi et al. 2008; Houk et al. 1995; Haber et al. 2000; Ikemoto 2007). The circuitry allows propagation of information from limbic networks to associative and sensorimotor networks so that behaviors can be more stimulus driven and therefore more efficient (Haber et al. 2000; Ikemoto 2007; Haber and Knutson 2010; Haber 2003; Balleine and O’Doherty 2010). This connectivity is consistent with ideas that behavior is first goal-directed and then become habitual with extended training (for more information on habitual action mechanisms, please see the chapter by O’Doherty in this volume). However, several studies have now shown that the NAc and DLS can function independent of each of each other, especially after learning. That is, these functions do not seem to always work in series, but can operate in parallel. For example, we have found that NAc lesions do not impair task-related correlates in DLS, but instead enhances neural correlates related to stimulus and response processing (Burton et al. 2013). These results suggest that rats normally use goal-directed mechanisms to perform the task, but rely more heavily on stimulus–response (S-R) processes after the loss of NAc (Lichtenberg et al. 2013). Consistent with this interpretation, rats recovered function after initial disruption observed after NAc lesions. Similar results have been described for NAc after loss of DLS function; rats recover function after an initial impairment was observed due to DLS interference (Nishizawa et al. 2012). This recovery was abolished with subsequent NAc lesions suggesting that it was NAc that was guiding behavior in DLS’s absence (Nishizawa et al. 2012). Together, these results suggest that NAc and DLS both guide behavior normally during performance of reward-related tasks and that these two regions can compensate for each other if the need arises. Note that this is not to suggest that they do not normally interact, particular during learning or when reward contingencies change.
5 Integration of Positive and Negative Information into a Common Output Signal
How independent representations of positive and negative valences are converted to a unified representation of expected outcome value, which ultimately leads to motivated behavior, is still unclear. A number of brain areas are thought to represent abstract value in the service of making comparisons, allowing the brain to compare apples to oranges (or coffee to tea). Under choice paradigms, Lee and colleagues have found integrative encoding of value in several brain regions (dorsal striatum, ventral striatum, and DLPFC) using an intertemporal choice task (Cai et al. 2011; Kim et al. 2008). Other studies have clearly shown neural activity reflecting value in several frontal areas in primate cortex (Kim et al. 2008; Roesch and Olson 2005a, b). In primates, several prefrontal regions contain neurons that integrate several economic factors, including cost, size, delay, and probability (Padoa-Schioppa and Assad 2006; Padoa-Schioppa and Cai 2011; Rich and Wallis 2014; Wallis and Kennerley 2011; Kennerley et al. 2011; Wallis 2007; Padoa-Schioppa 2007). However, others have reported that distinct prefrontal areas encode rewards and punishments, with ventral and dorsal aspects being more active for appetitive and aversive trial types, respectively (Monosov and Hikosaka 2012). It is difficult to find one clear path by which predicted outcomes lead to motivated behavior. This is likely because there are multiple paths that work in parallel, which interact at different phases of behavior and are highly dependent on the context. For an in depth discussion on valuation systems in the brain in the context of decision-making processes, see Redish et al., in this volume. It has also been difficult to clearly separate value signals from signals that might represent attention, salience, and/or motivation. With that being said, there is a trend by which brain areas more closely tied to attention and motor networks appear to better reflect the integration of economic functions.
For example, both ACC and parietal cortex are thought to be critical for functions related to attention and motor planning, but they have also been described as encoding value in the service of making economic decisions. The fact that value and salience signals have been observed in these regions might reflect the need for increased attentional control when potential rewards are available. Increases in neural firing depending on the value of expected actions would help ensure that neural processes are prioritized depending on expected events (whether they are positive or negative). Interestingly, integration of value predictions and spatial attention has been shown to be integrated in clusters of neurons in primate prefrontal and parietal cortex (Kaping et al. 2011). Further research will need to be done to fully appreciate how specific nodes contribute to the process of transforming value signals into executive control signals.
Out of all the rat brain areas that we have recorded from in our laboratory, only the firing of DA neurons exhibited a strong correlation between delay and size of reward, reflecting a common representation of value. DA neurons fired more strongly to cues that predicted high reward, whether it be a short delay or a large-sized reward, and were inhibited by cues that predicted low reward, whether it be a long delay or a small sized reward. Importantly, these were the same neurons that encoded reward prediction errors during unexpected reward delivery and omission. This is interesting, considering that dopaminergic input is thought to build associations regarding value in these brains areas. In our rat studies, very few brain areas appear to be computing this function. That is, we found correlates related to delay length and reward magnitude in multiple brain areas including OFC, ABL, ACC, DMS, and DLS, all of which maintained dissociable activity patterns, which were signaled by different neurons. Even in NAc, where the population of neurons fired more strongly for cues that predicted shorter delays to reward and larger magnitude rewards, and whose activity was correlated with motivated output (i.e., reaction time), there was only a weak trend for single neurons to represent both size and delay components. Although some neurons did encode both factors at the single cell level, many more neurons represented one economic variable but not the other. This trend toward common encoding likely reflects the conversion of expected outcome information into appropriate motor signals at the level of NAc (i.e., limbic-motor interface). Consistent with this hypothesis, when looking even further downstream, activity in the substantia Nigra pars reticulata (SNr) does appear to reflect a common evaluation of goals, which likely better reflects its role as a motor output structure rather than a reporter of economic value (Bryden et al. 2011b). Even within SNr, significant correlations between delay and size were relatively weak, suggesting that we might have to move even deeper into the motor system to find a common output signal for similarly valued outcomes.
6 Conclusion
Conditioned stimuli that predict reward simultaneously set into motion several functions as illustrated in Fig. 1a. These functions are highly interrelated but can be distinguished through manipulations of reward certainty, positive and negative outcomes, and division into goal- and stimulus-driven behaviors. To a degree, these functions map on to specific brain areas in the reward/decision circuit depicted in Fig. 1b, but they clearly depend on each other and there is redundancy within the circuit. OFC appears to represent value expectancies necessary for guiding decision-making and learning. These signals are informed by ABL, which represents associative information as well as the intensity or salience of behavioral outcomes. Simultaneously, OFC and ABL broadcast their information to NAc and DA neurons. Prediction error and salience signals generated by DA neurons provide feed-forward information to more dorsal-medial and dorsal-lateral regions in striatum, which are critical for goal-directed and habitual behaviors, respectively. These regions receive specific inputs from cortex consistent with these roles. The medial regions of striatum receive dense projections from the medial prefrontal cortex (mPFC) and the anterior cingulate cortex (ACC). The DLS is densely innervated by sensorimotor cortex (SMC), but also receives projections from mPFC and ACC. Importantly, basal ganglia signals that loop back onto cortex are likely to be critical for modulating representations related to reward-guided decision-making in cortical areas and can drive behavior in their own right. Cortical input from parietal and ACC likely increase attention al control in conjunction with value encoding to ensure that neural processes are prioritized in downstream areas depending on expected actions and errors in reward prediction.
Well-designed behavioral tasks that continue to dissociate these highly related brain functions, combined with techniques that can probe for causality are necessary to continue to disentangle the roles that neural correlates play in converting positive and negative events into abstract representations of value and motivated behavior, and to elucidate critical relationships between nodes within the circuit. Understanding how these circuits work to produce behavior allows us to look for alterations in neural signals in animal models of human disorders, such as models of psychological disorders, drug addiction, and aging (Hernandez et al. 2015; Roesch et al. 2007b, 2012a, b; Gruber et al. 2010; Stalnaker et al. 2009). Future therapeutic methods (behavioral, psychological, pharmacological, etc.) should focus on restoring the lost or changed signals as observed in these animal models to restore lost functions observed in psychiatric disorders.
References
Ambroggi F, Ishikawa A, Fields HL, Nicola SM (2008) Basolateral amygdala neurons facilitate reward-seeking behavior by exciting nucleus accumbens neurons. Neuron 59:648–661
Amiez C, Joseph JP, Procyk E (2005) Anterior cingulate error-related activity is modulated by predicted reward. Eur J Neurosci 21:3447–3452
Amiez C, Joseph JP, Procyk E (2006) Reward encoding in the monkey anterior cingulate cortex. Cereb Cortex 16:1040–1055
Anderson AK (2005) Affective influences on the attentional dynamics supporting awareness. J Exp Psychol Gen 134:258–281
Anderson BA, Laurent PA, Yantis S (2011) Value-driven attentional capture. Proc Natl Acad Sci USA 108:10367–10371
Anstrom KK, Miczek KA, Budygin EA (2009) Increased phasic dopamine signaling in the mesolimbic pathway during social defeat in rats. Neuroscience 161:3–12
Apicella P, Ljungberg T, Scarnati E, Schultz W (1991) Responses to reward in monkey dorsal and ventral striatum. Exp Brain Res. Experimentelle Hirnforschung. Experimentation cerebrale 85:491–500
Balleine BW, O’Doherty JP (2010) Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacol: Official Publ Am Coll Neuropsychopharmacol 35:48–69
Balleine BW, Daw ND, O’Doherty JP (2008) Multiple forms of value learning and the function of dopamine. In: Glimcher PW, Camerer CF, Fehr E, Poldrack RA (eds) Neuroeconomics: decision making and the brain. Elsevier, Amsterdam
Barto A (ed) (1995) Adaptive critics and the basal ganglia. MIT Press, Cambridge
Bechara A, Damasio H, Damasio AR, Lee GP (1999) Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making. J Neurosci 19:5473–5481
Belova MA, Paton JJ, Morrison SE, Salzman CD (2007) Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55:970–984
Belova MA, Patton JJ, Salzman CD (2008) Moment-to-moment tracking of state value in the amygdala. J Neurosci 28:10023–10030
Berridge KC (2007) The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology 191:391–431
Berridge KC, Robinson TE (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev 28:309–369
Bissonette GB et al (2013) Separate populations of neurons in ventral striatum encode value and motivation. PLoS ONE 8:e64673
Bissonette GB, Gentry RN, Padmala S, Pessoa L, Roesch MR (2014) Impact of appetitive and aversive outcomes on brain responses: linking the animal and human literatures. Frontiers Syst Neurosci 8:24
Blokland A (1998) Reaction time responding in rats. Neurosci Biobehav Rev 22:847–864
Brinschwitz K et al (2010) Glutamatergic axons from the lateral habenula mainly terminate on GABAergic neurons of the ventral midbrain. Neuroscience 168:463–476
Brischoux F, Chakraborty S, Brierley DI, Ungless MA (2009) Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci USA 106:4894–4899
Bromberg-Martin ES, Matsumoto M, Hikosaka O (2010a) Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68:815–834
Bromberg-Martin ES, Matsumoto M, Hikosaka O (2010b) Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons. Neuron 67:144–155
Bryden DW, Johnson EE, Tobia SC, Kashtelyan V, Roesch MR (2011a) Attention for learning signals in anterior cingulate cortex. J Neurosci: Official J Soc Neurosci 31:18266–18274
Bryden DW, Johnson EE, Diao X, Roesch MR (2011b) Impact of expected value on neural activity in rat substantia nigra pars reticulata. Eur J Neurosci 33:2308–2317
Burke KA, Franz TM, Miller DN, Schoenbaum G (2008) The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 454:340–344
Burton AC, Bissonette GB, Lichtenberg NT, Kashtelyan V, Roesch MR (2013) Ventral striatum lesions enhance stimulus and response encoding in dorsal striatum. Biol Psychiatry
Burton AC, Bissonette GB, Lichtenberg NT, Kashtelyan V, Roesch MR (2014) Ventral striatum lesions enhance stimulus and response encoding in dorsal striatum. Biol Psychiatry 75:132–139
Burton AC, Nakamura K, Roesch MR (2015) From ventral-medial to dorsal-lateral striatum: neural correlates of reward-guided decision-making. Neurobiol Learn Mem 117:51–59
Cai X, Kim S, Lee D (2011) Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron 69:170–182
Cardinal RN (2006) Neural systems implicated in delayed and probabilistic reinforcement. Neural Netw 19:1277–1301
Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ (2001) Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292:2499–2501
Cardinal RN, Parkinson JA, Hall J, Everitt BJ (2002a) Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 26:321–352
Cardinal RN et al (2002b) Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats. Behav Neurosci 116:553–567
Carelli RM, Deadwyler SA (1994) A comparison of nucleus accumbens neuronal firing patterns during cocaine self-administration and water reinforcement in rats. J Neurosci 14:7735–7746
Carter CS et al (1998) Anterior cingulate cortex, error detection, and the online monitoring of performance. Science 280:747–749
Cassell MD, Wright DJ (1986) Topography of projections from the medial prefrontal cortex to the amygdala in the rat. Brain Res Bull 17:321–333
Catanese J, van der Meer M (2013) A network state linking motivation and action in the nucleus accumbens. Neuron 78:753–754
Churchwell JC, Morris AM, Heurtelou NM, Kesner RP (2009) Interactions between the prefrontal cortex and amygdala during delay discounting and reversal. Behav Neurosci 123:1185–1196
Cousens GA, Otto T (2003) Neural substrates of olfactory discrimination learning with auditory secondary reinforcement. I. Contributions of the basolateral amygdaloid complex and orbitofrontal cortex. Integr Physiol Behav Sci 38:272–294
Cromwell HC, Hassani OK, Schultz W (2005) Relative reward processing in primate striatum. Exp Brain Res. Experimentelle Hirnforschung. Experimentation cerebrale 162:520–525
Daw ND, Kakade S, Dayan P (2002) Opponent interactions between serotonin and dopamine. Neural Networks: Official J Int Neural Network Soc 15:603–616
Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704–1711
Day JJ, Jones JL, Carelli RM (2011) Nucleus accumbens neurons encode predicted and ongoing reward costs in rats. Eur J Neurosci 33:308–321
Di Chiara G (2002) Nucleus accumbens shell and core dopamine: differential role in behavior and addiction. Behav Brain Res 137:75–114
Di Ciano P, Cardinal RN, Cowell RA, Little SJ, Everitt BJ (2001) Differential involvement of NMDA, AMPA/kainate, and dopamine receptors in the nucleus accumbens core in the acquisition and performance of pavlovian approach behavior. J Neurosci 21:9471–9477
Dziewiatkowski J et al (1998) The projection of the amygdaloid nuclei to various areas of the limbic cortex in the rat. Folia Morphol (Warsz) 57:301–308
Floresco SB, St Onge JR, Ghods-Sharifi S, Winstanley CA (2008) Cortico-limbic-striatal circuits subserving different forms of cost-benefit decision making. Cogn Affect Behav Neurosci 8:375–389
Gallagher M (2000) The amygdala and associative learning. In: Aggleton JP (ed) The amygdala: a functional analysis. Oxford University Press, Oxford, pp 311–330
Ghods-Sharifi S, Floresco SB (2010) Differential effects on effort discounting induced by inactivations of the nucleus accumbens core or shell. Behav Neurosci 124:179–191
Ghods-Sharifi S., St Onge, J.R. & Floresco, S.B. Fundamental contribution by the basolateral amygdala to different forms of decision making. J Neurosci 29, 5251-5259 (2009)
Giertler C, Bohn I, Hauber W (2003) The rat nucleus accumbens is involved in guiding of instrumental responses by stimuli predicting reward magnitude. Eur J Neurosci 18:1993–1996
Gold JI, Shadlen MN (2007) The neural basis of decision making. Annu Rev Neurosci 30:535–574
Goldstein BL et al (2012) Ventral striatum encodes past and predicted value independent of motor contingencies. J Neurosci: Official J Soc Neurosci 32:2027–2036
Gruber AJ, Hussain RJ, O’Donnell P (2009) The nucleus accumbens: a switchboard for goal-directed behaviors. PLoS ONE 4:e5062
Gruber AJ et al (2010) More is less: a disinhibited prefrontal cortex impairs cognitive flexibility. J Neurosci 30:17102–17110
Haber SN (2003) The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat 26:317–330
Haber SN, Knutson B (2010) The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacol: Official Publ Am Coll Neuropsychopharmacol 35:4–26
Haber SN, Fudge JL, McFarland NR (2000) Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20:2369–2382
Haney RZ, Calu DJ, Takahashi YK, Hughes BW, Schoenbaum G (2010) Inactivation of the central but not the basolateral nucleus of the amygdala disrupts learning in response to overexpectation of reward. J Neurosci 30:2911–2917
Hatfield T, Han JS, Conley M, Gallagher M, Holland P (1996) Neurotoxic lesions of basolateral, but not central, amygdala interfere with Pavlovian second-order conditioning and reinforcer devaluation effects. J Neurosci 16:5256–5265
Hayden BY, Heilbronner SR, Pearson JM, Platt ML (2011) Surprise signals in anterior cingulate cortex: neuronal encoding of unsigned reward prediction errors driving adjustment in behavior. J Neurosci 31:4178–4187
Hernandez A, Burton AC, O’Donnell P, Schoenbaum G, Roesch MR (2015) Altered basolateral amygdala encoding in an animal model of schizophrenia. J Neurosci 35:6394–6400
Hikosaka O (2010) The habenula: from stress evasion to value-based decision-making. Nat Rev Neurosci 11:503–513
Hillman KL, Bilkey DK (2010) Neurons in the rat anterior cingulate cortex dynamically encode cost-benefit in a spatial decision-making task. J Neurosci 30:7705–7713
Hollerman JR, Tremblay L, Schultz W (1998) Influence of reward expectation on behavior-related neuronal activity in primate striatum. J Neurophysiol 80:947–963
Holroyd CB, Coles MG (2002) The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev 109:679–709
Hong S, Hikosaka O (2008) The globus pallidus sends reward-related signals to the lateral habenula. Neuron 60:720–729
Hong S, Jhou TC, Smith M, Saleem KS, Hikosaka O (2011) Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. J Neurosci 31:11457–11471
Hosokawa T, Kato K, Inoue M, Mikami A (2007) Neurons in the macaque orbitofrontal cortex code relative preference of both rewarding and aversive outcomes. Neurosci Res 57:434–445
Houk J, Adams JL, Barto AG (ed) (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement
Ikemoto S (2007) Dopamine reward circuitry: two projection systems from the ventral midbrain to the nucleus accumbens-olfactory tubercle complex. Brain Res Rev 56:27–78
Ikemoto S, Panksepp J (1999) The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Brain Res Rev 31:6–41
Ito M, Doya K (2009) Validation of decision-making models and analysis of decision variables in the rat basal ganglia. J Neurosci 29:9861–9874
Ito S, Stuphorn V, Brown JW, Schall JD (2003) Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302:120–122
Izquierdo AD, Suda RK, Murray EA (2004) Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J Neurosci 24:7540–7548
Janak PH, Chen MT, Caulder T (2004) Dynamics of neural coding in the accumbens during extinction and reinstatement of rewarded behavior. Behav Brain Res 154:125–135
Jhou TC, Fields HL, Baxter MG, Saper CB, Holland PC (2009) The rostromedial tegmental nucleus (RMTg), a GABAergic afferent to midbrain dopamine neurons, encodes aversive stimuli and inhibits motor responses. Neuron 61:786–800
Ji H, Shepard PD (2007) Lateral habenula stimulation inhibits rat midbrain dopamine neurons through a GABA(A) receptor-mediated mechanism. J Neurosci 27:6923–6930
Jo YS, Lee J, Mizumori SJ (2013) Effects of prefrontal cortical inactivation on neural activity in the ventral tegmental area. J Neurosci 33:8159–8171
Joel D, Niv Y, Ruppin E (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks: Official J Int Neural Network Soc 15:535–547
Johansen JP, Tarpley JW, LeDoux JE, Blair HT (2010) Neural substrates for expectation-modulated fear learning in the amygdala and periaqueductal gray. Nat Neurosci 13:979–986
Jones B, Mishkin M (1972) Limbic lesions and the problem of stimulus-reinforcement associations. Exp Neurol 36:362–377
Kalenscher T, Lansink CS, Lankelma JV, Pennartz CM (2010) Reward-associated gamma oscillations in ventral striatum are regionally differentiated and modulate local firing activity. J Neurophysiol 103:1658–1672
Kaping D, Vinck M, Hutchison RM, Everling S, Womelsdorf T (2011) Specific contributions of ventromedial, anterior cingulate, and lateral prefrontal cortex for attentional selection and stimulus valuation. PLoS Biol 9:e1001224
Karachi C et al (2002) Three-dimensional cartography of functional territories in the human striatopallidal complex by using calbindin immunoreactivity. J Comp Neurol 450:122–134
Kaufling J, Veinante P, Pawlowski SA, Freund-Mercier MJ, Barrot M (2009) Afferents to the GABAergic tail of the ventral tegmental area in the rat. J Comp Neurol 513:597–621
Kennerley SW, Wallis JD (2009) Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables. Eur J Neurosci 29:2061–2073
Kennerley SW, Walton ME, Behrens TE, Buckley MJ, Rushworth MF (2006) Optimal decision making and the anterior cingulate cortex. Nat Neurosci 9:940–947
Kennerley SW, Dahmubed AF, Lara AH, Wallis JD (2009) Neurons in the frontal lobe encode the value of multiple decision variables. J Cogn Neurosci 21:1162–1178
Kennerley SW, Behrens TE, Wallis JD (2011) Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat Neurosci 14:1581–1589
Kesner RP, Williams JM (1995) Memory for magnitude of reinforcement: dissociation between amygdala and hippocampus. Neurobiol Learn Mem 64:237–244
Kim S, Hwang J, Lee D (2008) Prefrontal coding of temporally discounted values during intertemporal choice. Neuron 59:161–172
Kim H, Sul JH, Huh N, Lee D, Jung MW (2009) Role of striatum in updating values of chosen actions. J Neurosci: Official J Soc Neurosci 29:14701–14712
Klavir O, Genud-Gabai R, Paz R (2013) Functional connectivity between amygdala and cingulate cortex for adaptive aversive learning. Neuron 80:1290–1300
Koch M, Schmid A, Schnitzler HU (2000) Role of muscles accumbens dopamine D1 and D2 receptors in instrumental and Pavlovian paradigms of conditioned reward. Psychopharmacology 152:67–73
Lammel S, Ion DI, Roeper J, Malenka RC (2011) Projection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli. Neuron 70:855–862
Lang PJ, Davis M (2006) Emotion, motivation, and the brain: reflex foundations in animal and human research. Prog Brain Res 156:3–29
Lansink CS, Goltstein PM, Lankelma JV, Pennartz CM (2010) Fast-spiking interneurons of the rat ventral striatum: temporal coordination of activity with principal cells and responsiveness to reward. Eur J Neurosci 32:494–508
Lau B, Glimcher PW (2007) Action and outcome encoding in the primate caudate nucleus. J Neurosci 27:14502–14514
Lau B, Glimcher PW (2008) Value representations in the primate striatum during matching behavior. Neuron 58:451–463
Leathers ML, Olson CR (2012) In monkeys making value-based decisions, LIP neurons encode cue salience and not action value. Science 338:132–135
Leathers ML, Olson CR (2013) Response to comment on “in monkeys making value-based decisions, LIP neurons encode cue salience and not action value”. Science 340:430
LeDoux JE (2000) The amygdala and emotion: a view through fear. In: Aggleton JP (ed) The amygdala: a functional analysis. Oxford University Press, New York, pp 289–310
Lex B, Hauber W (2010) The role of nucleus accumbens dopamine in outcome encoding in instrumental and Pavlovian conditioning. Neurobiol Learn Mem 93:283–290
Lichtenberg NT, Kashtelyan V, Burton AC, Bissonette GB, Roesch MR (2013) Nucleus accumbens core lesions enhance two-way active avoidance. Neurosci
Louie K, Glimcher PW (2010) Separating value from choice: delay discounting activity in the lateral intraparietal area. J Neurosci 30:5498–5507
Magno E, Foxe JJ, Molholm S, Robertson IH, Garavan H (2006) The anterior cingulate and error avoidance. J Neurosci 26:4769–4773
Malkova L, Gaffan D, Murray EA (1997) Excitotoxic lesions of the amygdala fail to produce impairment in visual learning for auditory secondary reinforcement but interfere with reinforcer devaluation effects in rhesus monkeys. J Neurosci 17:6011–6020
Matsumoto M, Hikosaka O (2007) Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447:1111–1115
Matsumoto M, Hikosaka O (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459:837–841
Matsumoto M, Matsumoto K, Abe H, Tanaka K (2007) Medial prefrontal cell activity signaling prediction errors of action values. Nat Neurosci 10:647–656
McCullough LD, Salamone JD (1992) Involvement of nucleus accumbens dopamine in the motor activity induced by periodic food presentation: a microdialysis and behavioral study. Brain Res 592:29–36
McGinty VB, Lardeux S, Taha SA, Kim JJ, Nicola SM (2013) Invigoration of reward seeking by cue and proximity encoding in the nucleus accumbens. Neuron 78:910–922
Monosov IE, Hikosaka O (2012) Regionally distinct processing of rewards and punishments by the primate ventromedial prefrontal cortex. J Neurosci 32:10318–10330
Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci: Official J Soc Neurosci 16:1936–1947
Morrison SE, Saez A, Lau B, Salzman CD (2011) Different time courses for learning-related changes in amygdala and orbitofrontal cortex. Neuron 71:1127–1140
Murray EA (2007) The amygdala, reward and emotion. Trends Cogn Sci 11:489–497
Nakamura K, Santos GS, Matsuzaki R, Nakahara H (2012) Differential reward coding in the subdivisions of the primate caudate during an oculomotor task. J Neurosci 32:15963–15982
Newsome WT, Glimcher PW, Gottlieb J, Lee D, Platt ML (2013) Comment on “in monkeys making value-based decisions, LIP neurons encode cue salience and not action value”. Science 340:430
Nicola SM, Yun IA, Wakabayashi KT, Fields HL (2004) Cue-evoked firing of nucleus accumbens neurons encodes motivational significance during a discriminative stimulus task. J Neurophysiol 91:1840–1865
Nishizawa K et al (2012) Striatal indirect pathway contributes to selection accuracy of learned motor actions. J Neurosci: Official J Soc Neurosci 32:13421–13432
Niv Y, Schoenbaum G (2008) Dialogues on prediction errors. Trends in cognitive sciences 12:265–272
Nunes EJ, Randall PA, Podurgiel S, Correa M, Salamone JD (2013) Nucleus accumbens neurotransmission and effort-related choice behavior in food motivation: effects of drugs acting on dopamine, adenosine, and muscarinic acetylcholine receptors. Neurosci Biobehav Rev 37:2015–2025
O’Doherty J et al (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454
Oleson EB, Gentry RN, Chioma VC, Cheer JF (2012) Subsecond dopamine release in the nucleus accumbens predicts conditioned punishment and its successful avoidance. J Neurosci: Official J Soc Neurosci 32:14804–14808
Oliveira FT, McDonald JJ, Goodman D (2007) Performance monitoring in the anterior cingulate is not all error related: expectancy deviation and the representation of action-outcome associations. J Cogn Neurosci 19:1994–2004
Omelchenko N, Bell R, Sesack SR (2009) Lateral habenula projections to dopamine and GABA neurons in the rat ventral tegmental area. Eur J Neurosci 30:1239–1250
Ostlund SB, Balleine BW (2007) Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental learning. J Neurosci 27:4819–4825
Padoa-Schioppa C (2007) Orbitofrontal cortex and the computation of economic value. Ann N Y Acad Sci 1121:232–253
Padoa-Schioppa C (2011) Neurobiology of economic choice: a good-based model. Annu Rev Neurosci 34:333–359
Padoa-Schioppa C, Assad JA (2006) Neurons in the orbitofrontal cortex encode economic value. Nature 441:223–226
Padoa-Schioppa C, Assad JA (2008) The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nat Neurosci 11:95–102
Padoa-Schioppa C, Cai X (2011) The orbitofrontal cortex and the computation of subjective value: consolidated concepts and new perspectives. Ann N Y Acad Sci 1239:130–137
Parkinson JA et al (2001) The role of the primate amygdala in conditioned reinforcement. J Neurosci 21:7770–7780
Paus T (2001) Primate anterior cingulate cortex: where motor control, drive and cognition interface. Nat Rev Neurosci 2:417–424
Pearce JM, Hall G (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 87:532–552
Phelps EA, Ling S, Carrasco M (2006) Emotion facilitates perception and potentiates the perceptual benefits of attention. Psychol Sci 17:292–299
Pickens CL et al (2003) Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task. J Neurosci 23:11078–11084
Platt ML, Glimcher PW (1999) Neural correlates of decision variables in parietal cortex. Nature 400:233–238
Quilodran R, Rothe M, Procyk E (2008) Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 57:314–325
Rangel A, Hare T (2010) Neural computations associated with goal-directed choice. Curr Opin Neurobiol 20:262–270
Redish AD (2004) Addiction as a computational process gone awry. Science 306:1944–1947
Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokasy WF (eds) Classical conditioning II: current research and theory. Appleton-Century-Crofts, New York, pp 64–99
Rich EL, Wallis JD (2014) Medial-lateral organization of the orbitofrontal cortex. J Cogn Neurosci 26:1347–1362
Roesch MR, Bryden DW (2011) Impact of size and delay on neural activity in the rat limbic corticostriatal system. Frontiers Neurosci 5:130
Roesch MR, Olson CR (2004) Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304:307–310
Roesch MR, Olson CR (2005a) Neuronal activity in primate orbitofrontal cortex reflects the value of time. J Neurophysiol 94:2457–2471
Roesch MR, Olson CR (2005b) Neuronal activity dependent on anticipated and elapsed delay in macaque prefrontal cortex, frontal and supplementary eye fields, and premotor cortex. J Neurophysiol 94:1469–1497
Roesch MR, Calu DJ, Burke KA, Schoenbaum G (2007a) Should I stay or should I go? Transformation of time-discounted rewards in orbitofrontal cortex and associated brain circuits. Ann N Y Acad Sci 1104:21–34
Roesch MR, Takahashi Y, Gugsa N, Bissonette GB, Schoenbaum G (2007b) Previous cocaine exposure makes rats hypersensitive to both delay and reward magnitude. J Neurosci 27:245–250
Roesch MR, Singh T, Brown PL, Mullins SE, Schoenbaum G (2009) Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J Neurosci 29:13365–13376
Roesch MR, Calu DJ, Esber GR, Schoenbaum G (2010a) Neural correlates of variations in event processing during learning in basolateral amygdala. J Neurosci: Official J Soc Neurosci 30:2464–2471
Roesch MR, Calu DJ, Esber GR, Schoenbaum G (2010b) All that glitters … dissociating attention and outcome expectancy from prediction errors signals. J Neurophysiol 104:587–595
Roesch MR et al (2012a) Normal aging alters learning and attention-related teaching signals in basolateral amygdala. J Neurosci 32:13137–13144
Roesch MR, Bryden DW, Cerri DH, Haney ZR, Schoenbaum G (2012b) Willingness to wait and altered encoding of time-discounted reward in the orbitofrontal cortex with normal aging. J Neurosci 32:5525–5533
Rolls ET, Sienkiewicz ZJ, Yaxley S (1989) Hunger modulates the responses to gustatory stimuli of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey. EurJ Neurosci 1:53–60
Rothe M, Quilodran R, Sallet J, Procyk E (2011) Coordination of high gamma activity in anterior cingulate and lateral prefrontal cortical areas during adaptation. J Neurosci: Official J Soc Neurosci 31:11110–11117
Rudebeck PH, Murray EA (2008) Amygdala and orbitofrontal cortex lesions differentially influence choices during object reversal learning. J Neurosci: Official J Soc Neurosci 28:8338–8343
Rudebeck PH, Murray EA (2011) Balkanizing the primate orbitofrontal cortex: distinct subregions for comparing and contrasting values. Ann N Y Acad Sci 1239:1–13
Rudebeck PH, Murray EA (2014) The orbitofrontal oracle: cortical mechanisms for the prediction and evaluation of specific behavioral outcomes. Neuron 84:1143–1156
Rudebeck PH, Bannerman DM, Rushworth MF (2008) The contribution of distinct subregions of the ventromedial frontal cortex to emotion, social behavior, and decision making. Cogn Affect Behav Neurosci 8:485–497
Rudebeck PH, Saunders RC, Prescott AT, Chau LS, Murray EA (2013a) Prefrontal mechanisms of behavioral flexibility, emotion regulation and value updating. Nat Neurosci 16:1140–1145
Rudebeck PH, Mitz AR, Chacko RV, Murray EA (2013b) Effects of amygdala lesions on reward-value coding in orbital and medial prefrontal cortex. Neuron 80:1519–1531
Rushworth MF, Behrens TE (2008) Choice, uncertainty and value in prefrontal and cingulate cortex. Nat Neurosci 11:389–397
Rushworth MF, Walton ME, Kennerley SW, Bannerman DM (2004) Action sets and decisions in the medial frontal cortex. Trends Cogn Sci 8:410–417
Rushworth MF, Behrens TE, Rudebeck PH, Walton ME (2007) Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour. Trends Cogn Sci 11:168–176
Saddoris MP, Gallagher M, Schoenbaum G (2005) Rapid associative encoding in basolateral amygdala depends on connections with orbitofrontal cortex. Neuron 46:321–331
Salamone JD (1986) Different effects of haloperidol and extinction on instrumental behaviours. Psychopharmacology 88:18–23
Salamone JD (1994) The involvement of nucleus accumbens dopamine in appetitive and aversive motivation. Behav Brain Res 61:117–133
Salamone JD, Correa M (2002) Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav Brain Res 137:3–25
Salamone JD, Correa M (2012) The mysterious motivational functions of mesolimbic dopamine. Neuron 76:470–485
Salamone JD et al (1991) Haloperidol and nucleus accumbens dopamine depletion suppress lever pressing for food but increase free food consumption in a novel food choice procedure. Psychopharmacology 104:515–521
Salamone JD, Correa M, Nunes EJ, Randall PA, Pardo M (2012) The behavioral pharmacology of effort-related choice behavior: dopamine, adenosine and beyond. J Exp Anal Behav 97:125–146
Sallet J et al (2007) Expectations, gains, and losses in the anterior cingulate cortex. Cogn Affect Behav Neurosci 7:327–336
Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310:1337–1340
Scheffers MK, Coles MG (2000) Performance monitoring in a confusing world: error-related brain activity, judgments of response accuracy, and types of errors. J Exp Psychol Hum Percept Perform 26:141–151
Schoenbaum G (2004) Affect, action, and ambiguity and the amygdala-orbitofrontal circuit. Focus on “combined unilateral lesions of the amygdala and orbital prefrontal cortex impair affective processing in rhesus monkeys”. J Neurophysiol 91:1938–1939
Schoenbaum G, Eichenbaum H (1995) Information coding in the rodent prefrontal cortex. I. Single-neuron activity in orbitofrontal cortex compared with that in pyriform cortex. J Neurophysiol 74:733–750
Schoenbaum G, Esber GR (2010) How do you (estimate you will) like them apples? Integration as a defining trait of orbitofrontal function. Curr Opin Neurobiol 20:205–211
Schoenbaum G, Roesch M (2005) Orbitofrontal cortex, associative learning, and expectancies. Neuron 47:633–636
Schoenbaum G, Chiba AA, Gallagher M (1998) Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat Neurosci 1:155–159
Schoenbaum G, Chiba AA, Gallagher M (1999) Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J Neurosci 19:1876–1884
Schoenbaum G, Chiba AA, Gallagher M (2000) Changes in functional connectivity in orbitofrontal cortex and basolateral amygdala during learning and reversal training. J Neurosci 20:5179–5189
Schoenbaum G, Setlow B, Nugent SL, Saddoris MP, Gallagher M (2003a) Lesions of orbitofrontal cortex and basolateral amygdala complex disrupt acquisition of odor-guided discriminations and reversals. Learn Mem 10:129–140
Schoenbaum G, Setlow B, Saddoris MP, Gallagher M (2003b) Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala. Neuron 39:855–867
Schoenbaum G, Roesch MR, Stalnaker TA (2006) Orbitofrontal cortex, decision-making and drug addiction. Trends Neurosci 29:116–124
Schoenbaum G, Gottfried JA, Murray EA, Ramus SJ (2007) Linking affect to action: critical contributions of the orbitofrontal cortex. Preface. Ann N Y Acad Sci 1121:xi–xiii
Schoenbaum G, Roesch MR, Stalnaker TA, Takahashi YK (2009) A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat Rev Neurosci 10:885–892
Schoenbaum G, Takahashi Y, Liu TL, McDannald MA (2011a) Does the orbitofrontal cortex signal value? Ann N Y Acad Sci 1239:87–99
Schoenbaum G, Roesch MR, Stalnaker TA, Takahashi YK (2011b) Orbitofrontal cortex and outcome expectancies: optimizing behavior and sensory perception. In: Gottfried JA (ed) Neurobiology of sensation and reward. Boca Raton, Florida
Schultz W (1997) Dopamine neurons and their role in reward mechanisms. Curr Opin Neurobiol 7:191–197
Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1–27
Schultz W (1999) The reward signal of midbrain dopamine neurons. News Physiol Sci: Int J Physiol Produced Jointly Int Union Physiol Sci Am Physiol Soc 14:249–255
Schultz W, Apicella P, Scarnati E, Ljungberg T (1992) Neuronal activity in monkey ventral striatum related to the expectation of reward. J Neurosci 12:4595–4610
Setlow B, Schoenbaum G, Gallagher M (2003) Neural encoding in ventral striatum during olfactory discrimination learning. Neuron 38:625–636
Shabel SJ, Janak PH (2009) Substantial similarity in amygdala neuronal activity during conditioned appetitive and aversive emotional arousal. Proc Natl Acad Sci USA 106:15031–15036
Shidara M, Richmond BJ (2004) Differential encoding of information about progress through multi-trial reward schedules by three groups of ventral striatal neurons. Neurosci Res 49:307–314
Singh T, McDannald MA, Haney RZ, Cerri DH, Schoenbaum G (2010) Nucleus accumbens core and shell are necessary for reinforcer devaluation effects on pavlovian conditioned responding. Front Integr Neurosci 4:126
Solomon RL, Corbit JD (1974) An opponent-process theory of motivation. I. Temporal dynamics of affect. Psychol Rev 81:119–145
Sripanidkulchai K, Sripanidkulchai B, Wyss JM (1984) The cortical projection of the basolateral amygdaloid nucleus in the rat: a retrograde fluorescent dye study. J Comp Neurol 229:419–431
Stalnaker TA, Franz TM, Singh T, Schoenbaum G (2007) Basolateral amygdala lesions abolish orbitofrontal-dependent reversal impairments. Neuron 54:51–58
Stalnaker TA, Takahashi Y, Roesch MR, Schoenbaum G (2009) Neural substrates of cognitive inflexibility after chronic cocaine exposure. Neuropharmacology 56(Suppl 1):63–72
Stalnaker TA, Calhoon GG, Ogawa M, Roesch MR, Schoenbaum G (2010) Neural correlates of stimulus-response and response-outcome associations in dorsolateral versus dorsomedial striatum. Frontiers Integr Neurosci 4:12
Steinberg EE et al (2013) A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16:966–973
Stopper CM, Floresco SB (2011) Contributions of the nucleus accumbens and its subregions to different aspects of risk-based decision making. Cogn Affect Behav Neurosci 11:97–112
Sugrue LP, Corrado GS, Newsome WT (2004) Matching behavior and the representation of value in the parietal cortex. Science 304:1782–1787
Sugrue LP, Corrado GS, Newsome WT (2005) Choosing the greater of two goods: neural currencies for valuation and decision making. Nat Rev Neurosci 6:363–375
Sutton RA, Barto AG (ed) (1998) Reinforcement learning: an introduction
Takahashi Y, Roesch MR, Stalnaker TA, Schoenbaum G (2007) Cocaine exposure shifts the balance of associative encoding from ventral to dorsolateral striatum. Frontiers Integr Neurosci 1:11
Takahashi Y, Schoenbaum G, Niv Y (2008) Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Front Neurosci 2:86–99
Takahashi YK et al (2009) The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62:269–280
Takahashi YK et al (2011) Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 14:1590–1597
Totah NK, Kim YB, Homayoun H, Moghaddam B (2009) Anterior cingulate neurons represent errors and preparatory attention within the same behavioral sequence. J Neurosci 29:6418–6426
Tsujimoto S, Genovesio A, Wise SP (2011) Comparison of strategy signals in the dorsolateral and orbital prefrontal cortex. J Neurosci 31:4583–4592
Tsujimoto S, Genovesio A, Wise SP (2012) Neuronal activity during a cued strategy task: comparison of dorsolateral, orbital, and polar prefrontal cortex. J Neurosci 32:11017–11031
Tye KM, Janak PH (2007) Amygdala neurons differentially encode motivation and reinforcement. J Neurosci: Official J Soc Neurosci 27:3937–3945
Tye KM, Cone JJ, Schairer WW, Janak PH (2010) Amygdala neural encoding of the absence of reward during extinction. J Neurosci: Official J Soc Neurosci 30:116–125
van der Meer MA, Redish AD (2009) Covert expectation-of-reward in rat ventral striatum at decision points. Front Integr Neurosci 3:1
van der Meer MA, Redish AD (2011) Ventral striatum: a critical look at models of learning and evaluation. Curr Opin Neurobiol 21:387–392
van der Meer MA, Johnson A, Schmitzer-Torbert NC, Redish AD (2010) Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 67:25–32
Wadenberg ML, Ericson E, Magnusson O, Ahlenius S (1990) Suppression of conditioned avoidance behavior by the local application of (-)sulpiride into the ventral, but not the dorsal, striatum of the rat. Biol Psychiatry 28:297–307
Wakabayashi KT, Fields HL, Nicola SM (2004) Dissociation of the role of nucleus accumbens dopamine in responding to reward-predictive cues and waiting for reward. Behav Brain Res 154:19–30
Wallis JD (2007) Orbitofrontal cortex and its contribution to decision-making. Annu Rev Neurosci 30:31–56
Wallis JD, Kennerley SW (2010) Heterogeneous reward signals in prefrontal cortex. Curr Opin Neurobiol 20:191–198
Wallis JD, Kennerley SW (2011) Contrasting reward signals in the orbitofrontal cortex and anterior cingulate cortex. Ann N Y Acad Sci 1239:33–42
Wallis JD, Miller EK (2003) Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur J Neurosci 18:2069–2081
Wallis JD, Anderson KC, Miller EK (2001) Single neurons in prefrontal cortex encode abstract rules. Nature 411:953–956
Walton ME, Devlin JT, Rushworth MF (2004) Interactions between decision making and performance monitoring within prefrontal cortex. Nat Neurosci 7:1259–1265
Wang AY, Miura K, Uchida N (2013) The dorsomedial striatum encodes net expected return, critical for energizing performance vigor. Nat Neurosci 16:639–647
Wilson RC, Takahashi YK, Schoenbaum G, Niv Y (2014) Orbitofrontal cortex as a cognitive map of task space. Neuron 81:267–279
Winstanley CA, Theobald DEH, Cardinal RN, Robbins TW (2004) Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J Neurosci 24:4718–4722
Yin HH, Knowlton BJ (2006) The role of the basal ganglia in habit formation. Nat Rev Neurosci 7:464–476
Yun IA, Wakabayashi KT, Fields HL, Nicola SM (2004) The ventral tegmental area is required for the behavioral and nucleus accumbens neuronal firing responses to incentive cues. J Neurosci 24:2923–2933
Acknowledgements
This work was supported by grants from the NIDA (R01DA031695, MR).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Bissonette, G.B., Roesch, M.R. (2015). Neurophysiology of Reward-Guided Behavior: Correlates Related to Predictions, Value, Motivation, Errors, Attention, and Action. In: Simpson, E., Balsam, P. (eds) Behavioral Neuroscience of Motivation. Current Topics in Behavioral Neurosciences, vol 27. Springer, Cham. https://doi.org/10.1007/7854_2015_382
Download citation
DOI: https://doi.org/10.1007/7854_2015_382
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-26933-7
Online ISBN: 978-3-319-26935-1
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)