Keywords

1 Introduction

Close to the finish line of the 2013 Boston marathon, there were two explosions that killed three people and injured over two hundred others. These explosions caused a chaotic burst of activity in the group of people gathered in the area for the event. Immediately upon detonation of the bombs, people in the crowd oriented toward the explosions and froze, and then, they began to flee the site. This sequence of reaction and action is repeatedly seen in emotionally evocative experiences in human and non-human animals (LeDoux 1996a; LeDoux and Gorman 2001). While reactions are inflexible responses automatically elicited by a stimulus, actions are responses or behaviors that are produced by an organism. Reactive behaviors are genetically embedded into an organism’s response repertoire as prepared consequences to significant environmental pressures (e.g., predators; Hirsch and Bolles 1980; Coss and Biardi 1997). On the other hand, actions are behaviors learned by their reinforcing consequences (for example, a child learning to cry in order to receive attention) performed in order to obtain some goal or reward.

Using animal models, research on the neural basis of emotion has uncovered a great deal on how threatening conditioned stimuli come to control defensive reactions (LeDoux 2014, 2015). Progress on the neural foundations of aversively motivated actions has been slower, though recent times have seen a resurgence of interest in deciphering the circuits and mechanisms underlying these behaviors. In this chapter, we will describe the relationship between reactions and actions insofar as they pertain to defensive motivation, and the neural substrates of these phenomena.

2 Taxonomy of Behavior

Among the forms of behaviors exhibited by humans, many can be seen as fitting into four categories: reflexes, reactions, actions and habits (see Balleine and Dickinson 1998; Cardinal et al. 2002; Lang and Davis 2006; Yin and Knowlton 2006).

Reflexes are genetically prepared, stimulus-evoked behaviors that involve simple mechanisms and limited muscle groups. For example, myotatic reflexes (e.g., monosynaptic stretch reflex) do not require control outside of a single spinal level. Examples of defensive reflexes include withdrawal reflexes seen when painful stimuli are applied to the skin, or eyelid closure when an air puff is delivered to the region (i.e., NMR preparation).

Reaction and reflex behaviors share the attribute that the response itself is unlearned. Both of these types of responses are innate and are typically elicited by stimuli that have, through evolutionary processes, come to be embedded in the genetic wiring of the nervous system. However, reactions are distinct from reflexes in that they involve the whole organism and require control from components of the central nervous system (CNS) outside of the processing level of the sensory stimulus. For example, freezing behavior, in which the organism crouches in a certain posture and non-respiratory-related movement ceases, occurs as an initial reaction to a threatening event in many organisms. This reaction requires sustained contraction of the skeletal musculature throughout the body to maintain the characteristic posture. Metabolic support for this energy-demanding activity is provided by changes in visceral organ physiology controlled by the autonomic nervous system under the direction of the CNS. Freezing reactions can be thought of as an example of a fixed action pattern (Lorenz and Tinbergen 1938; Tinbergen 1951), or more specifically, a species-typical defensive behavior (Blanchard and Blanchard 1972; Bolles 1970; Bolles and Fanselow 1980).

The stimuli that elicit reaction and reflex behaviors, called releasers by ethologists, are specific to the response they command: freezing, for example, is an innate reaction to predators (Lorenz and Tinbergen 1938; Tinbergen 1951; Thorpe 1963; Hinde 1966; Bolles 1970; Blanchard and Blanchard 1969b, 1972; Rosen 2004). For example, Hirsch and Bolles (1980) found that two strains of deer mice that were removed from their natural environment for two breeding generations still demonstrated responses specific to the particular predator present in that environment when encountered. These prepared and unlearned responses proved useful—subjects that were presented with their ancestral predator tended to fare better than those exposed to the historically unfamiliar one. Even more dramatic, Coss and Biardi (1997) reported that a group of squirrels living in the higher elevations of a mountain and separated from snakes at lower elevations by an earthquake millions of years ago still exhibited species-typical defense responses to snakes. While such responses require no prior personal experience with an innate releaser, they can come under the control of previously neutral stimuli paired with these events through Pavlovian associative learning processes (Blanchard and Blanchard 1969a, b, 1972; Bolles 1972; Fanselow 1980). When these responses are produced by acquired stimuli, they are referred to as conditioned or learned defensive responses. This is somewhat of a misnomer since the response itself is not learned or conditioned. Instead, the response comes to be conditional on the presence of the stimulus. Some therefore prefer then terminology conditional response over conditioned response (Pavlov 1927).

Actions are also complex behaviors, but they are not simply elicited by a stimulus. Rather, they are emitted in the combined presence of certain stimuli and internal factors such as motivation and arousal and performed in order to obtain a goal or reward (Skinner 1938; Estes and Skinner 1941; Estes 1948; Rescorla and LoLordo 1965; Rescorla 1968; Lovibond 1983; Balleine and Dickinson 1998; Holland and Gallagher 2003; Niv et al. 2006). These responses are understood associatively as response-outcome (R-O) learning and include hierarchical control by the surrounding stimuli. A further distinction of actions from reactions is that actions are flexible behaviors rather than fixed responses. Depending on the circumstances, the appropriate action one must take in order to reach a goal can be quite different (Fanselow and Lester 1988). Therefore, a certain degree of preparedness may allow for some responses to be acquired or performed more readily than others (see Cain and LeDoux 2007). Nevertheless, the response can take multiple physical forms. For example, in the description of the aftermath of the Boston bombings above, the general goal would be to escape the danger; thus, the appropriate response would be to run away from the area. However, if the explosion had taken place on a boat and survivors were in the water, the appropriate action would be to swim away. Furthermore, if one had friends or loved ones with them, the goal would be to find them before fleeing, regardless of the specific behavioral demands. Much of the research on action learning, especially in terms of brain mechanisms, has focused on appetitive motivation (i.e., food, sex, or drugs); the neural substrates for instrumental motivation based on aversive outcomes (e.g., shock omission, punishment) has received less attention. A major aim of this chapter is to summarize recent research on aversively motivated actions.

When an action has been performed so frequently that it has become automatic or inflexible, it is referred to as a habit (Thorndike 1898; Skinner 1938; Hull 1943; Killcross and Coutureau 2003). Initially, actions are guided by the outcome they obtain, which can be demonstrated using outcome revaluation manipulations (including devaluation and inflation). If a food reward is made less valuable through pairings with emetics or simply by free access, relevant instrumental actions are attenuated. However, if these manipulations are conducted after extended training, no changes in behavior are seen and subjects still respond despite the outcome value having been reduced. Thus, following extended training, the action becomes an inflexible habit and is not sensitive to outcome devaluation. At this point, the behavior is guided by the stimuli present around the response instead of the outcome and is referred to as a stimulus-response (S-R) habit. This makes habits similar to reactions in that they are elicited by trigger stimuli, but they are distinct in that the response is innate in the case of reactions and learned to varying degrees (e.g., shuttle responses vs. lever press) in the case of habits. The underlying associative nature of the learned action changes over time from the hierarchical stimulus[response-outcome] (S[R-O]) representation to an S-R habit. Behavioral automation can be beneficial as it frees cognitive resources to attend to other demands. For instance, perfecting a skill such as bike-riding can become second nature requiring less attention. Furthermore, turning off the light when one leaves a room can save money. However, habits have a well-known downside too, with consequences much worse than simply leaving a roommate in a dark room. Habits achieve pathological status when they become maladaptive (e.g., compulsive hand-washing or compulsive taking of addictive drugs), and are very resistant to extinction when contingencies change.

3 Historical Context of Studies on Aversive Learning

The background above defines reactions and actions among various classes of behavior. Below, we focus on reactions and actions. However, prior to discussing the current state of research on these behaviors, we will briefly review the history of this field.

In aversive conditioning a neutral conditioned stimulus (CS), such as a tone, is repeatedly paired with an aversive unconditioned stimulus (US), usually an electric shock. Following this episode, the CS alone elicits conditioned freezing responses (CRs; Blanchard and Blanchard 1969a; Bolles and Fanselow 1980). This learning process was originally studied by Pavlov (1927) who referred to it as ‘defensive conditioning.’ And while there were some notorious studies of the basic phenomenon in humans such as John Watson’s famous studies involving little Albert (Watson 1929), early psychologists were more interested in Thorndike’s (1898) ‘law of effect’ and instrumental conditioning. As a result, in the first half of the twentieth century, psychologists interested in aversive motivation were more likely to use aversive instrumental tasks, such as avoidance conditioning (Miller 1948, 1951; Mowrer 1947; Mowrer and Lamoreaux 1946) instead of Pavlovian conditioning.

In avoidance conditioning, animals learn to perform (active avoidance) or withhold (passive avoidance) an action in order to avoid negative or harmful results (typically electric shock). Below, we focus on active avoidance, which we will refer to as avoidance conditioning throughout. Responses that are measured in avoidance studies include shuttling (moving between rooms), wheel turning, or pressing a lever. Avoidance conditioning has traditionally been understood as a two-stage process. First, stimuli become associated with the shock via Pavlovian conditioning, and then, instrumental responses are performed that terminate or escape (or avoid) those threatening stimuli, thus providing negative reinforcement of the response (Brown and Jacobs 1949; Kalish 1954; Levis 1989; McAllister and McAllister 1971; Miller 1948, 1951; Mowrer 1947; Mowrer and Lamoreaux 1946; Overmier and Lawry 1979; Solomon and Wynne 1954). Because it helps cope with a threat, the avoidance response can, therefore, be used to study aversive motivation. Typically, Pavlovian conditioning processes were not directly measured or studied outside of avoidance procedures in the heyday of avoidance.

Starting in the 1950s, avoidance tasks were used to explore the brain mechanisms of aversive behavior, typically involving imprecise tools to understand neural function. Because avoidance conditioning intermixes Pavlovian (reaction) and instrumental (action) learning, studies using these tasks (Weiskrantz 1956; Goddard 1964; Gabriel et al. 1983; Isaacson 1982) were not able to provide a clear picture of the neural mechanisms underlying either the Pavlovian or instrumental learning processes (for review, see LeDoux et al. 2009; Sarter and Markowitsch 1985).

While psychologists struggled to understand aversive motivation with avoidance studies, research on the neural basis of learning and memory turned to Pavlovian conditioning, especially in invertebrates (Alkon 1983; Carew et al. 1983; Dudai et al. 1976; Kandel and Spencer 1968; Walters et al. 1979). Success in identifying neural mechanisms with simple tasks in invertebrates leads vertebrate researchers to take a similar approach (Cohen 1974; Thompson 1976). In the 1980s and 1990s, considerable progress was made in identifying the neural systems underlying Pavlovian aversive conditioning (Davis 1986; Kapp et al. 1979; LeDoux et al. 1983, 1984, 1988; for review, see: Davis 1992; Kapp et al. 1992; LeDoux 1992). Because simpler behaviors depend on simpler mechanisms, progress was rapidly made in understanding the brain mechanisms of defensive learning, which we will review below.

4 Neural Basis of Reactions

Pavlov’s defense conditioning procedure came to be called fear conditioning. However, because this term implies that a subjective mental state of fear is being acquired, we have recently argued that this procedure should be called Pavlovian threat conditioning (PTC; LeDoux 2014, 2015). This term is preferable to defense conditioning since the latter term implies that the defensive responses are being learned, whereas PTC implies that the stimulus is acquiring a new meaning.

PTC is a widely used paradigm for studying the brain because of its simplicity, robustness, and repeatability (Fig. 1). In most studies, the CS of choice is an auditory tone and the US is a mild footshock. Learning can occur following the first CS–US pairing as is evidenced by defensive responding (freezing) during the second presentation of the CS (before the shock US is presented; Blanchard and Blanchard 1969a, 1972; Bolles and Fanselow 1980. Moreover, these associations are long-lasting, as they can be observed hours, days, weeks, and even years following conditioning (Gale et al. 2004).

Fig. 1
figure 1

Auditory Pavlovian threat conditioning: depiction of a standard Pavlovian threat conditioning (PTC) paradigm using an auditory cue as the CS (a) and typical results obtained (b). Prior to training, rats are habituated to the conditioning chamber in the absence of stimuli. The next day, animals are returned to the conditioning chamber and an unconditioned stimulus (US; a mild shock) is paired with the conditioned stimulus (CS; a tone), usually one to five times. After three hours for short-term memory (STM) or 24–48 h for long-term memory (LTM, depicted), animals are placed in a new context and exposed to multiple presentations of the CS alone. Innate defensive responses, i.e., ‘freezing’ behavior, are then scored as total time spent in this state over the course of the CS presentation. To test for non-associative effects, an ‘unpaired’ control group is often used in which CS and US do not co-occur. This unpaired control group shows very little CS-elicited freezing, indicating that associative learning processes are responsible for this defensive behavior

Full size image

In addition to behavioral responses to the CS, physiological changes such as autonomic and endocrine responses also occur (Kapp et al. 1979; LeDoux et al. 1983; Schneiderman et al. 1974), indicating that the CS, like the US, can elicit an integrated organismic response to perceived danger.

Research over the past several decades has clearly identified the amygdala as a key site where CS–US associations are formed as well as a necessary site for the later expression of defensive reactions elicited by the CS (Davis et al. 1997; Fanselow and LeDoux 1999; LeDoux 1996b, 2000; Maren 2001; Maren and Fanselow 1996; Fig. 2). In particular, the neural circuitry of auditory cued PTC has been well established.

The amygdala may be separated into several regions, nuclei and subnuclei based on their emergence in evolution and neuroanatomical or cytoarchitectural features (Fig. 2). According to one scheme, the amygdala can be separated into an evolutionarily primitive division extending from subcortical areas (cortical, medial, and central nuclei) and a newer division associated with the neocortex (lateral, basal, and accessory basal nuclei) (Johnston 1923). The basal nucleus can be separated into basolateral and basomedial nuclei and the central nucleus into lateral, capsular, and medial subdivisions (Pitkänen 2000; Pitkänen et al. 2000). Considering the heterogeneity of these subnuclei, in terms of neuroanatomy, functional circuitry, and molecular expression, it is important to study these various subnuclei in isolation. Indeed, classic brain manipulations, including lesion studies and intracranial drug infusions, did not always account for the functional differences in these subareas (as highlighted in Romanski et al. 1993; Amano et al. 2011). Recent advances in molecular genetics, such as optogenetics, are allowing researchers to identify, with spatial and temporal precision, the distinct functions for subnuclear partitions within the amygdala (Ehrlich et al. 2009; Johansen et al. 2012). Furthermore, advances in transgenic technology are allowing for manipulations of distinct cell subtypes (as defined by their molecular expression profile) within these subnuclei not only in mice, but also in rats and non-human primates (Gafford and Ressler 2015).

Fig. 2
figure 2

Neural circuits underlying Pavlovian threat conditioning: a Anatomy of the amygdala: the rat amygdala may be separated into at least 12 distinct nuclei as identified using (a) Nissl cell body stain, b Acetylcholinesterase stain, and a c silver fiber stain. Abbreviations: Amygdala areas: AB accessory basal; B basal nucleus; Ce central nucleus; CO cortical nucleus; ic intercalated cells; La lateral nucleus; M medial nucleus. Non-amygdala areas: AST amygdalo-striatal transition area: CPu caudate putamen; CTX cortex. d Amygdala circuitry involved in the acquisition, consolidation, and expression of threat memories. The auditory CS (tone) and US (mild shock) converge in the lateral amygdala (LA) where synaptic plasticity and memory formation are thought to occur. The LA communicates with CeA directly and indirectly by way of connections in the basal, accessory basal, and intercalated cell masses (ICM). The CeA (CeM) sends projections to hypothalamic and brainstem areas to elicit the conditioned defensive response (freezing) as well as autonomic and hormonal responses. CeL lateral nucleus of Ce

Full size image

Decades of research indicate that these amygdala structures work together to coordinate defensive responses to threatening stimuli. Importantly, learning and expression of threat memories require the lateral nucleus of the amygdala (LA), the major input nucleus of the amygdala. Without the LA, and its efferent communication with the basal nucleus (B) and central nucleus (CeA), the CS cannot gain control of innate defensive responses.

Auditory PTC is thought to involve a Hebbian mechanism within the LA, whereby relatively weak auditory inputs on LA neurons are potentiated by concurrent strong depolarization produced by somatosensory inputs processing the US. The potentiation of auditory inputs increases the probability that LA neurons will show conditioned increases in cell firing when an auditory CS is presented. Indeed, this model is supported by significant work both in vitro (McKernan and Shinnick-Gallagher 1997; Rumpel et al. 2005; Clem and Huganir 2010; Tsvetkov et al. 2002; Schroeder and Shinnick-Gallagher 2004, 2005) and in vivo (Rogan and LeDoux 1995; Quirk et al. 1995; Grace and Rosenkranz 2002). With the development of new molecular tools for manipulating neural circuits, (i.e., optogenetics and chemogenetics), recent studies have provided more causal evidence for the importance of LA plasticity in PTC. For example, expanding on earlier studies shows that LA neurons can be selectively recruited to form memory by synthetically increasing their plasticity (i.e., infecting these neurons with the plasticity-related transcription factor CREB (cAMP Response Element Binding protein; Kida et al. 2002; Han et al. 2007); Josselyn and colleagues showed that activation of these specific neurons with DREADD (designer Receptors Exclusively Activated by Designer Drugs) increases PTC learning (Yiu et al. 2014). Moreover, it was recently shown that replacing footshock with light activation of LA principal cells expressing the excitatory opsin, channelrhodopsin-2 (ChR2), is sufficient for threat conditioning to an auditory tone (Johansen et al. 2010, 2014), and another recent study used optogenetics to ‘engineer’ a synthetic memory by inducing LTP at auditory synapses in LA paired with footshocks (Nabavi et al. 2014). Together, these data highlight the LA as a critical mediator of associative, Hebbian plasticity and memory formation.

The LA, in addition to receiving and integrating CS and US information for the formation of associative memories, also controls defense reactions via direct and indirect connections to CeA, a major output nucleus of the amygdala (Fig. 2). The indirect pathway is more prominent and involves synaptic connections to the basal nucleus (B) and the accessory basal nucleus (ABA), which make synaptic contacts on neurons in the central nucleus of the amygdala CeA. The CeA then coordinates the behavioral and autonomic responses to threat via long-range projections to the brainstem periaqueductal gray (PAG) and hypothalamus, respectively. Plasticity has also been shown to occur in CeA (Pascoe and Kapp 1985; Wilensky et al. 2006), and recent efforts have begun to dissect the microcircuitry within the CeA and the efferents that control defensive behaviors (see Ehrlich et al. 2009; Johansen et al. 2012 and Lüthi and Lüscher 2014 for a review of recent studies).

With the advent of modern molecular genetic techniques, recent work has corroborated and extended findings previously shown with more traditional techniques. The newfound ability to manipulate specific cell subtypes and circuits has allowed for detailed probing of circuitry to, within, and from the amygdala.

5 Diverse Functions of a Threat Stimulus: The Nature of Action and Reaction Learning in Aversive Motivation

The transition of the field toward the studies of purely Pavlovian conditioning led to considerable progress in understanding this fundamental phenomenon. However, studies of avoidance behavior are also valuable because they can provide additional insight into more complex forms of aversively motivated behavior. Knowledge obtained about the neural mechanisms involved in these processes could be very useful in understanding human disorders characterized by pathological avoidance. This is because in addition to eliciting anticipatory prepared responses, Pavlovian stimuli can do much more. As we have briefly pointed out earlier, Pavlovian stimuli play important roles in reinforcing instrumental responses during avoidance learning in tasks. Avoidance itself can be an adaptive way to cope with threats and exert control in dangerous situations, above and beyond the anticipatory responses produced by Pavlovian cues. Pavlovian stimuli can also function as incentives that motivate previously learned instrumental responses. Thus, Pavlovian processes can control the expression of maladaptive avoidance behaviors, and understanding this process on a neural level can help treat such behavioral problems. Below, we will review studies that have explored the neural circuits involved in supporting these different roles of aversive Pavlovian conditioned stimuli in avoidance and related behaviors. But first, we will briefly consider some conceptual issues regarding the psychological underpinnings of avoidance behavior.

A general feature of avoidance learning is negative reinforcement (Rescorla 1968) where some expected aversive event is removed contingent upon the performance of some target action (e.g., lever press or shuttling). Behaviors performed that remove these aversive events are subsequently more likely to be performed. While this would appear to reflect instrumental learning processes similar to those that guide actions when the reinforcer is a primary and appetitive one, there is some debate as to the exact nature of the underlying psychological substrates of this form of learning. An argument can be made that Pavlovian learning processes can explain avoidance behaviors without the involvement of goal-directed processes. This view is supported by observations that behaviors that closely approximate prepared defensive reactions (e.g., fleeing, rearing) are more readily acquired as avoidance responses than artificial responses (e.g., lever press; Bolles 1970). Additionally, some of these prepared behaviors (e.g., freezing) cannot be eliminated by punishment with shock, and this has been taken as evidence that they are not sensitive to control by instrumental contingencies and are, therefore, not truly instrumental behaviors (Fanselow and Lester 1988; Fanselow 1997). However, this is a complicated discussion that we will not focus on here. There has been evidence on both sides of this issue that suggest there is some basis for control of these behaviors by instrumental contingencies (Matthews et al. 1974; Killcross et al. 1997). For example, in addition to any modulation by Pavlovian associations, CS-escape and US-avoidance contingencies contribute independently to avoidance learning. Studies by Kamin (1956) as well as Herrnstein and Hineline (1966) show that learning is impaired if either contingency is discontinued. Furthermore, studies in appetitive learning suggest that similarly, some behaviors are more easily used as a means to food reinforcement (Shettleworth 1978). More germane to our current discussion, the neural control of these behaviors differs from basic Pavlovian learning processes, and differences also exist in neural control between different action-based tasks (i.e., different forms of avoidance or escape behavior). These differences will be considered below. Regardless of the exact nature of the representation developed through the learning of aversive actions, and specifically whether or not it is a true instance of instrumental learning, we believe unique processes underlie these actions and that further pursuit of their neural control is important. Through avoidance and related behaviors, organisms are able to reduce danger under a variety of circumstances, which attenuates defensive reactions and, therefore, has broad implications for understanding aversive motivation.

6 The Neural Basis of Aversively Motivated Actions

While a variety of preparations can be used to study avoidance and aversively motivated action learning, we focus on four main procedures here: signaled avoidance, unsignaled avoidance, escape from threat, and Pavlovian-to-instrumental transfer (PIT). We will discuss the underlying circuitry related to these tasks in the sections below.

6.1 Neural Control of Signaled and Unsignaled Avoidance Behavior

In a signaled avoidance procedure (SigA), a CS (e.g., tone) is first established as a threat via standard delay pairings of the CS with an aversive US (e.g., footshock). During later trials, presentation of the CS or the US can be terminated by performance of a target response. In many of our studies, subjects are trained to perform a shuttle response in a two-way chamber in order to control stimulus deliveries (e.g., see Fig. 3); we chose this seminatural response because of its similarity to prepared fleeing behaviors among those found to have been used in the available literature. If the subject performs the shuttle response during the CS, the CS is removed, and additionally, the expected US is omitted. While both events (CS-offset and US-omission) possess the power to reinforce shuttling (see Kamin 1956), the contribution of each is difficult to disentangle in a SigA procedure because trials without responses result in further US deliveries.

Despite the limitations, early studies (see Gabriel et al. 2003) found that amygdala was important for avoidance, but what exact function it served could not be determined from these efforts. Using small lesions of amygdala subnuclei placed following baseline SigA training in rats, Choi et al. (2010) found that damage restricted to lateral (LA) or basal (BA) amygdala impaired performance, but that central (CeA) amygdala damage had no impact on SigA learning. Based on decades of studies in pure Pavlovian conditioning, the role of LA in SigA can be inferred; the Pavlovian learning presumed to be required for reinforcing and motivating the action via the CS is stored in LA. Without this associative knowledge, SigA learning cannot proceed. Given this role for LA, these findings suggest that BA is important for using the LA-based CS–US association to reinforce responding downstream. This pathway out of the amygdala as it relates to SigA will be further considered below.

It should be noted that a similar circuit is implicated when avoidance responses are acquired in the absence of any auditory CS. In unsignaled Sidman active avoidance (USA; see Sidman 1953), subjects are similarly placed in the two-way chamber, but are never exposed to a tone CS. Instead, in this procedure, footshock USs are presented at a fixed interval and any shuttle response that occurs postpones the next US, thus negatively reinforcing the response. Lazaro-Munoz et al. (2010) found that LA and BA but not CeA lesions impaired acquisition of USA behavior compared to sham controls. While the tone CS is absent from USA training procedures, the role of the context in promoting the response becomes more important, as do feedback cues presented when the response is performed. Because the context cannot be removed following a response in USA as a CS can be in SigA, the presence of discreet feedback cues has been found to help reinforce and shape USA. The feedback cues are negatively correlated with shock and may become safety signals that reinforce avoidance responding by counteracting the threat (Dinsmoor 2001; Rescorla 1968). Whether signaled by a discreet CS (as in SigA) or a context (as in USA), the associative component of avoidance learning appears to depend on LA, which after decades of Pavlovian conditioning studies may not appear surprising, but given that starting point, further analysis of avoidance circuitry can now proceed more quickly than before that was common knowledge. In both SigA and USA, BA appears critical for producing actions via LA-based associative knowledge regarding threats.

Before actions can be effectively promoted, incompatible reactions normally evoked by the CS must be sufficiently opposed to allow for the expression of action. There is evidence that this regulation is accomplished in a parallel manner via infra limbic ventromedial prefrontal cortex (PFCIL) inhibition of CeA during signaled avoidance (Moscarello and LeDoux 2013; but see Bravo-Rivera et al. 2015). Preventing PFCIL from normally functioning (permanently or temporarily) impairs avoidance behav2013ior. Subjects with lesions or muscimol inactivation of IL show reduced avoidance responding and increased freezing relative to control subjects (Moscarello and LeDoux 2013). Further evidence of this antagonistic process comes from subjects that fail to acquire SigA and USA, so-called poor-performers. These subjects exhibit more freezing and less avoidance relative to animals that acquire normally. If CeA is lesioned in these poorly performing subjects, normal responding emerges compared to sham-lesioned poor-performers (Choi et al. 2010; Lazaro-Munoz et al. 2010). This suggests that excessive CeA activity interferes with acquisition and expression of avoidance responses to cause poor performance in both tasks. Thus, in normal subjects, PFCiL comes to oppose CeA-mediated defensive reactions, which allows for expression of SigA. Whether this same regulatory process applies to USA has not been systematically investigated, but the findings regarding basic circuitry and response competition via CeA-induced freezing suggest convergence. Furthermore, Martinez et al. (2013) found that good USA performers showed more c-Fos in PFCiL compared to poor performers (however, see Bravo-Rivera et al. 2014, 2015 for results implicating PFCPL).

While there has been a considerable amount of attention paid to the striatum in the context of appetitive instrumental conditioning and motivation (Cardinal et al. 2002; Setlow et al. 2002; Balleine and Killcross 2006; Shiflett and Balleine 2010), not much is known about how these structures contribute to aversive motivational processes. Cheer and colleagues (Oleson et al. 2012) have recently shown using fast-scan cyclic voltammetry that the amount of dopamine released during sub-second release events is increased in nucleus accumbens (NAcc) core following an avoidance response in a footshock SigA procedure. This result suggests that dopamine release in NAcc may play a critical role in reinforcement during SigA.

In agreement with these findings, Ramirez et al. (2015) found that expression of c-Fos in NAcc (but in the shell rather than the core) was significantly higher in subjects that were trained on SigA compared to yoked and box control subjects (see also Bravo-Rivera et al. 2014, 2015). Furthermore, this study also showed that muscimol inactivation of NAcc shell but not core impaired SigA performance. More powerful still temporary disconnection of BA and NAcc shell with muscimol impaired SigA. By inactivating BA and NAcc shell unilaterally, but on contralateral (or opposite) sides of the brain, communication between these structures, but not overall function is compromised. Because a single nucleus remains active in each hemisphere, this can mostly compensate for the inactivated counterpart. This manipulation impaired SigA compared to control subjects that received inactivation of these structures on the same side of the brain, thus leaving communication intact (Ramirez et al. 2015). Together, the Oleson et al. (2012) and Ramirez et al. (2015) findings clearly suggest a role for BA and NAcc in mediating acquisition and expression of SigA. There are many procedural and design difference between these studies; therefore, it is difficult to make easy comparisons and address discrepant findings regarding subregions of the NAcc (i.e., core vs. shell). For example, lever-press avoidance requires much more training than shuttle-based avoidance, and therefore, this could be a potential reason for the functional neuroanatomical difference. The NAcc shell may be required early on in training and the core later, when the behavior has transitioned to habit. Nevertheless, these findings provide strong evidence that a LA-BA-NAcc circuit is important for SigA behavior and that prefrontal regulation of CeA facilitates this process by reducing reactive freezing CRs. Whether this is also true of USA is currently not known.

A note on habitual avoidance behavior

As briefly discussed above, actions that are repeatedly performed or overtrained are referred to as habits. Studies in appetitive motivation using reinforcer revaluation have found that early in training, instrumental actions are guided by the associated outcome and changes in the value of this outcome directly influence responding (Adams 1982; Balleine and Dickinson 1991, 1998). After extended training, performance of the response is no longer sensitive to the outcome value and persists despite manipulations that render the outcome devalued (e.g., taste aversion or selective satiety). The transition from an S[R-O] ‘goal-directed’ action to an S-R ‘habit’ has been found to depend on how the action is coordinated by the ventromedial prefrontal cortex. Killcross and Coutureau (2003) found that while sham control subjects showed reductions in lever-press behavior following pre-feeding only when given limited training, subjects with prelimbic lesions continued to demonstrate this devaluation effect regardless of training amount. This suggests that the transition to outcome independence and habitual action involved prelimbic function coming online to guide the response in a way that is unbound from the outcome representation. While the development of habits has not been thoroughly studied in the context of aversive motivation there is some evidence that a similar transition might apply. Poremba and Gabriel (1999) have shown that the passage of time renders avoidance behavior amygdala-independent. Using a discriminative footshock avoidance procedure in rabbits, muscimol inactivation of the amygdala impaired wheel-turning responses very early in training but not after extended training or following a seven-day rest period. In agreement with this result, using shuttling in rats, Lazaro-Munoz et al. (2010) found that amygdala lesions placed after extended USA training had no effect, while these lesions in another group impaired initial acquisition. This is in contrast to overtrained Pavlovian learning, which has been shown to not undergo this transition and remain amygdala dependent (Zimmerman et al. 2007). Avoidance behavior can be an effective means of coping with threats; however, habitual avoidance is maladaptive and disruptive. Whether the transition to amygdala independent avoidance discussed above reflects outcome independence as well is not known.

6.2 Neural Control of Escape from Threat

Acquisition of SigA responses requires BA, though it is not clear whether BA is sensitive to CS-offset, US-omission, or both. From a behavioral perspective, SigA does not permit isolation of these processes. As noted above, there have been studies that demonstrate the importance of the CS-escape and US-avoidance contingencies to learning (Kamin 1956) and also those that emphasized the importance of feedback in avoidance learning (Sidman 1953; Rescorla 1968). However, because Pavlovian and instrumental learning occur together in SigA, it is unclear which neural processes might pertain to which psychological phenomenon. In order to study this issue on a psychological level in the past, tasks were designed in which Pavlovian and instrumental conditioning occur in separate phases as opposed to the intermixed nature of SigA training described above. These tasks were referred to as ‘escape from fear’ tasks. We will refer to them as ‘escape from threat’ or EFT (see LeDoux 2014). EFT demonstrates that once a CS has been established as a threat, the removal of this threat alone can support acquisition of an instrumental response. This is because no shocks are presented during instrumental training (see Fig. 4). A trial never ends with a shock US during EFT training, even when no response occurs. Thus, the Pavlovian contingency is under extinction during the instrumental session. That responding is acquired under these conditions at all suggests that threat removal is a powerful reinforcement process on its own.

EFT has not been as thoroughly studied on a neural level as have SigA or USA due to complications with the task that have made it historically controversial. For example, the reliability and parametric uniformity of EFT learning have been sources of criticism for EFT as has a lack of testing procedures to demonstrate response retention (see Cain and LeDoux 2007). What research has been done suggests that the same amygdala circuitry is involved as in SigA. For example, using a one-way shuttle task, Amorapanth et al. (2000) found that LA and BA lesions impaired EFT, while CeA lesions did not. These findings suggest that a similar circuit may control both SigA and EFT forms of avoidance learning.

6.3 Incentive Motivation: Neural Circuits of Aversive Pavlovian-to-Instrumental Transfer

In addition to conditioned reinforcement, another function served by the CS in avoidance and related phenomena is conditioned motivation, which can be studied in isolation with Pavlovian-to-instrumental transfer (PIT). In PIT, an instrumental response is performed at some baseline rate, and presentation of the CS increases the rate with which this response is performed (Bolles and Popp 1964; Rescorla and LoLordo 1965; Rescorla 1968; Weisman and Litner 1969; Overmier and Payne 1971; Overmier and Brackbill 1977; Patterson and Overmier 1981; Campese et al. 2013; Nadler et al. 2011). The neural basis of PIT has been extensively studied in cases where the US is appetitive (e.g., food, or drugs; for a review see Holmes et al. 2010; Laurent et al. 2015). While some early studies used similar procedures in aversive motivation, only recently has there been progress on identifying the neural foundations of aversive PIT. Using early studies as a guide, we have developed a simple aversive PIT task for rats using USA shuttling in a two-way chamber as the instrumental response. As described earlier, subjects learn to avoid footshocks in the USA procedure independently from any discreet Pavlovian stimuli with an excitatory relationship to shock (i.e., in the absence of the tone CS). When a previously trained CS is presented during USA, subjects show little freezing to the CS and, instead, display enhanced USA behavior. An example of this PIT effect is shown below in Fig. 5 along with figures depicting the training procedure (see Campese et al. 2013).

Fig. 3
figure 3

Neural circuits underlying signaled active avoidance (SigA): Left panel shows data collected using a two-way shuttle response. Following an inescapable CS–US pairing, shuttle responses can terminate presentations of the CS and prevent the US. On trials where no response occurs during the CS, shuttling can escape the subsequent shock. a Lateral, basal, and central amygdala lesions impair SigA, from Choi et al. (2010). b Disconnection of basal amygdala and nucleus accumbens shell with muscimol impairs SigA, from Ramirez et al. (2015). Muscimol-induced inactivation of the infralimbic region of the prefrontal cortex increases CS-evoked freezing (c) and reduces avoidance responses (d) from Moscarello and LeDoux (2013). Right panel shows results using the step platform procedure (Bravo-Rivera et al. 2014, 2015). US delivery can be avoided by stepping onto the platform into the safety zone. Prelimbic (a) and not infralimbic (b) inactivation impairs SigA, as does inactivation of the ventral striatum (c) and basolateral complex of the amygdala (d)

Full size image

We have begun to use this procedure to examine the neural circuits important for aversive PIT and have found evidence for a different pathway that processes incentive motivation than that seen for SigA, USA, and EFT acquisition. Using lesions, Campese et al. (2014) found that LA and CeA but not BA are required for aversive PIT. LA and CeA damage eliminated the facilitative effect of the CS, but BA lesions had no effect. Further studies (McCue et al. 2014) found that medial amygdala (MeA) is also required for PIT. One possibility suggested by these findings is that LA-CeA-MeA circuitry (see Fig. 6) is important for aversive PIT, departing from the LA-BA-NAcc circuit implicated in CS-based avoidance tasks discussed above. An important procedural difference between PIT and CS-based avoidance/escape (i.e., SigA) tasks is that avoidance/escape responses are learned in the repeated presence of the CS. This likely results in the CS becoming part of the associative structure during learning, where it comes to act as a discriminative or occasion-setting stimulus that signals when the instrumental (i.e., avoidance) contingency is in effect. This process engages extinction-like regulation of CeA defensive behavior (e.g., freezing) via the prefrontal cortex (Moscarello and LeDoux 2013; Bravo-Rivera et al. 2014, 2015), which allows for expression of active responding along the ‘informational’ LA-BA-NAcc pathway. This is in contrast to PIT where the CS is not present during USA training and cannot come to influence the instrumental contingency. The CS is previously trained and is not presented again until transfer tests. Thus, the increased USA behavior observed in PIT tests can be considered a reaction because the CS has no control over the instrumental contingency. Therefore, during PIT, the CS modulates behavior through ‘emotional/affective’ routes (mediated by CeA/MeA) rather than ‘informational’ routes (mediated by BA). While studies in appetitive motivation have implicated NAcc (Shiflett and Balleine 2010; Laurent et al. 2015) in PIT, the contribution of this region to aversive PIT has not been explored as of yet (Figs. 5 and 6).

Fig. 4
figure 4

Neural circuits underlying escape from threat (EFT): Experimental procedure developed by Cain and LeDoux (2007) (Upper right corner) Following standard Pavlovian conditioning, rearing can terminate CS presentations and retention of this response can be tested the following day. Over the course of EFT training, rearing frequency increases in subjects that received the R-O (i.e., CS-offset) contingency only, and yoked and unpaired control groups did not show this pattern. During EFT training, freezing decreases for both groups that received CS–US pairings during Pavlovian conditioning (c). More rearing is seen during the test session in EFT-trained subjects and (b) freezing spontaneously recovers in the yoked control group only (d). Using a one-way shuttle response, Amorapanth et al. (2000) found that EFT was impaired by lateral and basal amygdala lesions, but not central amygdala lesions

Full size image
Fig. 5
figure 5

Neural circuits underlying aversive Pavlovian-to-instrumental transfer (PIT): procedure used to study aversive PIT from Campese et al. (2013) (top). Following Pavlovian conditioning subjects undergo unsignaled Sidman active avoidance (USA) in the two-way shuttle chambers. During transfer testing, the CS is presented in this context and the effect on USA rate is measured. Subjects that received CS–US pairings during Pavlovian conditioning show an elevation of shuttle rate compared to the pre-CS period, while unpaired and naïve controls do not (a from Campese et al. 2013). Lateral, central but not basal amygdala lesions impair PIT (b from Campese et al. 2014). Medial amygdala lesions also impair PIT (c) but have no effect on CS-elicited freezing (d from McCue et al. 2014)

Full size image
Fig. 6
figure 6

Schematic representation of the circuits underlying aversively motivated instrumental reactions and actions: CS–US processing occurs in LA and ‘informational’ actions (SigA, EFT, USA) require the LA–BA pathway, while ‘emotional’ reactions (CS-elicited freezing, facilitation or suppression) engage the LA–CeA circuitry

Full size image

7 Summary

The systems described above depict a flexible and adaptive network capable of producing appropriate responses under a variety of circumstances motivated by threat or general aversive processes. These responses can be innate and reflexive, produced by evolutionarily selected stimuli or those given the power to convey threat through associative learning. The responses could alternatively be learned, artificial to some extent, but still prepared in other ways. It is clear that these dimensions of an aversively motivated response depend on unique circuitry more complex than that which accomplishes simple Pavlovian learning processes. There is evidence that microcircuits within the amygdala may contribute to conditioned motivation and conditioned reinforcement in different ways. These forms of aversively motivated learning have great relevance to clinical issues concerning coping strategies in treating anxiety, phobic disorders, and post-traumatic stress. Understanding the behavioral, neurobiological, and pharmacological basis of these phenomena can lead to profound progress in applications to these problems. While the fragility of extinction treatments are well known (e.g., spontaneous recovery, and renewal), avoidance learning produces reductions in defensive responding (i.e., active coping responses) that survive the passage of time and shifts in physical context (Bouton 2004; Cain and LeDoux 2007; LeDoux 2015). Additionally because maladaptive and inappropriate habitual avoidance behavior underlies emotional disorders in humans (e.g., agoraphobia, obsessive compulsive disorder), the importance of developing a rich understanding of these processes is twofold. Because of the great deal of knowledge obtained on PTC processes over decades of research, we are poised to learn more about these complex and elaborate circuits than was permitted in the past.