Introduction

Aggressive, submissive, and defensive behaviors under intraspecific territorial conflict or competition are collectively designated agonistic behavior (Scott and Fredericson 1951; Scott 1966; Young 2019). The resident–intruder (RI) paradigm (Krsiak 1975) is one of the most widely used tests of agonistic behavior in laboratory animals such as rodents (Miczek and O’Donnell 1978; Miczek 1979), monkeys (Miczek et al. 1981; Miczek and Yoshimura 1982), and zebrafish (Zabegalov et al. 2019). In this experimental setting, spontaneous and natural expression of agonistic behavior can be reciprocally provoked between a resident and an intruder placed into the resident’s cage. The agonistic behavior of the animals is observationally divided into behavioral elements. Some elements (e.g., upright posture) are characterized as displays that send signals to the opponent. In parallel with the magnitude and immediacy of behavioral signals from an opponent, agonistic behaviors are elicited through neurotransmission and neuroendocrine responses (Summers and Winberg 2006). Numerous neurotransmitters, including serotonin (5-hydroxytryptamine, 5-HT), dopamine (DA), glutamate, γ-aminobutyric acid, adenosine, and neuropeptides are associated with agonistic behaviors in the RI test (Takahashi et al. 2012; Veroude et al. 2016).

In neurotransmission, 5-HT and DA exert functionally opposing roles in agonistic behaviors. In humans, lower cerebrospinal fluid (CSF) concentrations of the 5-HT metabolite 5-hydroxyindolacetic acid (5-HIAA) are associated with higher levels of impulsive and violent behaviors (Brown et al. 1982; Linnoila et al. 1983; Virkkunen et al. 1995). Similar associations between aggression and lower metabolism (Mehlman et al. 1994; Higley et al. 1996) or concentration (van Erp and Miczek 2000) of 5-HT have also been reported. Moreover, mice that are deficient in both serotonergic neurons (Mosienko et al. 2012) and the biosynthesis of brain 5-HT (Hendricks et al. 2003) show exaggerated aggression in RI tests. Conversely, the administration of selective 5-HT reuptake inhibitors, which increase 5-HT levels in the brain, can attenuate aggressive behaviors in both humans and animals (Takahashi et al. 2012). The negative correlation between 5-HT levels and aggression is consistent with results from transgenic mice studies in which serotonergic transmission is altered (Saudou et al. 1994; Audero et al. 2013; Nautiyal et al. 2015). Together, these studies demonstrate that lowered 5-HT signaling at projection sites of 5-HT neurons is associated with the facilitation of aggressive behavior. In contrast, heightened DA levels are associated with the facilitation of aggressive behavior in both psychopathic (Soderstrom et al. 2001) and acute neurological (Ramirez-Bermudez et al. 2010) patients. In rodents, treatment with the DA agonist apomorphine (APO) induces aggressive behavior (Gianutsos and Lal 1976; Pohto 1979; Pruus et al. 2000a, b), and aggression is enhanced during morphine withdrawal (Puri and Lal 1973; Kantak and Miczek 1988). Additionally, increased aggressive behavior is associated with increasing mesolimbic DA levels (van Erp and Miczek 2000; Ricci et al. 2009). Disruption of the DA transporter gene (Dat1) in mice leads to increased extracellular DA concentrations and increased aggression (Rodriguiz et al. 2004). Increased mesolimbic DA release is associated with an increase not only in aggressive behavior, but also in defensive and submissive behaviors (Tidey and Miczek 1996; Anstrom et al. 2009). Thus, 5-HT seems to inhibit aggressive behavior, whereas DA appears to stimulate a broad spectrum of agonistic behavior. It is therefore of interest how alterations in the 5-HT/DA balance might affect the expression of agonistic behavior.

The functional interplay between the 5-HT and DA systems in agonistic behaviors has been reported previously in rodents (Rolinski and Herbut 1979; Rudissaar et al. 1999; Pruus et al. 2000a, b) and teleost fish (Hoglund et al. 2001). However, little is known about the neural mechanisms of the functional interplay between the 5-HT and DA systems. We therefore aimed to investigate the effects of central 5-HT depletion in combination with systemic DA administration on agonistic behavior in rats. To do this, we selectively lesioned 5-HT neurons by injecting 5,7-dihydroxytryptamine (5,7-DHT) (Da Prada et al. 1978), a 5-HT neurotoxin, into the dorsal and median raphe nuclei (DRN and MRN) of resident rats. Additionally, we studied the behavioral effects of DA by administering l-DOPA, the DA precursor, to the 5-HT-lesioned resident rats directly before they underwent RI testing. Following the RI tests, we investigated neural activation in the resident rats using c-Fos immunohistochemistry in a range of brain areas. We observed both the induction of aberrant agonistic behavior and the corresponding neural activation of the ventroposterior part of the anterior olfactory nucleus (AOVP) in the group that received both 5-HT lesions and DA administration; therefore, we used bilateral cytotoxic lesions of the AOVP in rats to identify any causal links between neuronal activity of this nucleus and the induction of aberrant agonistic behavior.

Materials and methods

Animals

Male Wistar rats weighing 350–400 g were purchased at the age of 10 weeks from Charles River Japan (Yokohama, Japan). Male rats of the same strain, weighing 300–320 g and aged 8 weeks, were obtained from the same breeder to use as intruders in the RI tests. After arriving, subjects were allowed to acclimate to the new environment for 1 week before they were used in the experiments. In the acclimation period, rats were housed in groups of two in polycarbonate plastic cages, in a temperature- and humidity-controlled colony room under a reversed 12-h light/dark cycle (lights on at 0:00). The rats were allowed free access to laboratory pellet chow and water. The rats that were to be used as intruders were kept in groups of two in home cages until the behavioral tests.

Experimental design

For the experiment that examined how 5-HT depletion in combination with DA administration affects agonistic behavior, 35 rats were randomly divided into four experimental groups, with no significant differences in average body weights. The following groups were used for behavioral testing: sham-operated (control; n = 7), 5-HT-lesioned (5-HT−; n = 7), DA-administered (DA +; n = 10), and combination-treated (DA +/5-HT−; n = 11) animals. After the behavioral tests, five animals from each group were used for immunohistochemical experiments.

For lesions of the AOVP, another cohort of DA +/5-HT-rats (n = 11) were prepared and assigned to sham-lesioned (n = 5) and lesioned (n = 6) groups.

An additional six age-matched rats were purchased to use for the behavioral and c-Fos analyses of the DA +/5-HT- group in the RI paradigm without an intruder. These rats received either combination-treatment (DA + /5-HT-; n = 3) or sham injections (control; n = 3).

Surgery

Each rat was anesthetized by intraperitoneal (i.p.) injection of a mixture of medetomidine hydrochloride (0.15 mg/kg, Nippon Zenyaku Kogyo Co., Ltd., Koriyama, Fukushima, Japan), midazolam (2.0 mg/kg, Astellas Pharma Co., Ltd, Tokyo, Japan), and butorphanol tartrate (2.5 mg/kg, Meiji Seika Pharma Co., Ltd., Tokyo, Japan). Subjects were placed in a stereotaxic apparatus (David Kopf, Tujunga, CA, USA) fitted with ear bars, with the skull horizontal between bregma and lambda. The scalp was retracted to expose the skull, and a small burr hole was drilled above the injection sites. Rats were pretreated with desipramine hydrochloride (25 mg/kg, i.p.) to minimize damage to noradrenergic neurons. Rats from the 5-HT- and DA + /5-HT- groups were injected with 5 µL of 5,7-DHT (ICN Biomedicals, Inc., Aurora, OH, USA; 10 µg/µL in 0.01% ascorbic acid in saline) into the DRN and MRN with a 30-gauge microsyringe (ITO Co., Fuji, Shizuoka, Japan). In the case of the control and DA + groups, each rat underwent the same procedure, except that the vehicle (ascorbic acid in saline) was injected. All coordinates relative to bregma were as follows: anteroposterior, − 7.8 mm; mediolateral, ± 0 mm; and dorsoventral, − 6.5 mm for the DRN and − 8.5 mm for the MRN. Each injection was made over 5 min and was allowed to diffuse for an additional 5 min before removing the needle. After surgery, the rats were housed individually in their home cages for 1 week to recover.

For lesions of the AOVP, the DA +/5-HT- rats received bilateral injections into the AOVP of either 0.1 M phosphate-buffered saline (PBS; sham-lesioned group) or ibotenic acid (Wako Pure Chemical Industries, Osaka, Japan; lesioned group) dissolved in PBS (8 mg/mL) at the same time as the 5-HT lesions were performed. We used borosilicate glass pipettes (outer diameter = 1.0 mm, Sutter Instrument Co, Novato, CA, USA) pulled to tip diameters of 40 µm for injections into the AOVPs. The coordinates were as follows: 3.2 mm anterior to bregma; ± 1.1 mm lateral to the midline; and 5.7 mm below the brain surface. Injection volume was 0.05 µL of ibotenic acid or PBS, which was delivered to each injection site over 1 min using a micro-infusion pump connected to a 10 µL syringe (Hamilton, Reno, NV). The pipette was left in place for a further 5 min to prevent the toxin from spreading along the capillary track.

l-DOPA treatments and RI tests

After the recovery period, subjects were transferred into five identical 100 × 50 × 50 cm (w × l × h) observation cages, made in-house and located in the adjacent test room from the colony room. The base and interior walls of the cages were made of wooden panels and painted matte gray, and the ceilings and front panels were made of transparent acrylic resin. The ceilings were drilled for ventilation, and the floors were covered with shavings. The light/dark cycle of the test room was matched to that of the colony room. Rats were housed individually in the arena for 1 week, and then exposed to the RI test as residents. In the observation cages, rats were maintained on a restricted diet of 17 g of lab chow per day. Water was available ad libitum.

On the test day, the DA + and DA +/5-HT- rats were injected with l-DOPA (100 mg/kg, i.p., Sigma, St Louis, MO, USA) 1 h before the RI testing, in combination with carbidopa (a peripheral decarboxylase inhibitor; 10 mg/kg, i.p., Sigma) administered 2 h before the test. Carbidopa was administered to reduce peripheral side effects of the decarboxylation of l-DOPA to DA and increase the entry of l-DOPA into the brain. Both drugs were dissolved in saline. The sham-operated and 5-HT- rats received saline injections. The 15 min RI test began with an intruder being placed in the observation cage. The resident rat’s behavior during testing was recorded as a movie file using a horizontally mounted charge-coupled device camera under 75 W red light illumination. Observations were made between 16:00 and 18:30. A trained observer classified the recorded behaviors of the resident rats into 13 chosen elements based on a series of clearly definable postures and acts (Blanchard and Blanchard 1977; Olivier 1981; Koolhaas et al. 2013). The 13 selected behavioral items for the present study were categorized into social exploration (sniffing at the intruder; approaching or moving toward the intruder; contacting the intruder), offensive behaviors (thrusting; wrestling, lateral attack, or offensive sideways movement to the intruder; keeping down or on top of the intruder), conflicting behaviors (upright posture; lateral threat), defensive behaviors (flight; jumping; freezing), and other activities (sexual mounting; submissive posture). We evaluated the frequencies of these behaviors and the amount of time spent in these behaviors during the testing period.

Immunohistochemistry

After RI testing was complete, the intruders were removed from the observation cages and the residents were left undisturbed for 2 h. After this period, the rats were anesthetized by a lethal dose of sodium pentobarbital (150 mg/kg i.p., Kyorithu Seiyaku, Tokyo, Japan) and perfused transcardially with 100 mL ice-cold saline, followed by 500 mL fixative (4% paraformaldehyde and 0.2% picric acid in 0.1 M phosphate buffer). The brains were removed, blocked in the coronal plane, and post-fixed in the same fixative for 24 h. The blocks were then left in 30% sucrose in 0.1 M phosphate buffer until they had sunk. Next, the blocks were coronally sectioned on a freezing microtome (Microm, Walldorf, Germany) into 35 µm sections, which were then washed three times for 30 min each in 0.1 M PBS at room temperature (RT). The sections were incubated for 30 min in PBS containing 1% hydrogen peroxide to neutralize endogenous peroxidases, and were then processed for immunohistochemistry for either the neuronal activation marker c-Fos or the neuron-specific nuclear protein NeuN.

For immunohistochemistry, the sections were rinsed in PBS. To block nonspecific binding sites, the sections were incubated for 1 h in 5% normal goat (for c-Fos staining) or horse (for NeuN staining) serum (Vector Laboratories, Burlingame, CA, USA) in PBS containing 0.3% Triton X-100 (PBS-T). Sections were then incubated overnight at 4 °C with the primary antibody: either rabbit anti-c-Fos polyclonal antibody (Santa Cruz Biotechnology, Dallas, TX, USA; sc-52; 1:500 dilution in PBS-T) or mouse anti-NeuN monoclonal antibody (Millipore, Billerica, MA, USA; 1:1000 dilution in PBS-T). After three 5 min rinses in PBS, the sections were incubated for 2 h at RT with biotinylated goat anti-rabbit antibody (Vector, 1:1000 in PBS-T) for the c-Fos staining, or biotin-conjugated donkey anti-mouse antibody (Vector, 1:1000 in PBS-T) for the NeuN staining. Next, the sections were washed three times in PBS for 5 min each, and then incubated with avidin–biotin–peroxidase complex (Vector) for 2 h at RT. Sections were again rinsed three times with PBS for 5 min, and the peroxidase reaction was then developed in 3,3′-diaminobenzidine tetrahydrochloride (0.2 mg/mL, Wako) and hydrogen peroxide (0.003%, Wako) dissolved in 50 mM Tris–Cl (pH 7.2). All sections were then mounted onto 3% gelatin-coated glass slides, dehydrated with xylene, and coverslipped with Entellan New mounting medium (Merck, Damstadt, Germany).

Quantification of neuronal activity

The neuroanatomical areas containing the regions of interest (ROIs) in the sections were determined according to the Rat Brain in Stereotaxic Coordinates atlas (Paxinos and Watson 2009). Microscopic images of the ROIs were digitized using a CCD camera (DP70, Olympus, Tokyo, Japan) on a microscope (BX51, Olympus) at a magnification of 100 ×. The numbers of positive nuclei within a 0.25 × 0.25 mm area in each ROI were counted from one section per rat by an observer who was blind to the group assignment. If the designated area to be counted was substantially larger than the frame size, the center of the frame was placed in a fixed position within the ROI relative to known anatomical landmarks. In contrast, if the designated area was smaller than the frame size, then only the ROIs (and not the extraneous areas) were counted. The ROI images were acquired using a desktop computer using custom software that was supplied with the camera system. All images were directly imported into Adobe Photoshop CS6 (Adobe Systems, San Jose, CA, USA) and reduced in size. The only post-production enhancements were the conversion of color images to grayscale and the uniform adjustment of brightness and contrast for printing purposes.

Statistics

Statistical analyses were performed using EZR 1.27 (Saitama Medical Centre, Jichi Medical University; http://www.jichi.ac.jp/saitama-sct/SaitamaHP.files/statmedEN.html; Kanda 2013), which is a graphical user interface for R (The R Foundation for Statistical Computing, Vienna, Austria, version 2.13.0). Data were expressed as the mean ± SEM. The behavioral data from the RI tests were analyzed using the Kruskal–Wallis analysis of variance (ANOVA). To further investigate specific group differences, we used the Steel–Dwass multiple comparison test. The data from the c-Fos-immunoreactive (IR) cells were analyzed using one-factor ANOVA. Tukey post hoc comparisons were also used when the main effects were significant. The behavioral data from the ibotenic acid lesion experiments and the RI paradigm without an intruder were analyzed using the Mann–Whitney U test. The c-Fos-IR cell data from the RI paradigm without an intruder were analyzed using a two-sample t test. The statistical significance threshold was set at p < 0.05.

Results

Social exploration behaviors

Among the groups, there were significant differences in the mean frequencies of social exploration (sniffing, \(\chi_{3}^{2}\) = 15.2377, p < 0.01; approaching, \(\chi_{3}^{2}\) = 17.7512, p < 0.001; and contacting, \(\chi_{3}^{2}\) = 17.3265, p < 0.001), as assessed by Kruskal–Wallis ANOVA. Steel–Dwass multiple comparison tests revealed that the DA + rats, who received DA administration only, had markedly lower frequencies of sniffing (t(15) = 3.14, p < 0.01), approaching (t(15) = 3.55, p < 0.01), and contacting (t(15) = 3.42, p < 0.01) compared with the control rats (Fig. 1a). Similar to the DA + rats, the DA +/5-HT- rats, who received both 5-HT lesions and DA administration, had significantly lower frequencies of sniffing (t(16) = 3.28, p < 0.01), approaching (t(16) = 3.60, p < 0.01), and contacting (t(16) = 3.22, p < 0.01) compared with controls. In the 5-HT- group, who received 5-HT depletion only, the frequency of sniffing was significantly lower compared with controls (t(12) = 2.77, p < 0.05).

Fig. 1
figure 1

The frequency of nine social behavior elements categorized into social exploration (a); offensive, submissive, and conflicting (b); and defensive (c) behaviors in the resident–intruder tests of sham-operated (control), DA-administered [DA +], 5-HT lesioned [5-HT-], and combination-treated [DA +/5-HT-] animals. The columns represent the means and the error bars represent the SEMs. *p < 0.05 and **p < 0.01

Full size image

Offensive, submissive, and conflicting behaviors

Some items of offensive (wrestling, \(\chi_{3}^{2}\) = 16.3059, p < 0.001; and keeping down, \(\chi_{3}^{2}\) = 12.7293, p < 0.01) and conflicting (lateral threat, \(\chi_{3}^{2}\) = 16.8455, p < 0.001) behavior, as well as displays of submissive posture (\(\chi_{3}^{2}\) = 14.8832, p < 0.01) behavior, were different among the groups. The mean frequencies of thrusting, upright position, and mounting were not different among the groups (p > 0.1) and were therefore not analyzed further. The post hoc test revealed lower incidences of submissive postures (t(15) = 3.21, p < 0.05) and keeping down (t(15) = 3.43, p < 0.01) in the DA + rats compared with control rats (Fig. 1b). We also observed lower levels of submissive postures (Fig. 1b, t(16) = 3.21, p < 0.01) in the DA +/5-HT- rats compared with the control rats. In the 5-HT- group, the frequency of lateral threats was significantly higher than in the control (t(12) = 2.61, p < 0.05), DA + (t(15) = 2.74, p < 0.05), and DA +/5-HT- (t(16) = 2.97, p < 0.05) groups. The 5-HT- rats also displayed significantly more wrestling behavior compared with the DA + (t(15) = 3.43, p < 0.01) and DA +/5-HT- (t(16) = 2.80, p < 0.05) groups.

Defensive behaviors

There were significant differences in the mean frequencies of defensive behaviors among the groups (freezing, \(\chi_{3}^{2}\) = 16.0654, p < 0.01; flight, \(\chi_{3}^{2}\) = 15.6925, p < 0.01; and jumping, \(\chi_{3}^{2}\) = 25.5208, p < 0.001). However, there were no significant differences in the defensive behaviors between the DA + and control groups. The defensive behaviors were also unaffected in the 5-HT- rats compared with controls. In contrast, the DA +/5-HT- group displayed markedly higher levels of defensive behaviors (Fig. 1c). This group exhibited higher levels of freezing (t(16) = 3.13, p < 0.01), flight (Supplementary Video 1, t(16) = 3.06, p < 0.05), and jumping (Supplementary Video 2, t(16) = 3.64, p < 0.01) than the control group. Moreover, the DA +/5-HT- group had higher frequencies of defensive behaviors than the DA + (flight, t(19) = 2.95, p < 0.05; and jumping, t(19) = 3.55, p < 0.01) and 5-HT- (freezing, t(16) = 2.99, p < 0.05; flight, t(16) = 2.86, p < 0.05; and jumping, t(16) = 3.54, p < 0.01) groups.

During the testing of the DA +/5-HT- group, the flight and jumping episodes occurred abruptly from a freezing posture, and this group frequently displayed jumping behavior in the middle of flight behavior (Supplementary Videos 1 and 2). Once they occurred, the flight and jumping behaviors continued for several seconds. Vocalization (squealing) occurred frequently in this group during the tests (Supplementary Video 2). These characteristics of defensive behavior have previously been reported in wild rats reacting to humans as the threat stimulus (Blanchard et al. 1989). Thus, the behaviors of the DA +/5-HT- rats imply that this group exhibits anti-predatory-like defensive behaviors toward the conspecifics. To test this hypothesis, we focused on the brain areas that are believed to control anti-predatory defensive behaviors (namely, the olfactory system, amygdala, hypothalamic areas, and periaqueductal gray; Fig. 2), and c-Fos immunoreactivity in these areas was compared between the different groups.

Fig. 2
figure 2

adapted from the rat brain atlas (Paxinos and Watson 2009), and shows the six brain areas where c-Fos immunoreactive cells were counted and reported

Schematic diagram of the brain regions of interest in coronal sections. The figure is

Full size image

c-Fos expression in the olfactory system and amygdala

Inspections of c-Fos-immunostained coronal sections through the main and accessory olfactory bulbs revealed a difference in c-Fos expression in the mitral cell layer of the accessory olfactory bulb (MiA) between groups. Using one-way ANOVA, we revealed that the number of c-Fos-IR cells in the MiA was affected by treatment (F(3,17) = 13.74; p < 0.001). In addition, Tukey’s post hoc test revealed a significantly higher number of c-Fos-IR cells in this area in the DA + (p < 0.05) and DA +/5-HT- (p < 0.01) rats compared with control rats (Figs. 3, 4a). The main olfactory bulb (across the glomerular, granular cell, and mitral cell layers) contained sparse c-Fos-IR cells in all groups, and there were no effects of treatment (data not shown).

Fig. 3
figure 3

Representative photomicrographs of coronal sections of the brain. The photomicrographs show c-Fos-immunoreactive nuclei in neuronal cells in the mitral cell layer of the accessory olfactory bulb (MiA), ventroposterior part of the anterior olfactory nucleus (AOVP), anterodorsal part of the medial amygdaloid nucleus (MeAD), dorsomedial part of the ventromedial hypothalamic nucleus (VMHdm), dorsal part of the premammillary nucleus (PMD), and dorsolateral periaqueductal gray (DLPAG) of the four different groups. The scale bars in the left-most panels represent 0.25 mm in each row of the same brain structure. ac anterior commissure, 3v third ventricle, opt optic tract

Full size image
Fig. 4
figure 4

Number of c-Fos-immunoreactive cells in the brain regions of interest after resident–intruder tests. The mean number ± SEM of c-Fos-IR cells (cells/0.0625 mm2) are shown. *p < 0.05, **p < 0.01, and ***p < 0.001

Full size image

In the coronal plane at the most anterior level of the piriform cortex, we observed a dense cluster of c-Fos-IR cells bilaterally in the DA +/5-HT- rats, medially adjacent to the olfactory limb of the anterior commissure and ventrally adjacent to the olfactory ventricle. The location of the c-Fos-IR cell cluster corresponds to the AOVP, according to the rat brain atlas (Paxinos and Watson 2009) and a description in previous literature (Brunjes et al. 2005). Using ANOVA to analyze the c-Fos-IR cell data from this area revealed a significant effect of treatment (F(3,17) = 20.08; p < 0.001). The post hoc analyses showed that the DA +/5-HT- rats had higher c-Fos-IR cell numbers than both control (p < 0.001) and DA + (p < 0.001) rats (Figs. 3, 4b).

The anterodorsal part of the medial amygdala (MeAD) had higher c-Fos-IR cell numbers in the DA + rats compared with controls. Using ANOVA, there was a significant effect of group on c-Fos-IR cell number in the MeAD: (F(3,17) = 18.66; p < 0.001). Subsequent post hoc tests revealed significantly higher numbers of c-Fos-IR cells in the MeAD of DA + rats compared with control rats (p < 0.01, Figs. 3, 4c). Furthermore, there was higher c-Fos expression in the MeAD in the DA +/5-HT- rats compared with controls (p < 0.001).

c-Fos expression in the hypothalamic area and periaqueductal gray

The dorsomedial part of the ventromedial hypothalamic nucleus (VMHdm) differed in c-Fos-IR cell numbers between groups (F(3,17) = 16.45, p < 0.001). The post hoc analysis of the VMHdm revealed significantly more c-Fos-IR cells in the DA +/5-HT- group compared with both the control (p < 0.001) and DA + (p < 0.01) groups (Figs. 3, 4d).

The dorsal part of the premammillary nucleus (PMD) also showed significant effects of group (F(3,17) = 9.09; p < 0.001). In the PMD, there were significantly more c-Fos-IR cells in the DA +/5-HT- rats compared with controls (p < 0.01, Figs. 3, 4e).

The dorsolateral parts of the periaqueductal gray (DLPAG) exhibited significant effects of group (F(3,17) = 31.25; p < 0.001). According to post hoc tests, the DA +/5-HT- rats had a higher number of c-Fos-IR cells compared with controls in the DLPAG (p < 0.001, Figs. 3, 4f). Moreover, the DA +/5-HT- rats also had more c-Fos-IR cells than the DA + rats (p < 0.001) in the DLPAG.

The c-Fos expression in the DLPAG of DA +/5-HT- rats, which exhibited intense defensive behaviors in the RI tests, supports the notion that this area is involved in eliciting panic-like defensive behavior (i.e., jumping) in rodents (Canteras and Goto 1999; Paschoalin-Maurin et al. 2018). The DA +/5-HT- rats had more c-Fos-IR cells in the DLPAG than the DA + rats, and exhibited a much higher incidence of flight and jumping behaviors compared with the DA + rats. Taken as a whole, the evidence suggests that the brain areas with more c-Fos-IR cells in DA +/5-HT- rats compared with DA + rats may be responsible for eliciting intense defensive behaviors. Our c-Fos results imply that the AOVP is a candidate for these defensive behaviors, because this area, as well as the DLPAG, contained more c-Fos-IR cells in the DA +/5-HT- rats than in the DA + rats. In addition, the AOVP has been repeatedly reported to have neural correlates with predator-odor-induced jumping behavior in rats (McGregor et al. 2004; Staples et al. 2008). To test whether the AOVP acts to elicit intense defensive behaviors in the RI tests, we also examined the social behaviors of resident rats with bilateral cytotoxic lesions of the AOVP, in conjunction with the 5-HT lesions and DA administration.

Effects of AOVP lesions

We verified the AOVP lesions histologically using coronal brain sections that were immunostained for NeuN, the neuronal nuclear protein. The ibotenic acid injections resulted in marked neuronal loss around the injection site, which was observed as an unstained region (Fig. 5a). The sham-lesioned rats showed no signs of cell loss around the injection site (Fig. 5b). As Fig. 5c shows, the lesions were centered bilaterally in the AOVP and partially spread to the lateral part of the anterior olfactory nucleus (AOL). The lesioned area extended anteroposteriorly from approximately 5.1 to 3.0 mm anterior to bregma, and did not extend rostrally or caudally into the olfactory bulb or the nucleus accumbens.

Fig. 5
figure 5

Results of the ventroposterior part of the anterior olfactory nucleus (AOVP) lesions. a, b Show photographs of NeuN-immunostained coronal sections from an AOVP-lesioned (a) and a sham-operated (b) rat. The black dotted lines delineate the structures of the brain. The solid lines outline the boundary of the anterior commissure, to facilitate its distinction from the lesioned area. c Represents a schematic diagram of a series of coronal sections of the rat brain, illustrating the extent of bilateral AOVP lesions (darkness represents the coincidence of lesions from different animals). The numbers refer to mm anterior to bregma according to the rat brain atlas. d Shows behavioral measurements for the AOVP-lesioned and AOVP-intact resident rats during resident–intruder testing. The scale bar in b applies to all photographs and represents 0.2 mm. ac anterior commissure, DTT dorsal tenia tecta, AOL lateral part of the anterior olfactory nucleus. Data are expressed as the mean ± SEM. *p < 0.05 and **p < 0.01

Full size image

The behavioral spectrum of the AOVP-lesioned DA +/5-HT- rats in the RI tests is represented in Fig. 5d. As expected, bilateral AOVP lesions reduced the frequency of defensive behaviors (freezing, flight, and jumping) compared with AOVP-intact DA +/5-HT- rats, indicating that the AOVP is involved in eliciting intense defensive behaviors to conspecific intruders in the DA +/5-HT- rats. The incidence of freezing was significantly lower in the AOVP-lesioned rats (p < 0.05) compared with the AOVP-intact DA +/5-HT- rats, as assessed by the Mann–Whitney U test. Frequency of flight was also significantly lower in the AOVP-lesioned rats compared with the AOVP-intact DA +/5-HT- rats (p < 0.01). Furthermore, the AOVP-lesioned rats had much less jumping behavior compared with the AOVP-intact DA +/5-HT- rats (p < 0.05). For social behaviors other than defensive behaviors, the AOVP-lesioned rats were not significantly different from the AOVP-intact DA +/5-HT- rats, except for approaching behavior, which was significantly more frequent in the AOVP-lesioned rats than the AOVP-intact DA +/5-HT- rats (Fig. 5d, p < 0.05). This result indicates that the decreased defensive behaviors in the lesioned rats did not arise from a general reduction in locomotion. We observed neither lateral threat nor wrestling behavior in this set of RI tests.

Behavioral and c-Fos analyses of the DA +/5-HT- rats in the RI paradigm without an intruder

Our findings of higher c-Fos levels in the AOVP, VMHdm, and DLPAG in DA +/5-HT- rats compared with controls do not conclusively indicate that the existence of an intruder rat induces c-Fos expression in these brain areas. Thus, to provide more evidence for this idea, we performed an additional experiment to compare the levels of c-Fos expression in these areas, as well as the frequencies of jumping and flight behaviors, between the DA +/5-HT- and control rats using the RI paradigm without an intruder. The c-Fos expression levels in the tested areas were not significantly different between the DA +/5-HT- and control rats (Supplementary Table 2). This finding was very different from the results that were obtained when an intruder was present in the RI test. In the behavioral analysis, we observed one jumping and flight episode in the DA +/5-HT- rats, and no such episode in the control rats. There was no significant difference (p = 0.51, Mann–Whitney U test) in the frequencies of these behaviors between the two groups. Taken together, these results suggest that the higher frequencies of defensive behaviors and the higher c-Fos expression levels in these areas in the DA +/5-HT- rats are related to the existence of an intruder, rather than merely being an effect of 5-HT/DA manipulation.

Discussion

In the present study, we observed intense flight behavior from intruders in resident rats that had received a combination of 5-HT depletion and DA administration. The RI test was developed to investigate the spontaneous and natural expression of social behaviors in laboratory rodents (Adams 1976; Koolhaas et al. 1980). Consequently, residents and intruders show distinct social behaviors in a species-typical manner. The residents display aggressive behaviors toward the intruder, whereas the intruders display defensive behaviors toward the resident (Adams 1976). However, in our study, the DA +/5-HT- residents exhibited higher levels of defensive behavior (flight), which deviated from the species-typical response. Thus, the DA +/5-HT- residents seemed to act in an aberrant way, as if they were the intruders.

In parallel with the increase in flight behavior, high levels of jumping were also observed in the DA +/5-HT- rats in the current study. In laboratory rodents, this behavior is induced by the presence (Paschoalin-Maurin et al. 2018) or odor (McGregor et al. 2004; Staples et al. 2008) of predators. Additionally, wild rats show high levels of jumping in response to an approaching experimenter, who is a potential predator to them (Blanchard et al. 1986). Thus, the behavior of our DA +/5-HT- rats seems similar to these reactions toward predators. In contrast, jumping has not been observed in typical RI tests of laboratory rats (Grant and Mackintosh 1963; Grant 1963). In summary, the DA +/5-HT- rats display aberrant defensive behaviors, and the functional interplay of 5-HT depletion and DA administration appears to be crucial for the generation of these behaviors, because neither 5-HT depletion nor DA administration alone can induce these aberrant behaviors. Further study is required to evaluate whether innate anti-predatory defensive behaviors (Roseboom et al. 2007) are strengthened in the DA +/5-HT- rats.

The results of c-Fos immunohistochemistry revealed neural activation patterns that support the similarity of the DA +/5-HT- group’s behavior to that of predator-threatened animals. The DA +/5-HT- rats had elevated numbers of c-Fos-IR cells in the DLPAG, VMHdm, and AOVP relative to the DA + and control groups. The VMHdm is highly interconnected with the PMD, and both nuclei are critical elements of the medial hypothalamic defensive system, subserving defensive responses toward predators (Canteras 2002). The DLPAG receives direct projections from the PMD (Canteras and Swanson 1992) and elicits c-Fos expression in response to exposure to predators (Canteras and Goto 1999). Thus, the DLPAG appears to be implicated in the medial hypothalamic defensive system. In support of this, manipulations of neural activity of the DLPAG (Bittencourt et al. 2004, 2005) and the PMD (Blanchard et al. 2003, 2005; Markham et al. 2004) affect anti-predatory defensive behaviors. Our c-Fos results suggest that defensive circuitry is activated toward conspecifics by the combination of 5-HT depletion and DA administration, which results in aberrant defensive behaviors.

The AOVP has increased c-Fos expression in response to predator odors, and this is associated with elevated levels of defensive behaviors and increased c-Fos expression in the medial hypothalamic defensive system (Dielenberg et al. 2001; Staples et al. 2008; Bowen et al. 2013). These previous studies have suggested that the AOVP plays a role in processing predatory odor stimuli (Dielenberg et al. 2001). However, its role in defensive behavior has not yet been tested. In the present study, we performed cell-body-specific chemical lesions in the bilateral AOVP and revealed a decline in defensive behaviors in the AOVP-lesioned DA +/5-HT- rats. To our knowledge, this is the first time that the AOVP has had a reported role in defensive behaviors. Moreover, the present result strongly suggests that the AOVP constitutes the neural circuitry subserving anti-predatory defensive behaviors. From the AOVP, projections to the medial hypothalamic areas, including the ventromedial hypothalamic nucleus, have been reported (Price et al. 1991). Thus, elevated neural activity of the AOVP might activate the medial hypothalamic defensive system through its projection to, and activation of, the VMHdm. Future studies will determine whether the AOVP projections to the VMHdm have a role in the regulation of defensive behaviors in predator-threatened animals.

The inhibitory effect of AOVP lesions on the aberrant defensive behaviors of the DA +/5-HT- rats suggests that interactions between 5-HT and DA occur in this area to activate the defensive system. The expression pattern of c-Fos in the DA + group, which received DA administration only, implies a possible role of DA in triggering AOVP activation. This group had higher c-Fos expression in the MiA and MeAD relative to the control group (Figs. 3, 4a, c). These structures are known to be functionally involved in anti-predatory defensive behaviors, as well as predator-odor-induced autonomic activity and hypothalamic–pituitary–adrenal stress hormone secretion (Blanchard et al. 2005; Takahashi 2014). Therefore, activation of these structures in the DA + group might cause the reduction in social exploration that was observed in this group (Fig. 1a). Both the DA + and DA +/5-HT- groups showed c-Fos expression in the MiA and the MeAD. The MiA has direct projections to the MeAD, which sends projections to the AOVP (Shipley et al. 1995; Brunjes et al. 2005). Given the functional connectivity between these structures, it is possible that the activation signal of the MiA is transmitted to the AOVP via the MeAD in both the DA + and DA +/5-HT- rats.

Compared with the control rats, the 5-HT- rats did not exhibit any differences in c-Fos expression in any ROIs in the present study. Thus, the precise role of 5-HT lesions in the aberrant behaviors of the DA +/5-HT- rats remains to be investigated. The simplest hypothesis is that 5-HT lesions potentiate the capability of AOVP excitability. The raphe nuclei send axons into the anterior olfactory nucleus (AON, McLean and Shipley 1987; Shipley et al. 1995), and mRNA for 5-HT1A and 5-HT2C receptors are expressed in the AOVP (Wright et al. 1995). In addition, neurons with immunoreactivity for 5-HT3 receptors are distributed throughout the AON (Morales et al. 1998). Thus, 5-HT released from axons innervating the AON may modulate the activity of the nucleus. Reduced serotonergic transmission may cause facilitation or “priming” of AON excitability, resulting in the neural disinhibition of AON-mediated behaviors. In the DA +/5-HT- rats, MiA activation by DA administration may be transmitted to the primed AOVP, and the resulting excessive activity of the nucleus may inappropriately trigger anti-predatory defensive behaviors. Recently, Li et al. revealed that 5-HT reduces the excitability of pyramidal neurons via 5-HT2C receptors in the anterior piriform cortex of mice, and that optogenetic activation of serotonergic neurons in the DRN decreases the odor-evoked activity of pyramidal neurons (Wang et al. 2020). These previous findings support our idea that 5-HT modulates the capability of AOVP excitability.

Previous studies appear to demonstrate a nonspecific effect of 5,7-DHT or l-DOPA treatment on territorial aggression (Supplementary Table 3). In the present study, the 5-HT- rats had a higher incidence of lateral threat (Fig. 1b), which is regarded as a defensive–offensive conflicting behavior (Manning 1972). These results suggest that 5,7-DHT treatment strengthens motivations for both offensive and defensive behavior, and that ambivalent motivations may cause frustration in this group. As for the l-DOPA treatment, previous studies have displayed dose-dependent variable effects, with increased offensive aggression at low doses and decreased offensive aggression at high doses (Supplementary Table 3). In our study, the DA + rats, who received a high dose of l-DOPA, displayed less offensive aggression (i.e., keeping down) compared with the saline-injected control group (Fig. 1b). This is in line with the results of previous studies. Collectively, the differing effects of 5,7-DHT and l-DOPA treatments can be explained both pharmacologically and ethologically. Different doses and sites of drug administration may cause varied effects owing to differences in neuroanatomical distributions and signaling properties among receptor subtypes. Furthermore, different contexts and circumstances, such as the size of the testing arena, can lead animals to display distinct behaviors (Blanchard et al. 1989; Eilam 2005).

In conclusion, we have shown that a combination of central 5-HT depletion and systemic DA administration causes aberrant defensive behaviors toward conspecifics in rats. The c-Fos immunohistochemistry and cytotoxic lesion results reveal for the first time that the AOVP is involved in eliciting defensive behaviors. This result sheds light on the possible role of the AOVP in modulating defensive behaviors, coupled with activation of the hypothalamic and midbrain areas. Furthermore, the aberrant agonistic behaviors induced by the disturbance of 5-HT/DA interactions may represent the critical role of 5-HT/DA balance for sustaining species-typical social behaviors between individuals.