Introduction

Olfaction is the primary sensory modality in mice and rats, the most common organisms used for models of psychiatric and behavioural disorders. Rodents rely heavily from birth on their sense of smell for locating and identifying food, avoiding predators, establishing social hierarchies, finding mates, caring for their young and a host of other behaviours. Modelling these diseases in rodents depends on distinguishing abnormal from normal behaviours (either conditioned or innate) in a controlled manner. An accurate interpretation of the emotional state of a mouse or an assessment of whether a rat is truly experiencing delusions or hallucinations as humans with some mental illnesses do is not possible. However, endophenotypes associated with psychiatric disorders, including depression, aggression, anxiety or social isolation, can be reasonably modelled in rodents. By utilizing natural olfactory-mediated rodent behaviours, an experimenter can design robust paradigms to test these endophenotypes. For example, social recognition testing exploits the preference rodents have for investigating an unfamiliar conspecific over a familiar one based mainly on their smell (Mathiasen and DiCamillo 2010). This interaction can be used to test deficits in memory (McIntyre et al. 2012; Migdalska et al. 2012a, b) and sociability (McFarlane et al. 2008; Sankoorikal et al. 2006).

Moreover, some specialised olfactory cues, such as pheromones and kairomones, initiate stereotypical behavioural reactions in an innate manner; these provide a controlled natural stimulus to drive behaviour on demand. For example, an innately attractive mouse pheromone (a secreted signal that releases a specific reaction in a member of the same species) was recently demonstrated to generate a two-week spatial memory with just a single trial (Roberts et al. 2012). Similarly purified kairomones (signals that release a specific reaction in a member of a different species) isolated from cats, rats and foxes can release intense fear-like behaviours in mice on first exposure (Papes et al. 2010; Wallace and Rosen 2000). This is in contrast to artificial fear-conditioning (typically using a foot-shock associated with an otherwise innocuous visual or auditory cue), which requires training. Rodents can respond to conditioned and unconditioned fear stimuli with various behaviours (Morrow et al. 2000), illustrating the importance of using an appropriate test stimulus and an appreciation of the context in which fear is generated.

Given their reliance on smell, anosmia or hyposmia (a complete or partial deficiency in olfactory function) unsurprisingly results in severe behavioural abnormalities in rodents. Prior to olfactory-mediated behavioural phenotyping, we need to establish that the test animals can indeed smell. This is particularly important when working with models of psychiatric or neurological disease, as olfactory dysfunction is frequently associated with schizophrenia (Moberg and Turetsky 2003; Nguyen et al. 2010; Turetsky et al. 2009), bipolar disorder (Hardy et al. 2012), depression (Gopinath et al. 2011; Negoias et al. 2010), posttraumatic stress disorder (Croy et al. 2010) and Parkinson’s and Alzheimer’s diseases (Rahayel et al. 2012). Therefore, whereas an olfactory deficient rodent might recapitulate a psychiatric endophenotype, it will also limit the range of tests available to assess other behavioural traits.

In this review, we describe the complex rodent olfactory system, the types of odour cues that they detect and the types of behaviours that each subsystem mediates. We additionally evaluate some methods of testing olfactory function and examine some common olfactory-mediated behavioural paradigms that can be used to assess specific psychiatric endophenotypes.

The rodent olfactory system

Rodents have four major olfactory subsystems, in order of decreasing size: the main olfactory epithelium (MOE), vomeronasal organ (VNO), the septal organ (SO) and Grueneberg ganglion (GG; Fig. 1). Despite being anatomically segregated and expressing diverse olfactory receptor (OR) families, these subsystems have synergistic and overlapping roles in detecting odours, pheromones and kairomones (Ma 2010). Consequently, most have been implicated in mediating behavioural responses to olfactory cues.

Fig. 1
figure 1

Rodent olfactory systems and their circuits. The major olfactory system (purple) consists in the main olfactory epithelium (MOE) projecting neurons to the olfactory bulb (OB). The accessory olfactory system (blue) consists in the vomeronasal organ (VNO) projecting neurons to the accessory olfactory bulb (AOB). Each subsystem sends axons to a number of central brain regions, with further intracortical projections to hypothalamic nuclei (BST bed nucleus of the stria terminalis, nLOT nucleus of the lateral olfactory tract, nAOT nucleus of the accessory olfactory tract, AON anterior olfactory nucleus). The two minor olfactory subsystems are the septal organ (SO) and Grueneberg ganglion (GG)

Full size image

The MOE, the largest rodent olfactory subsystem, is largely tasked with smelling inherently “neutral” odorants via approximately 1000 different ORs. Olfactory sensory neurons project their axons to the olfactory bulb (OB), where they synapse with second-order neurons that in turn project to various cortical centres in the brain involved in olfactory perception and discrimination (Fig. 1). However, surgical ablation and genetic engineering studies have consistently found that behaviour is also dramatically altered when rodents lack MOE-OB-mediated neuronal signalling. Mice with mutant alleles of genes involved in canonical signal transduction of odours in the MOE, e.g., Nav1.7 or Adcy3, fail to display general odour-guided behaviours such as innate odour investigation, habituation, discrimination, associative learning or recognition of odour qualities (Weiss et al. 2011; Wong et al. 2000). They also display deficits in stereotyped sexual, aggressive and maternal behaviours and at least some show behaviours indicative of anxiety (Belluscio et al. 1998; Mandiyan et al. 2005; Wang et al. 2006; Wang and Storm 2011; Weiss et al. 2011). Whether these are attributable to the ablation of some specialised MOE circuits or are the consequence of general anosmia is unclear (Stowers and Logan 2010a).

Also present in the nasal epithelium are afferents from the trigeminal nerve that predominantly detect noxious volatile stimuli (Finger et al. 1990). The trigeminal is a somatosensory cranial nerve that can sense touch, temperature and pain and perceive atmospheric humidity. Thus, it is responsible for detecting the coolness associated with the smell of menthol, the burning sensation of ammonia and the stinging effect of carbonated beverages (Brand 2006). When toxic chemicals are detected, the trigeminal nerve mediates a protective reflex, namely sneezing, which minimises penetration into the nasal cavity. Many odour molecules simultaneously stimulate the main olfactory and trigeminal systems, e.g., n-butanol and pyridine (Brand 2006; Doty 1975). Several studies have shown that the activation of the trigeminal system can impact on olfactory signal processing, influencing odour perception (Daiber et al. 2013; Frasnelli et al. 2007).

The second major olfactory subsystem in the rodent nose is the VNO, a bilateral fluid-filled tubular structure located rostral of the MOE and dorsal to the palate (Fig. 1). Although the VNO can detect some odorants (Trinh and Storm 2003), it is largely tuned to sense semiochemicals, including pheromones and kairomones, which promote innate behaviours (for a review, see Tirindelli et al. 2009). These are detected by neurons selectively expressing approximately one or a small number of 360 vomeronasal receptors (VRs; Isogai et al. 2011) that project axons to the accessory olfactory bulb (AOB). Neurons from the AOB project to the hypothalamus via the amygdala. Surgical ablation or removal of the rodent VNO affects numerous social behaviours, including territorial marking, aggression, maternal, sexual and courtship behaviours (Wysocki and Lepri 1991). Genetic ablation of transduction proteins in the VNO, such as Trpc2 and Gnao1, also alters aggressive and sexual behaviours (Chamero et al. 2011, 2007; Kimchi et al. 2007; Leypold et al. 2002; Stowers et al. 2002) and predator avoidance (Papes et al. 2010). However, the VR repertoire is highly variable between rodent species (Yang et al. 2005; Zhang et al. 2007) and even within different strains of the same inbred species (Wynn et al. 2012). Caution should therefore be exercised in drawing general conclusions about VNO-mediated behaviour from rodents, as its function, morphology and even presence varies significantly across mammals (Salazar and Quinteiro 2009).

The SO is a small bilateral island of sensory neurons on the base of the nasal septum (Fig. 1; Ma et al. 2003). Its sensory receptor profile is a small subset of those found in the MOE, although it appears to detect an unusually broad range of odorants (Ma 2010). SO neurons are also mechanosensory and might play a role in sensing air flow through the nostrils or synchronising sniffs with rhythmic activity in the OB (Grosmaitre et al. 2007). The behavioural role of the SO is unclear; it has been proposed to act as a general “early warning” odour detector, although an ablation study in rats has not supported this hypothesis (Giannetti et al. 1995).

The smallest olfactory subsystem is the GG, bilaterally paired clusters of grape-like neurons found close to the opening of the naris (Fig. 1; Fuss et al. 2005; Koos and Fraser 2005). GG neurons have been reported to express canonical ORs, a VR and other receptor sub-types (Fleischer et al. 2006, 2007). They also respond to multiple sensory stimuli including odorants, cool temperatures and a pheromone (Brechbuhl et al. 2008; Mamasuew et al. 2008, 2011). Mice with ablated GG neurons do not display induced freezing behaviour when exposed to volatile alarm pheromones from conspecifics, illustrating a role in innate olfactory-mediated behaviour (Brechbuhl et al. 2008).

Humans have a significantly simplified olfactory system compared with rodents, consistent with our reduced olfactory acuity and a diminished reliance on olfaction for communication. With the possible exception of some cetaceans, a MOE is found in all mammals but the number of sensory neurons and OR genes varies considerably (Hayden et al. 2010). A much smaller proportion of the human nasal cavity is lined by olfactory epithelium than in rodents (Tirindelli et al. 2009) and we have approximately one third of the number of functional receptors (Niimura and Nei 2007). The GG has been located in all mammalian species thus far studied, humans included but the SO is found in only some mammals and has not been identified in humans (Ma 2010). The existence of a VNO in humans is a matter of some historical controversy. The organ certainly begins to develop during human embryogenesis but appears to regress to a simple diverticulum in the post-natal nasal septum. Critical reviews conclude that the adult human VNO pit is unlikely to contain sensory neurons or axons projecting to the brain (Meredith 2001; Tirindelli et al. 2009) and notably, genes specialised for VNO-mediated chemosensation in rodents, such as Trpc2, are pseudogenised in Old World monkeys and apes (Liman and Innan 2003).

Maternal behaviours

Interactions between a rodent dam and her young are critical for their survival to adulthood and are heavily olfactory influenced. Stereotypical maternal behaviours displayed soon after birth include the intense licking clean of newborn mice and rats to stimulate respiration, suckling and defaecation (Brouette-Lahlou et al. 1999; Logan et al. 2012). During the subsequent weeks, a dam will typically display pup retrieval, nursing, grooming and nest-building behaviours and respond aggressively to unfamiliar intruders.

To assess a dam’s interactions with her young, pups are typically separated from their mother and then either cross-fostered or returned to her cage in a controlled manner (for a review, see Hahn and Lavooy 2005). Video-recording is advisable, followed by blind scoring of behaviours such as latency of pup retrieval, duration of pup licking, duration of presence in the nest and latency and duration of nursing. Interference with extremely young litters might trigger infanticide by the dam but, on the other hand, maternal behaviour generally decreases with the maturation of the litter (Gelhaye et al. 2011); therefore, the timing of the experiment is important. A further consideration is the strain of rodent, as some inbred strains display notably lower maternal behavioural responses than others (Crawley et al. 1997).

A second measure of maternal behaviour is maternal aggression. A modified resident-intruder (RI) assay is typically carried out in which an unfamiliar male intruder is introduced to a resident dam (Fig. 2a). In mice, the removal of the litter immediately prior to the introduction of the male does not significantly diminish aggression; therefore, this is encouraged to safeguard the welfare of the pups (Svare et al. 1981). The female’s offensive behaviour is then recorded, including biting, lunging, tail-rattling and chasing (Kolunie and Stern 1995). Both maternal aggression and nest building appear to be regulated by semiochemical cues, or pheromones, as these behaviours are lacking in mice lacking a functional VNO (Hasen and Gammie 2009; Kimchi et al. 2007; Leypold et al. 2002). A number of neuropsychiatric disease models are associated with alterations in maternal behaviour. Neurotensin (NT), a neuropeptide that has been implicated in schizophrenia (Caceda et al. 2006), when delivered by intracerebroventricular injection reduces maternal aggression in mice but has no impact on pup retrieval (Gammie et al. 2009). Consistent with this, NT is down-regulated in the brains of mice bred for high maternal aggression (Gammie et al. 2007). Chronic treatment of rats with some antipsychotic drugs disrupts nest building, pup retrieval and licking behaviour soon after dosing (Li et al. 2005). Maternal behaviours are also affected in models of stress and anxiety, notably when the corticotropin-releasing factor signalling pathways are altered (Gammie et al. 2005, 2008).

Fig. 2
figure 2

Olfactory-mediated behavioural paradigms. a A resident male or lactating female mouse (R) will display aggression towards an unfamiliar intruder (I) in the resident-intruder test. b Measurement of the time a hungry rodent takes to locate a morsel of buried food can be used to assess anosmia or hyposmia. c Olfactory habituation can be measured by quantifying the decreasing sniffing or investigation time to the same odour presented several times. d Dishabituation will occur when a second novel odour is presented to a habituated rodent, resulting in an increase in sniffing and investigation time

Full size image

Male aggression

Whereas female rodents typically only display overtly aggressive behaviour towards males in a time-limited maternal context, inter-male aggression occurs frequently within wild rodent colonies, as adult males jostle for territory or position in complex social hierarchies. Male aggression among conspecific rodents typically takes one of two major forms: offensive (typically in response to a territorial or dominance challenge) and defensive (usually in response to an offensive attack), both behaviours being transient and neither being predatory (for a review, see Blanchard et al. 2003). In contrast laboratory-bred rodents are often maintained from birth in small single-sex groups with limited territorial space. Thus, whereas groups of males in a cage might form dominant/sub-ordinate relationships that are established by initial aggressive bouts, they subsequently display low levels of aggression compared with wild-derived animals. Therefore, the majority of studies into aggression in rodent models quantify the offensive behaviour of a socially isolated male, the “resident”, after a single unfamiliar male “intruder” is introduced into his territory (Fig. 2a). Although a dyadic interaction is being measured, numerous other factors can influence the resident-intruder (RI) test, including the age, strain, prior sexual experience and, most importantly, the relative social status of each animal (Brodkin et al. 2002; Pletzer et al. 2007). Thus, particular care must be taken to control these factors when planning a RI test.

Compared with other behaviours, the olfactory basis of mouse inter-male aggression is relatively well understood. Genetic ablation of either the MOE or VNO inhibits aggression against an intruder, suggesting that several sensory stimuli might be necessary (Leypold et al. 2002; Mandiyan et al. 2005; Stowers et al. 2002). Consistent with this, at least two different classes of compound found in male urine are known to promote aggression (Chamero et al. 2007; Novotny et al. 1985). One of these comprises the major urinary proteins (MUPs), a species-specific family of small secreted proteins found at high concentrations in the urine of male rats and mice (Logan et al. 2008). Although exposure to MUPs alone does not elicit aggression, these substances are effective when daubed onto the back of a castrated intruder (which would otherwise not provoke an attack; Chamero et al. 2007). MUPs activate specific sets of neurons in the VNO of mice, suggesting that they activate a dedicated olfactory circuit that mediates inter-male aggressive behaviour (Chamero et al. 2011). Identification of the receptors for these MUPs will permit the mapping of this circuit and perhaps reveal the neural logic underpinning the difference between the sexes in aggressive behaviour (Stowers and Logan 2010b). Additionally, the use of chemically defined MUPs to promote aggression in the context of a docile castrate might go some way to minimising the impact of variability in intruder behaviour on RI test data (Miczek et al. 2001).

The aggression displayed in the resident-intruder test has been suggested to differ significantly from natural offensive behaviour in rodents and be symptomatic of stress from an “isolation syndrome” (Valzelli 1973). Indeed, increased aggressiveness and unusual attack behaviours have been reported in rats that are socially isolated from a young age, as a model of child neglect (Toth et al. 2008). Nevertheless, the RI test is used widely to measure aggression as an endophenotype associated with a range of psychiatric disorders. For example, neuregulin 1 and its ErbB2/B4 receptors have been associated with schizophrenia. Genetic ablation of this signalling pathway in the central nervous system of mice results in a hyper-aggressive RI phenotype that can be rescued by treatment with the antipsychotic drug, clozapine (Barros et al. 2009). Other mutants with enhanced aggressive phenotypes include mice lacking the serotonin receptor, 5htr1b, which is associated with depression and anxiety (Saudou et al. 1994) and models of monoamine oxidase A deficiency (Cases et al. 1995; Scott et al. 2008), which is associated with an impulsive aggression disorder. In contrast, mice lacking a dopamine receptor isoform associated with attention-deficit hyperactive disorder display decreased aggression in an RI test, whereas mice lacking dopamine β-hydroxylase and therefore unable to synthesise noradrenaline, show almost no RI aggression whatsoever (Marino et al. 2005; Vukhac et al. 2001).

Anxiety and depression

The RI test is also widely used to generate the “social defeat” model of depression in rodents. Animals repeatedly used as intruders, often paired with particularly aggressive residents (Golden et al. 2011), will eventually display depressive and anxious behaviours, such as social isolation, anhedonia and increased thigmotaxis. These can be rescued by the chronic administration of antidepressants (Berton et al. 2006). The social consequence of this interaction is the establishment of an extremely submissive intruder that will go to great lengths to avoid provoking further attacks. Dominant/submissive relationships between male rodents are olfactory mediated, as males are able to identify rival individuals by their urinary odour profiles and adjust their behaviour in a process that probably involves at least some VNO-mediated cues (Hurst et al. 2001). Whereas social defeat has a number of advantages in modelling human depression over other stress-based methods, some caveats exist to its use. As has been documented, approximately 30% of inbred C57BL/6 J mice used in this paradigm are resistant to developing social avoidance and anhedonia but most display anxiety-like behaviour in thigmotaxic tests (Krishnan et al. 2007). This phenotypic variability might be a useful model of human resistance to stress-related affective disorders but can also make the interpretation of the experimental data more challenging. The underlying cause of this variation is unknown but the answer might lie in the previous social experience of the test mice. The intruders are typically housed in all male groups for up to 2 months prior to RI testing (Golden et al. 2011) and thus are likely to have formed dominant/submissive relationships among themselves. Further work will be required to establish whether resistance to social defeat correlates with prior social status.

A second problem with the social defeat paradigm is that it is limited to males. Thus, another experimental paradigm of chronic social stress has been developed that is effective in both male and female mice (Schmidt et al. 2010, 2007). This involves the random assortment of animals in groups of four, twice weekly for seven weeks. The test mice are therefore faced with unfamiliar cage-mates every four days, whereas control mice are maintained with the same cage-mates. The former creates a highly stressful, unpredictable social environment that has long-term effects on depressive and anxious behaviours. These include perturbations in locomotor adaptation to an open field, novelty-induced suppression of feeding and thigmotaxic behaviour in an elevated plus maze, all of which behaviours can be attenuated by antidepressant treatment (Schmidt et al. 2007). In males, the persistent need to re-establish social hierarchies through aggressive bouts recapitulates the social defeat paradigm. However, because the test starts with mice aged only 28 days, strong dominance/submissive relationships are less likely to have previously been formed and thus less variability might be seen among the test animals. The mechanism of social stress is less clear in females and the olfactory dependence of this test has yet to be tested directly but the rapidly changing odour profiles of the home cages of the animals are probably partly responsible for inducing stress (Gerdin et al. 2012).

Olfactory bulbectomy (OBX) is another common rodent model of depression and anxiety (Willner and Mitchell 2002). The majority of the research involving this technique has been on rats, although studies of OBX mice do exist and tend to describe similar behavioural and neurochemical characteristics (Hellweg et al. 2007; Zueger et al. 2005). The OB is the location of the first synapse of sensory neurons from all four olfactory subsystems. Accordingly, a successful OBX should result in a complete anosmia, although this might be time-limited, as studies of neonatal OBX rats have shown that forebrain tissue grows forward into the OB cavity by 3 months of age. At least some of these rats have sensory projections from the MOE that form glomerular structures in the forebrain and are able to discriminate between odours (Slotnick et al. 2004). OBX also influences social behaviours, including sexual, aggressive, neonatal and maternal responses, plus some that are not directly olfactory cue-dependent: locomotor activity, conditioned taste aversion and some cognitive processes including spatial learning (Brunjes 1992; Harkin et al. 2003). OBX surgery typically involves the removal of the entire bulb by aspiration, which inevitably results in local tissue damage, inflammation and a disruption of local blood flow. Moreover, since projections pass from the OB to a great number of both cortical and limbic brain loci, the process of neural degeneration and rewiring as a consequence of OBX is likely to be widespread (Harkin et al. 2003) and has been proposed to account for the unanticipated cognitive or locomotor deficits that underpin the depression and anxiety model.

Perhaps the best known behaviour that is associated with the OBX rat but that is not obviously olfactory cue-driven is hyperactivity in open field or open maze tests under high illumination (Leonard and Tuite 1981). OBX mice also demonstrate significantly increased activity and less time in the centre of the open field under similar conditions, compared with sham-operated controls (Zueger et al. 2005). This characteristic is widely interpreted as an anxiety-like behaviour and is attenuated by the chronic (but not acute) administration of various anti-depressants thereby mimicking the therapeutic action of these drugs in humans (van Riezen and Leonard 1990). The precise mechanism of this atypical behaviour is unknown but it is not thought to be olfactory-mediated. This is reinforced by comparisons in the literature between OBX rats and those with their MOE chemically destroyed displaying differences in open field behaviour (Harkin et al. 2003). More recently, mice with genetically ablated VNO and MOE have been challenged in a range of tests for anxiety and depression (Glinka et al. 2012). Trpc2 null (VNO-deficient) mice display no atypical behaviours but Cnga2 mutant (MOE-deficient) mice spend less time in the centre of an open field, similar to OXB mice (Zueger et al. 2005), suggesting that congenital peripheral anosmia does indeed produce anxiety-like behaviours in mice (Glinka et al. 2012). No associated hyperactivity was observed in the Cnga2 mutant mouse, although this might be because the open field was not illuminated. Glinka and co-workers have also tested a mutant mouse that has a “monoclonal nose” and therefore undergoes persistently increased odour-evoked MOE signalling in response to a single odorant found in mouse urine (Fleischmann et al. 2008). This line has been found to have a similar anxious behavioural phenotype, which is dependent on the aberrant odour-evoked signalling (Glinka et al. 2012). Taken together, one can conclude that olfaction has a greater influence on the anxiety-related behaviours observed in the OBX rodent than previously thought. As hyper-activation of the MOE results in a similar phenotype to no activation whatsoever, a general disturbance in olfactory signalling, rather than the loss of a specific olfactory circuit, appears to be a causative factor. Further work will be required to ascertain the way that this promotes anxiety in a task that does not appear to require odour processing, although, given the strong social reliance of rodents on smell, an alteration in olfactory perception is likely to be persistently stressful. To our knowledge, rodents with both VNO and MOE genetically or chemically inactivated have not been behaviourally tested; therefore, complete peripheral congenital anosmia might recapitulate OBX even more fully.

Olfactory testing

Some psychiatric disorders have hyposmia or anosmia as one of their symptoms (Hardy et al. 2012, Negoias et al. 2010), whereas neurodegenerative diseases such as Alzheimer’s and Parkinson’s can show deficits in olfaction many years before any other cognitive or motor symptoms (Rahayel et al. 2012). One has to consider that an olfactory dysfunction in a rodent model could have indirect consequences that influence other subsystems and eventually behaviour. Anosmia of the MOE, for example, can reduce their sniffing rate and limit chemo-investigatory behaviour, which in turn might inhibit VNO function and affect aggressive behaviour (Meredith 1994). Therefore, an assessment of the sense of smell of a rodent model prior to widespread behavioural testing is of fundamental importance. Many different tests are available for assessing olfactory function, with each paradigm providing different information (Cleland et al. 2002). Therefore, any conclusion regarding odour perception must be put carefully into context, with due consideration of the odorant, animal model and paradigm involved.

Identifying anosmia and hyposmia

Anosmia is usually considered to be the inability to smell odours via the MOE, whereas hyposmia is a reduced functioning of this subsystem. The inability to smell odours via the VNO is sometimes referred to as avnosmia (Del Punta et al. 2002) and here we refer to the inability to detect odours via all olfactory subsystems as “complete anosmia”. The identification of anosmic rodents is relatively easy, as they display high levels of perinatal lethality (Belluscio et al. 1998; Brunet et al. 1996; Wong et al. 2000). However, anosmic pups that survive their first 48 h tend to develop normally; this is because MOE-mediated olfaction is critical in guiding newborn rodents to their first milk meal but, once suckling is initiated, it is reinforced by other sensory cues (Logan et al. 2012; Teicher and Blass 1977). The survival rate of newborn anosmic rodents can be enhanced by reducing litter sizes, assisting the pups to locate their mother’s nipples, increasing the nutritional content of the maternal diet or hand-feeding. In contrast, hyposmic animals, even those with severe olfactory deficits, appear to have no problems suckling and are difficult to identify without specific testing. The simplest method for assessing anosmia or severe hyposmia is with a “buried food” or “hidden cookie” test (Fig. 2b). This involves determining the time it takes for a food-restricted rodent to locate a hidden morsel of familiar palatable food (Yang and Crawley 2009). Hungry normosmic rodents usually locate the food within 1–2 min and anosmic animals are typically unsuccessful within the 5–15 min time limit (Yang and Crawley 2009). Rodents with severe anosmia might take significantly longer than controls to locate the food but mild hyposmia is often not identifiable by using this test. A more sensitive discrimination test should therefore also be to carried out.

Olfactory habituation and discrimination

Habituation is a decrement of a behavioural response that results from repeated stimulation, does not involve sensory adaptation or fatigue and does not need a reward association (Rankin et al. 2009). It is a simple form of implicit or non-declarative memory when previous experience aids the performance of the task but no conscious awareness of this experience is apparent (Wilson and Linster 2008). Problems with habituation and sensory gating (or filtering) have been linked to disorders such as schizophrenia (Ludewig et al. 2003), autism (Ornitz et al. 1993), Alzheimer’s disease (Takeuchi et al. 2011) and substance abuse (Hunt and Morasch 2004). In rodents, rapid presentations of a neutral odour, with intervals of a few seconds, will induce a habituation that lasts a few minutes. This short-term habituation is mediated by metabotropic glutamate receptors at synapses from the OB afferents into the piriform cortex (Wilson and Linster 2008). Odour presentations spaced 5 min apart will produce a habituation that can last over an hour. This long-term habituation is dependent on N-methyl-D-aspartate receptors within the OB (McNamara et al. 2008).

Two types of behavioural tests for olfactory habituation can be used. The simplest type, both in terms of neural circuitry and interpretation, is the odour-evoked orienting reflex. Detection of a novel odour elicits investigative physiological and behavioural reactions in rodents. These include change in a breathing rate or of sniffing and re-directing the nose towards the stimulus (Wachowiak et al. 2009). These changes habituate with repeated stimulation and are odour-specific (Sundberg et al. 1982). Thus, the measurement of a simple behaviour such as sniffing can provide information about olfactory orientation and gating. Both spontaneous and novel odour sniffing rates are remarkably constant throughout the life of a mouse but, in models of Alzheimer’s disease, spontaneous breathing rates change with age, whereas odour-evoked sniffing remains unchanged (Wesson et al. 2011). In humans, on the other hand, patients with Parkinson’s disease have impaired odour-evoked sniffing. This is thought to be a contributing factor to poor performance in olfaction tests (Sobel et al. 2001).

A second and most commonly used test for olfactory habituation in mammals is measurement of the active investigation of an odour stimulus (Cleland et al. 2002). If an odour is presented to a normosmic rodent over repeated trials, the time that the subject spends investigating it decreases (Fig. 2c). In contrast, an anosmic mouse will typically display significantly less initial time investigating an odour stimulus and this does not decay over the trials. Olfactory habituation tests are often used to demonstrate that deficiencies in behavioural responses are not attributable to an inability to detect odour cues (Ferguson et al. 2000), although due consideration is not always given to the olfactory subsystem being tested. Banana or almond extract can be used to test MOE function and, whereas urine or a swab from a soiled cage of a conspecific is often used as a “social odour” (Stack et al. 2008), such odours are likely to contain ligands that activate both the MOE and VNO. To test VNO-mediated habituation specifically, synthesised or recombinant protein pheromones should be used, although whether rodents habituate differently to pheromones compared with neutral odours is not yet clear. After odour habituation, a dishabitution test can be carried out to test olfactory discrimination (Fig. 2d). This can be a second neutral odour, such as vanilla, or social odours from a second conspecific of a different sex or strain. If the test animal is able to discriminate the novel odour from the familiar odour, it will often spend more time investigating (Yang and Crawley 2009). By experimenting with test odours that smell relatively similar, hyposmia can be identified by using a habituation/dishabituation paradigm, although similar neutral odours might require reinforcement to motivate the animal to discriminate between them (Linster et al. 2002). A more sensitive method involves the measurement of rapid reaction times to odour choices when conditioned with a reward. Both healthy mice (Abraham et al. 2004) and rats (Uchida and Mainen 2003) can discriminate between odours with >90% accuracy in less than 0.5 s. These tests, although more technically challenging to conduct, have the capacity to resolve even mild olfactory impairments.

Motivation and odour hedonics

Several motivating factors can affect odour investigation. Accordingly, the hedonic nature of the stimulus is of particular importance. Odour hedonics can also influence olfactory function in psychiatric diseases. For example, whereas odour identification impairment is a common feature in schizophrenia, the deficits seem only to be present when identifying pleasant or neutral odours but not unpleasant ones (Kamath et al. 2011). Social odours reproducibly exploit the innate olfactory-mediated behaviour of rodents (Engelmann et al. 2011) but, ostensibly, neutral odours are not all equally attractive (Logan et al. 2012) and, thus, the absolute amount of investigation or sniffing might vary by stimulus. The neutral odours of vanilla, almond and banana are used because they are all mildly attractive natural food odours but are unrelated to the food with which laboratory rodents are likely to be familiar. Odours that are naturally aversive, such as butyric acid, butanol or 2,4,5-trimethyl-3-thiazoline (Endres and Fendt 2009; Logan et al. 2012) or those that act as trigeminal irritants, should be avoided, as should highly attractive odours that could elicit behaviours (e.g., gnawing) that interfere with the time spent investigating (Yang and Crawley 2009).

Future perspectives

Few mammals can rival the mouse and rat in their ability to sense odours. The VNO alone expresses many hundreds of receptors that are likely to influence every aspect of rodent behaviour, whereas the reliance on smell is such that mice and rats born without a functional MOE typically starve to death within days. One could argue that the rodent olfactory system is a particularly ill-suited model for investigating human behavioural disorders, not least because the VNO subsystem is entirely lacking in humans and our reliance on smell is reduced, such that congenitally anosmic children often remain undiagnosed for years. However, the research described in this review demonstrates that the modelling of human psychiatric disorders in rodents relies heavily on interpreting natural olfactory-mediated rodent behaviour and, with that, comes some unique advantages. First, the sources of complex olfactory cues can now be purified to isolate discrete ligands, such as single protein kairomone or pheromones (Chamero et al. 2007; Haga et al. 2010; Papes et al. 2010; Roberts et al. 2010), providing an experimental leverage over complex social behaviours that is unmatched by other exogenous stimuli. Eliciting social behaviours by biochemical moieties (rather than exposure to other animals) should result in more robust assays, less experimental variability, decreased costs and a reduction in the number of experimental animals used. Second, the one (or two) receptor(s) per neuron patterning logic of olfactory receptors and VRs provides the unique means to track discrete, behaviourally relevant neurons from the periphery to the brain (Luo and Katz 2004). Thus, an understanding of which olfactory ligand activates which cognate receptor neuron to release a distinct behaviour should eventually enable the study of the entire neural circuit, first in healthy animals and then in models displaying aberrant behaviours. When combined with the promise of optogenetic technology for centrally controlling these circuits (Lin et al. 2011), olfactory-mediated behaviour in rodents seems likely to remain at the forefront of psychiatric research for the foreseeable future.