Keywords

1 Introduction

Normal cognition requires the coordination of numerous complex processes, including sensory information processing, sustained and divided attention, short- and long-term memory, and executive functions. Many neurologic and psychiatric disorders, including senile dementia, Alzheimer’s disease (AD), and schizophrenia, are associated with severe impairments in cognitive functions that are directly correlated with poor social and functional outcomes (Green 1996; Green et al. 2004; Farlow and Cummings 2007).

There is now accumulating evidence that modulation of the muscarinic cholinergic system is involved in normal cognitive processes and that imbalances in the neurotransmission of this system may account, at least in part, for the cognitive deficits associated with AD and schizophrenia. For example, nonselective muscarinic acetylcholine receptor (mAChR) antagonists produce or exacerbate impairments in cognition in animals and in healthy control, normal aging and AD populations (Domer and Schueler 1960; Pazzagli and Pepeu 1965; Drachman and Leavitt 1974; Bartus et al. 1982; Sunderland et al. 1986; Newhouse et al. 1988; Rusted and Warburton 1988). In addition, mAChR antagonists can also induce psychotomimetic-like symptoms in healthy humans and/or aggravate existing behavioral disturbances in patients with dementia or psychosis (Osterholm and Camoriano 1982; Agnoli et al. 1983; Hamborg-Petersen et al. 1984; Strauss et al. 1990). Conversely, indirect-acting mAChR agonists, such as acetylcholinesterase inhibitors (AChEIs), and direct-acting mAChR agonists can improve aspects of normal cognitive function and/or improve cognitive impairments in AD patients, and in animals, they reverse deficits induced by mAChR antagonism or lesions of cholinergic basal forebrain circuitry (Aigner and Mishkin 1986; Robbins et al. 1989a, b; Rupniak et al. 1989, 1991; Matsuoka et al. 1991; Bodick et al. 1997a, b; Cummings 2003; Shekhar et al 2008). Nonselective mAChR agonists and AChEIs have also enhanced cognitive performance, particularly in the domains of attention and memory, in schizophrenic patients (see review in Chouinard et al. 2007; Edelstein et al. 1981; Shekhar et al. 2008). Taken together, these observations have led to the hypothesis that selective activators of mAChRs may provide an important alternative approach for the treatment of the cognitive impairments associated with neurologic and psychiatric disorders, such as AD and schizophrenia.

However, while AChEIs are clinically approved for the treatment of mild-to-moderate cognitive dementia associated with AD, the effects of these compounds on deficits in memory and other cognitive functions remain modest (Amenta et al. 2001). Unfortunately, early clinical studies using direct-acting mAChR agonists for AD and schizophrenia have ultimately failed in clinical development due to a lack of true subtype selectivity that resulted in a number of dose-limiting adverse effects from nonselective activation of peripheral mAChRs (Bruno et al. 1986; Bodick et al. 1997a, b; Shekhar et al. 2008). The high conservation of the acetylcholine (ACh) binding site across the five mAChR subtypes has presented a major impediment to the development of highly selective mAChR orthosteric-site ligands. The lack of subtype-selective mAChR ligands has also limited insights into the relative roles of the mAChR subtypes in the different aspects of cognition and the clinical efficacy observed with the AChEIs and nonselective muscarinic mAChR agonists.

Using an alternative strategy, our group and others have recently identified ligands for mAChRs that activate a specific receptor subtype through action at sites that are less highly conserved and topographically distinct relative to the orthosteric binding site of ACh, termed allosteric sites. Allosteric agonists activate the receptor subtype directly in the absence of the endogenous ligand ACh, while positive allosteric modulators (PAMs) bind to an allosteric site and potentiate the effects of ACh, but have no intrinsic activity. Because mAChR PAMs can only exert their effects in the presence of ACh at a given synapse, these ligands may maintain normal temporal and spatial components of endogenous ACh neurotransmission. This latter feature may provide an important advantage in the treatment of cognitive impairments in early stage dementia or schizophrenia, as recent findings suggest that optimal levels of ACh transmission for cognition are dynamic and task dependent (Kozak et al. 2006; Hasselmo and Sarter 2011). To date, these novel allosteric activators of the different mAChR subtypes have shown efficacy in preclinical models for the enhancement of cognition, and possess suitable physiochemical properties for optimization as potential clinical candidates.

In this chapter, we will provide a brief overview of cholinergic circuitry and mAChR distribution and function in the central nervous system (CNS). We will next review the effects of different mAChR antagonists and agonists in preclinical models of cognition and in clinical populations. Finally, we will highlight recent developments with novel subtype-selective allosteric agonists and PAMs of M1 and M4 mAChRs in preclinical models for the enhancement of cognition.

2 Anatomy of the Cholinergic System

2.1 Cholinergic Cell Groups and Their Target Regions

Within the CNS, cholinergic projection neurons are organized into relatively discrete cell groups in the basal forebrain and the caudal mesencephalon. As described in the seminal work of Mesulam and colleagues (Mesulam et al. 1983), six groups of cholinergic projection neurons, termed Ch1–Ch6, can be distinguished based on their localization and projection pattern (Fig. 1). Cell groups Ch1–Ch4, located in the basal forebrain of the rat, are thought to be involved in attention, learning, and memory functions (Everitt and Robbins 1997). The cholinergic neurons of the nucleus basalis magnocellularis (Ch4), which in primates is known as the nucleus basalis of Meynert (NBM), provide wide-spread cholinergic projections throughout most of the cerebral cortex, and degeneration of these neurons is a hallmark of AD (McGeer et al. 1986). In addition, the Ch4 cells innervate the amygdaloid complex (Mesulam et al. 1983; Price and Stern 1983). Cholinergic neurons in the medial septum (Ch1) and the vertical limb of the diagonal band of Broca (Ch2) send projections to the hippocampal formation and to the medial aspects of the cortex, such as the cingulate and retrosplenial cortices (Eckenstein et al. 1988). The olfactory bulb is the recipient of cholinergic projections from the Ch3 cell group, located in the horizontal limb of the diagonal band of Broca. The cholinergic projection neurons of the caudal midbrain, which are involved in arousal, sleep, and the regulation of dopaminergic cell groups (Datta and Siwek 1997), are located in the pedunculopontine tegmental nucleus (PPTg, Ch5) and the laterodorsal tegmental nucleus (LDTg, Ch6), from where they project to the thalamus, the pontine reticular formation, and areas of the ventral midbrain (Mesulam et al. 1983; Satoh and Fibiger 1986; Clarke et al. 1987; Hallanger et al. 1987; Semba et al. 1990). The parcellation scheme developed by Mesulam and colleagues (1983) has proven to be invaluable for conceptualizing the various aspects of cholinergic function. However, the analysis of forebrain cholinergic function is complicated by the fact that non-cholinergic projection neurons are embedded in the cholinergic cell groups (Woolf et al. 1986). Therefore, results from lesion studies targeting the cholinergic basal forebrain need to be interpreted carefully (see Robbins et al. 1989a, b).

Fig. 1
figure 1_7

Schematic diagram illustrating the location of the cholinergic cell groups of the rat brain and their projections. (a) Sagittal view showing Ch1 (medial septum), Ch2 (vertical limb of the diagonal band of Broca [DBB]), and Ch3 (horizontal limb of the DBB) and their projections to the hippocampal formation, cerebral cortex, and olfactory bulb. (b) Sagittal view depicting Ch4 (nucleus basalis magnocellularis) and its projections throughout the cortex and amygdala, as well as Ch5 (pedunculopontine tegmental nucleus) and Ch6 (laterodorsal tegmental nucleus) innervating the thalamus, substantia nigra, and ventral tegmental area. (c) Coronal section through the striatal complex showing large cholinergic interneurons in the dorsal striatum and nucleus accumbens. Drawings are based on the work of Kimura et al. (1980), Mesulam et al. (1983), Eckenstein et al. (1988), and Gould et al. (1989). Ch1–Ch6 cholinergic cell groups; AMG amygdala; cc corpus callosum; CP caudate-putamen (striatum); HPC hippocampus; NAS nucleus accumbens; OB olfactory bulb; THAL thalamus; SN substantia nigra; VTA ventral tegmental area

Full size image

2.2 Regional Distribution of Cholinergic Axons

Dense cholinergic fiber plexus originating from the basal forebrain are seen throughout neo- and allocortical areas. The laminar distribution of cholinergic fibers varies slightly across cortical areas, but layer V generally receives the most dense cholinergic fiber innervation (Eckenstein et al. 1988; Mechawar et al. 2000). The cholinergic innervation of the hippocampus is most prolific at the border between stratum oriens and pyramidal layer and in the molecular layer, while the densely packed pyramidal and granule cell layers themselves receive very little cholinergic input (Ichikawa and Hirata 1986; Schäfer et al. 1998). Cholinergic fiber density varies across the nuclei of the amydaloid complex; the most densely innervated area is the basolateral nucleus (Hellendall et al. 1986). In subcortical areas, moderate cholinergic innervations are seen in select thalamic nuclei, including the anteroventral, centromedial, and reticular nuclei (Gonzalo-Ruiz et al. 1995; Schäfer et al. 1998), and in the midbrain dopamine cell groups (Gould et al. 1989; Oakman et al. 1995; Omelchenko and Sesack 2006).

2.3 Striatal Cholinergic Interneurons

The striatal complex, including the nucleus accumbens, does not receive any extrinsic cholinergic innervation, but instead contains cholinergic interneurons as the sole source of ACh. These cholinergic interneurons are scattered throughout the striatal matrix compartment, but are largely absent from striatal patches (Gerfen and Bolam 2010). Although large cholinergic interneurons make up less than five percent of striatal neurons, their wide dendritic arbors enable them to exert control over a large striatal area (Kimura et al. 1980; Bolam et al. 1984; Phelps et al. 1985).

3 Muscarinic Receptor Distribution

For the purpose of this chapter, we will focus on the well-established distribution of the five mAChR subtypes in the rodent brain. Our description of the distribution of mAChRs will be limited to select brain regions that are thought to be involved in cognition and that either contain cholinergic neurons or receive cholinergic innervations. These areas include the cerebral cortex, hippocampus, thalamus, the basal ganglia, and basal forebrain and caudal midbrain cholinergic cell groups.

3.1 Expression of Muscarinic Receptor Message

Distribution maps of M1–M5 mAChR mRNA, obtained by in situ hybridization histochemistry, show that mAChRs are expressed throughout the rodent brain, albeit not uniformly (Fig. 2). There are pronounced differences in the overall expression levels of the five muscarinic receptors, with M1 and M5 receptors being the most and least abundant receptor subtype, respectively. Moreover, each muscarinic receptor exhibits a regional expression pattern that is strikingly different from other members of the muscarinic receptor family (Brann et al. 1988).

Fig. 2
figure 2_7

Distribution of M1–M5 muscarinic receptor mRNA in the mouse brain. This is a composite of images obtained from the Allen Mouse Brain Atlas (2009) developed by the Allen Institute for Brain Science (Lein et al. 2007) and available online at http://mouse.brain-map.org. CP caudate-putamen; HPC hippocampus; NAS nucleus accumbens; PFC prefrontal cortex; SN substantia nigra; VTA ventral tegmental area

Full size image

The M1 receptor is not only most prominently expressed in the hippocampus, but is also abundant throughout all layers of the cortex, where the superficial layers stand out by being more intensely labeled than the remaining layers. Striatal medium spiny neurons as well as interneurons also express high levels of M1 message (Bernard et al. 1992); caudal to the striatum, subcortical M1 expression decreases along a rostro-caudal gradient from the diencephalon to the midbrain. Moderately high M2 receptor expression is found mainly in the brain regions containing cholinergic cell bodies (Vilaró et al. 1992) as well as in some thalamic nuclei including the midline, parafascicular, and reticular nuclei. In the hippocampus and cortex, M2 message is sparse; in cortical layer IV, it is completely absent. The M3 receptor is mainly expressed in the hippocampus and in the cortex, except for layers III and IV which are mostly devoid of M3 message. Very low levels of M3 mRNA are seen in the striatum and basal forebrain (Brann et al. 1988). The highest density of M4 receptors is found in the striatal complex (Vilaró et al. 1991), followed by allocortical areas, such as the hippocampus and amygdala. Expression of M4 message is relatively high in all layers of the neocortex; like M2, M4 receptor message is prominently expressed in central cholinergic neurons (Sugaya et al. 1997). The muscarinic receptor with the most restricted expression is M5. It is found in low abundance in the ventral tegmental area and the pars compacta of the substantia nigra (Vilaró et al. 1990).

3.2 Muscarinic Receptor Protein Expression

The global distribution of muscarinic receptor protein was initially assessed using a monoclonal (M35) pan-muscarinic antibody (van der Zee et al. 1989; for review, see van der Zee and Keijser 2011). With the development of subtype-selective muscarinic receptor antibodies, it became feasible to quantitate levels of receptor protein in microdissected brain regions (Li et al. 1991; Wall et al. 1991; Yasuda et al. 1993) and to determine both the cell types expressing certain mAChR subtypes and the (sub)cellular localization of mAChRs at the light and electron microscopic level (Levey et al. 1991; Hersch et al. 1994; Hersch and Levey 1995). Immunohistochemical studies demonstrated that M1–M5 protein distribution corresponds to a large degree with the mRNA expression maps indicating receptor expression at the soma and dendritic level. Furthermore, they revealed that muscarinic receptor proteins were prominently expressed presynaptically as both autoreceptors and heteroceptors (Table 1).

Table 1 Distribution of muscarinic receptor protein in brain circuits involved in cognition
Full size table

3.2.1 Cortex

M1, M2, and M4 are the most abundant muscarinic receptor proteins in the cortex (Levey et al. 1991). M1 protein, expressed in pyramidal cells, is enriched in layers II/III and VI, whereas M4 is localized in somata of layer II–IV cells. Terminals located in layer IV and at the border between layers V and VI exhibit strong M2 labeling, which is in agreement with the dense cholinergic innervation of these cortical layers and the role of M2 as autoreceptor (Eckenstein et al. 1988; Mechawar et al. 2000).

3.2.2 Hippocampus

The complexity of hippocampal cholinergic circuitry is illuminated by the diverse pre- and postsynaptic distribution of mAChRs, suggesting an intricate muscarinic regulation of hippocampal function. Both intrinsic neurons (pyramidal neurons, granule cells, and interneurons) and terminals originating from basal forebrain and entorhinal cortex prominently express M1–M4 receptors (see Table 1) (Levey et al. 1995b; Rouse and Levey 1996, 1997, 1998; Rouse et al. 1999, 2000).

3.2.3 Amygdala

Pyramidal neurons in the basolateral amygdala, a limbic region involved in learning and expression of fear conditioning, prominently express M1 protein (McDonald and Mascagni 2010).

3.2.4 Striatum

Approximately eighty percent and close to half of medium spiny neurons, the principal cell type in the striatum, express M1 and M4 receptor proteins, respectively (Hersch et al. 1994). Interestingly, the M4 receptor is mainly localized to the medium spiny neurons projecting to the substantia nigra reticulata (Ince et al. 1997), making M4 an interesting target to alter striatal output pathways differentially. In contrast, M2 protein is mainly expressed in striatal cholinergic interneurons, where the M2 receptor subserves the function of an autoreceptor (Hersch et al. 1994; Hersch and Levey 1995). Presynaptically located M1–M3 receptor proteins are thought to be localized to corticostriatal (M1/M3) and thalamostriatal (M2/M3) terminals (Hersch et al. 1994). Overall, the high expression of mAChRs in the striatum suggests that muscarinic ligand may be useful for modifying striatum-mediated learning processes, in particular procedural learning (Saint-Cyr et al. 1988; Cayzac et al. 2011).

3.2.5 Thalamus

Expression of mAChR proteins in the thalamus is restricted to M1 and M3 in the anterodorsal and -ventral nuclei and to M2 in the reticular nucleus (Oda et al. 2001, 2007). The thalamus as an important relay station to the cortex and striatal complex may, therefore, be subject to muscarinic regulation via M1 and/or M3 mechanisms. The presence of M2 in the reticular nucleus, whose GABAergic projections inhibit thalamic relay nuclei, suggests that M2 may play a role in global control of thalamic output (Cox et al. 1997; Pinault and Deschênes 1998).

3.2.6 Cholinergic Neurons

In the basal forebrain and other cholinergic cell groups, the principal mucarinic receptor protein is M2, which is located both in cholinergic cell bodies and in unidentified axon terminals (Levey et al. 1995a).

4 Role of Muscarinic Receptor Subtypes in Cognition

4.1 Findings with mAChR Antagonists and KO Mice

Based on an extensive literature, nonselective mAChR antagonists, such as scopolamine, disrupt multiple domains of cognitive function, from sensory information gating, attention, and memory to higher problem-solving skills in rodents, monkeys, and humans, as shown in Table 3; also see chemical structures of representative mAChR antagonists and their in vitro affinities for the different mAChR subtypes in Fig. 3 and Table 2, respectively (see Terry et al. 2006; Barak 2009; Klinkenberg and Blokland 2010 for complete reviews). For example, scopolamine, trihexyphenidyl, and benztropine produced robust dose-dependent disruptions of prepulse inhibition (PPI) of the acoustic startle reflex, a model of sensory information processing, at doses that had no effects on startle response (Jones and Shannon 2000). Scopolamine markedly decreased accuracy and/or response rates in the 5-choice serial reaction time task, a preclinical model of attentional functions used to test rats and monkeys (Jäkälä et al. 1992; Callahan et al. 1993; Jones and Higgins 1995; Higgs et al. 2000; Mirza and Stolerman 2000; Shannon and Love 2005, 2006; Shannon and Eberle 2006; Spinelli et al. 2006). In addition, scopolamine induced impairments in attention in humans, including in the attentional components of the CogState Early Phase Battery and in the digit vigilance test (Ellis et al. 2006; Fredrickson et al. 2008). With regard to learning and memory, muscarinic antagonism with scopolamine produced robust deficits in performance accuracy in numerous memory-related behavioral tasks in rodents and monkeys, including spatial memory tasks such as the Morris water maze and radial arm maze, classic Pavlovian conditioned responding, delayed non-matching to sample, and object recognition tasks (Buresová et al. 1986; Riekkinen et al. 1990; Dennes and Barnes 1993; Anagnostaras et al. 1995, 1999; Rudy 1996; Mishima et al. 2000; Feiro and Gould 2005; Betz et al. 2007; Sheffler et al. 2009; Dietrich and Jenck 2010). In humans, scopolamine decreased performance accuracy in measures of visual and verbal learning and item recognition memory tasks (Sherman et al. 2003; Green et al. 2005; Fredrickson et al. 2008; Thienel et al. 2009). Scopolamine has also been reported to produce impairments in executive functions, including attentional set-shifting in rats and Groton maze learning in humans (Chen et al. 2004; Fredrickson et al. 2008). In review of the dose-related disrupting effects of scopolamine and other nonselective mAChR antagonists, the interpretation of these effects are clearest in measures of sensory discrimination and attentional function, in which deficits are observed within a dose range that does not produce confounding effects on general motor output and/or levels of arousal as observed in models of learning and memory.

Fig. 3
figure 3_7

Chemical structures of representative muscarinic receptor antagonists

Full size image
Table 2 Receptor affinities [nM]of orthosteric muscarinic receptor antagonists
Full size table

Recent findings from studies using either mAChR KO mice or antagonists are providing more defined roles for each of the mAChR subtypes in the modulation of cognition. In the case of M1 mAChRs, this particular subtype regulates a variety of physiologic effects in hippocampal and cortical brain regions, most notably enhancement of glutamatergic signaling through potentiation of N-methyl-d-aspartate (NMDA) receptor function (Marino et al. 1998). Modulation of NMDA receptor neurotransmission is key for the acquisition and consolidation of new learning and memories; and its disruption is speculated to account, at least in part, for the cognitive impairments observed in many neurological and psychiatric disorders (Marino and Conn 2002; Tsai and Coyle, 2002). Consistent with a role of M1 in learning and memory, the M1-preferring mAChR antagonist pirenzepine impaired accuracy and/or acquisition in tasks of passive avoidance, Morris water maze, and visual discrimination in rats (Fig. 3, Tables 2 and 3) (Hunter and Roberts 1988; Drinkenburg et al. 1995). Moreover, M1 mAChR KO mice have reduced long-term potentiation in response to theta burst stimulation, a physiologic endpoint thought to be procognitive in nature (Anagnostaras et al. 2003). In contrast to the effects of nonselective mAChR antagonists, M1 KO mice have shown normal performance in hippocampus-mediated tasks, including in the Morris water maze task with or without scopolamine challenge (Miyakawa et al. 2001), but distinct impairments in behavioral tasks that require medial prefrontal cortex (mPFC) function (Anagnostaras et al. 2003). For example, M1 KO mice relative to wild-type (WT) controls showed pronounced performance deficits in non-matching-to-sample tasks, including win-shift radial arm maze learning and social discrimination tests (Anagnostaras et al. 2003). Despite significant enhancement in the acquisition of contextual fear conditioning, M1 KO mice performed poorly after a time period when the task becomes independent of hippocampal function (Anagnostaras et al. 2003). In support of these findings, the highly selective M1 mAChR antagonist VU0255035 (see Fig. 3, Tables 2 and 3) had no effect on acquisition of contextual fear conditioning, a hippocampus mediated memory task (Sheffler et al. 2009). Taken together, these studies indicate a consistent role for M1 mAChR in the modulation of mPFC-mediated tasks, but future studies using the selective M1 mAChR antagonist VU0255035 are needed to further evaluate the effects of selective disruption of M1 activity in other cognitive functions.

Table 3 Effects of muscarinic receptor antagonists on cognition
Full size table

For the role of M2 in cognition, previous studies have postulated that selective M2 mAChR antagonists may provide improvements in the cognitive deficits observed in dementia patients by increasing cholinergic signaling through antagonism of M2 mAChRs on presynaptic cholinergic terminals (Rouse et al. 2000; Zhang et al. 2002; Tzavara et al. 2003). Consistent with this hypothesis, the selective M2 mAChR antagonists, BIBN-99 and SCH57790 (see Fig. 3, Tables 2 and 3) improved performance in the passive avoidance and Morris water maze tasks in normal and aged rats, and in fixed ratio discrimination in monkeys (Table 3) (Quirion et al. 1995; Carey et al. 2001; Rowe et al. 2003). However, M2 mAChRs also function as heteroceptors localized on the axon terminals of non-cholinergic neurons that mediate presynaptic regulation of release of other neurotransmitters (Rouse et al. 2000). Not surprisingly, M2 mAChR KO mice have shown deficits in tasks of working memory and cognitive flexibility, as well as hippocampal long-term potentiation, suggesting that blockade of M2 mAChRs on both cholinergic and non-cholinergic nerve terminals may disrupt, not enhance, overall cognitive function (Tzavara et al. 2003; Seeger et al. 2004). Consistent with the M2 KO mouse cognitive phenotype AFDX116, another selective M2 mAChR antagonist decreased accuracy and increased response latencies and omissions in a rodent visual discrimination task (see Fig. 3, Tables 2 and 3) (Drinkenburg et al. 1995). Thus, more detailed studies with M2 mAChR antagonists are needed to further understand the full therapeutic potential of M2 mAChR antagonists for the treatment of clinical populations with varying levels of cholinergic tone.

To date, the relative importance of the M3 mAChR in modulating different aspects of cognitive function remains undefined. M3 mAChR KO mice have shown robust impairments in contextual fear conditioning, a classic hippocampus-mediated memory task (Poulin et al. 2010). However, there are currently no selective M3 mAChR antagonists reported in the literature, and the M3-preferring antagonist imidafenacin had no effect on performance in the Morris water maze, another hippocampus-mediated memory task (Kobayashi et al. 2007) (see Fig. 3, Tables 2 and 3). Whether selective M3 mAChR activators may have procognitive properties remains unclear as does the issue whether a viable therapeutic index could be achieved between activation of central and peripheral M3 mAChRs.

The significance of M4 mAChRs in cognitive functions remains unclear because of the pre- and postsynaptic localization of M4 mAChRs within the CNS (Levey et al. 1991; Zang and Creese 1997; Zhang et al. 2002; Tzavara et al. 2004). Previous in vivo microdialysis studies have shown significant increases in basal midbrain extracellular ACh concentrations in M4, but not M2 mAChR KO mice (Tzavara et al. 2004). Moreover, scopolamine-induced increases in midbrain extracellular ACh concentrations were dampened in the M4 mAChR KO mice (Tzavara et al. 2004). M4 mAChR KO mice also displayed increased DA efflux in response to psychotomimetics (Tzavara et al. 2004). These findings suggest that activation of M4 mAChRs may provide feedback control on basal and evoked DA release in the striatum. The tight regulation of striatal DA and ACh neurotransmission by M4 mAChRs may be critical for cognitive functions, such as procedural learning and effort-based decision making, tasks that require striatal involvement. Interestingly, the M4-preferring mAChR antagonist tropicamide disrupted PPI of the acoustic startle reflex, a task that is dependent on proper mesolimbic DA neurotransmission (Ukai et al. 2004) (Fig. 3, Tables 2 and 3). Tropicamide administration also resulted in decreased accuracy in a visuospatial delayed non-matching-to-sample task in rats (Betz et al. 2007). Studies using selective M4 mAChR agonists and antagonists need to further dissect the role of M4 mAChRs in other aspects of cognition, as will be discussed in the allosteric modulator section of this chapter.

With the expression of M5 mAChRs limited to the VTA and substantia nigra pars compacta, it is not surprising that preliminary studies with M5 mAChR KO mice have reported disruptions in the proper regulation of dopaminec-mediated behavioral tasks (Vilaró et al. 1990; Weiner et al. 1990). In particular, M5 mAChR KO mice have impaired PPI (Thomsen et al. 2007) and reduced sensitivity to the effects of different drugs of abuse (Basile et al. 2002; Fink-Jensen et al. 2003; Yamada et al. 2003; Thomsen et al. 2005; Steidl and Yeomans 2009). While there are currently no available selective M5 mAChR antagonists, the studies with M5 mAChR KO mice suggest that selective blockade of M5 mAChRs might be useful for regulating the hyperactivation of mesolimbic dopaminergic circuitry in patients with schizophrenia. Moreover, the proper function of nonneuronal M5 mAChRs expressed in the cerebrovasculature that control cerebrovasodilation and blood flow may also indirectly impact cognitive functions (Yamada et al. 2001; Araya et al. 2006). Vascular pathology has been implicated in AD, and dysfunction in cholinergic control of cerebral blood vessel dilation may contribute, in part, to the pathophysiology of this disease. Cerebrovascular deficits in M5 mAChR KO mice are associated with neuronal atrophy and deficits in performance of the novel object recognition task (Araya et al. 2006), which further support the role of M5 mAChRs in the modulation of cognitive function through nonneuronal mechanisms.

4.2 Findings with mAChR Orthosteric Agonists

Over the last 2 decades, the drive to improve cognitive impairments in patient populations with AD and other dementias has resulted in the development of two major pharmacologic approaches that modulate mACh neurotransmission, specifically indirect modulation through the enhancement of general cholinergic tone with AChEIs and direct modulation by mAChR orthosteric agonists. To date, only the AChEIs tacrine, donepezil, galantamine, and rivastigmine are clinically approved for the treatment of cognitive impairments associated with mild-to-moderate AD. While AChEIs can improve cognitive deficits in dementia patients, their therapeutic benefits are limited by a short duration of action, dose-limiting side effects, relatively modest efficacy on memory deficits, and a large population of nonresponders (Pepeu and Giovannini 2010; Birks 2006; Birks and Flicker 2006; Persson et al. 2009; Hasselmo 2006; Barten and Albright 2008).

As an alternative to the limited clinical utility of AChEIs, considerable efforts have been focused on the development of highly selective mAChR orthosteric agonists for the treatment of cognitive impairments in AD; representative chemical structures for each compound are depicted in Fig. 4 with their in vitro binding affinities at each mAChR subtype described in Table 4 and highlighted efficacy in different cognitive tasks shown in Table 5. All of the mAChR agonists presented in Table 4, including the reported M1-preferring agonist WAY-132983 and the M1/M4-preferring mAChR agonist xanomeline, exhibit relatively nonselective profiles of binding affinities across the different mAChR subtypes, underscoring the drawback of designing orthosteric site ligands that target the highly conserved ACh binding site of the five mAChR subtypes. Due to the relatively nonselective in vitro binding profiles for each of these mAChR orthosteric agonists, the role(s) of the different mAChR subtypes in the observed in vivo effects of these compounds remain unclear. However, as shown in Table 5, the majority of mAChR orthosteric agonists produced robust reversals of pharmacologic and/or lesion-induced deficits in different cognitive domains, including sensory information processing, attention, and various aspects of learning and memory. For example, oxotremorine and xanomeline reversed deficits in PPI induced by the non-selective mAChR antagonist scopolamine and the D1/D2 dopamine receptor agonist apomorphine (Jones and Shannon 2000; Stanhope et al. 2001; Jones et al. 2005) (Table 5). Cevimeline improved performance in divided or visuospatial attentional tasks in monkeys (O’Neill et al. 1999; 2003) (Table 5). In models of learning and memory, the mAChR agonists milameline, xanomeline, WAY-132983, and cevimeline enhanced performance in spatial and delayed nonmatching to sample radial arm maze tasks in scopolamine-impaired, cholinergic-lesioned, and aged rats (M’Harzi et al. 1995; Brandeis et al. 1990; Hodges et al. 1999; Bartolomeo et al. 2000) (Table 5). In addition, oxotremorine and RS-86 reversed disruptions in Morris water maze tasks induced by hemicholinium-3 (Hagan et al. 1989). Notable nonhuman primate studies include improved reversal learning in delayed non-matching-to-sample tasks after administration of mAChR agonists arecoline and RS-86 (Rupniak et al. 1989, 1992) (Table 5). Moreover, milameline also had effects on cortical EEG parameters consistent with enhanced arousal in monkeys (Schwarz et al. 1999), while sabcolemine and arecoline induced hippocampal rhythmical slow wave activity, a procognitive biomarker, in anesthetized rats (Loudon et al. 1997) (Table 5). Finally, a potential disease-modifying effect of mAChR agonists in AD has been revealed by clinical studies with sabcomeline and talsaclidine in which treated AD patients showed decreases in cerebrospinal fluid (CSF) levels of total Aβ or Aβ40 and Aβ42, indicative of a reduction in the pro-amyloidogenic processing of the amyloid precursor protein (Hock et al. 2000, 2003). These data are consistent with earlier studies using another mAChR agonist, AF102B (Fisher 2007). However, other studies have shown that decreased CSF Aβ42 may predict cognitive decline in AD (Motter et al. 1995; Galasko et al. 1998; Sunderland et al. 2003; Fagan et al. 2006) and, thus raise the question which amyloid fraction in CSF may be the most suitable biomarker for predicting, predicting pro-amyloidogenic processing of amyloid precursor protein in brain tissue (Motter et al. 1995; Galasko et al. 1998; Sunderland et al. 2003; Fagan et al. 2006). Future studies are needed to clarify these important issues in the AD literature. Taken together, there is a robust preclinical, and in some cases clinical, profile for the efficacy of mAChR agonists in the enhancement of different aspects of cognition. However, as discussed in the introduction, all of the mAChR orthosteric agonists described in Table 5 have failed to advance into further clinical development due to a lack of true subtype selectivity.

Fig. 4
figure 4_7

Chemical structures of representative orthosteric muscarinic agonists

Full size image
Table 4 Receptor affinities [nM]of orthosteric muscarinic receptor agonists
Full size table
Table 5 Effects of orthosteric muscarinic agonists on cognition
Full size table

Despite the overall clinical failure of mAChR orthosteric agonists, two clinical studies with the M1/M4-preferring mAChR agonist xanomeline have provided critical proof-of-concept efficacy for the reversal of cognitive impairments and behavioral disturbances observed in AD and schizophrenia patients. In a clinical trial with mild-to-moderate AD patients, xanomeline improved aspects of cognitive performance as measured by the Alzheimer’s disease assessment scale cognitive (ADAS-cog) battery, including spoken language ability, word-finding difficulty in spontaneous speech, and constructional praxis (i.e., three-dimensional motor planning and execution) (Bodick et al. 1997a, b). Xanomeline also significantly improved a number of behavioral disturbances, including agitation, vocal outbursts, and hallucinations, observed in AD patients (Bodick et al. 1997a, b). In a separate clinical trial conducted in a small group of treatment refractory schizophrenic patients, xanomeline produced a significant enhancement in verbal learning and short-term memory functions, as well as decreased positive symptoms (Shekhar et al. 2008). The dose-limiting adverse effects observed in the xanomeline treatment groups in both clinical studies, due to the nonselective activation of peripheral mAChRs, halted further development of this compound.

4.3 Allosteric Agonists and Positive Allosteric Modulators

In recent years, several groups in both academia and industry have pursued a novel strategy for the discovery of mAChR ligands that stimulate a specific receptor subtype by targeting sites that are less highly conserved than the orthosteric ACh binding site, termed allosteric sites (Fig. 5a). As discussed in the following sections, allosteric activators of mAChRs exhibit high subtype selectivity and different mechanisms of action in comparison with orthosteric mAChR agonists. For example, PAMs of mAChRs exhibit no intrinsic activity at the receptor (Fig. 5b), but can bind to an allosteric site and potentiate the effects of the endogenous ligand ACh through enhancement of the affinity of ACh for the orthosteric site and/or increased coupling efficiency to the G-proteins (Fig. 5c). In contrast, allosteric mAChR agonists bind to an allosteric site on the receptor and can directly activate the receptor in the absence of ACh (Christopoulos 2002; Waelbroeck 2003; Conn et al. 2009). Discovery of these novel allosteric mAChR activators is providing exciting tools for further characterization of the roles of different mAChRs on cognition.

Fig. 5
figure 5_7

Schematic representation of a muscarinic acetylcholine receptor showing orthosteric and putative allosteric binding sites and effector mechanisms (a). Each of the five mAChR subtypes is a seven-transmembrane protein. Allosteric activators bind to sites other than the orthosteric Ach binding site to activate or potentiate the receptor. Muscarinic receptors are divided into two functional classes based on G-protein-coupled receptor coupling. M1, M3, and M5 mAChRs couple to Gq/G11, which results in increased intracellular calcium levels via phospholipase C activation. M2 and M4 mAChRs couple to Gi/o, resulting in the inhibition of adenylyl cyclase and ion channels. Unlike orthosteric agonists, PAMs have no intrinsic activity (b). The graph in (c) illustrates two potential modes of action of PAMs in a cell-based system: affinity modulation (PAM1) with a resulting leftward shift of the concentration–response curve and efficacy modulation (PAM2) leading to an increase in maximal response. AC adenylyl cyclase; ACh acetylcholine; cAMP cyclic AMP; IP3 inositol triphosphate; M1–M5 muscarinic cholinergic receptor subtypes 1–5; PAM positive allosteric modulator; PLC phospholipase C

Full size image

4.3.1 M1 Allosteric Modulators

As shown in Fig. 6, there has been excellent progress in the identification of several M1 allosteric activators for critical proof-of-concept studies in preclinical models (see representative chemical structures for the M1 allosteric agonists and PAMs in Fig. 6 with the in vitro functional potencies at each subtype, if available, described in Table 6 and highlighted efficacy in different preclinical cognitive tasks shown in Table 7.

Fig. 6
figure 6_7

Chemical structures of representative muscarinic receptor allosteric agonists and positive allosteric modulators

Full size image
Table 6 Functional responses of allosteric muscarinic receptor modulators at cloned muscarinic receptors
Full size table
Table 7 Effects of allosteric muscarinic receptor agonists on cognition
Full size table

AC-260584 is an analog of the first-generation M1 allosteric mAChR agonist AC-42 that was shown to have activity through binding at an allosteric site on the M1 mAChR (Heinrich et al. 2009; Spalding et al. 2002; Langmead et al. 2006). AC-260584 has been reported to enhance memory functions as assessed in the novel object recognition and Morris water maze tasks in mice, as well as produce effects in preclinical models predictive of antipsychotic-like effects (Bradley et al. 2010; Vanover et al. 2008) (Table 7). Unfortunately, interpretation of the in vivo efficacy of AC-260584 is confounded by off-target effects at dopamine D2, adrenergic α1A, and serotonin 5-HT2A receptors (Heinrich et al. 2009). The M1 allosteric agonist, 77-LH-28-1, is another systemically active AC-42 analog (Langmead et al. 2008) with high selectivity for M1 and some weak M3 agonist activity (Heinrich et al. 2009) (see Fig. 6, Tables 6 and 7). Functional and site-directed mutagenesis studies have established that 77-LH-28-1not only acts as a “bi-topic” agonist that binds to a site that overlaps with the orthosteric site, but also includes an allosteric site that modulates affinity of the ACh site (Avlani et al. 2010). Several physiologic effects thought to potentiate cognition, including increased hippocampal CA1 pyramidal cell firing in vitro and in vivo and induction of synchronous network activity through increased CA3 hippocampal γ oscillations, are increased with 77-LH-28-1 treatment (Langmead et al. 2008; Buchanan et al. 2010; Jo et al. 2010; Spencer et al. 2010). Another highly selective AC-42-based compound, Lu AE51090, reversed delay-dependent memory decay in a Y-maze delayed alternation paradigm (Sams et al. 2010) (Fig. 6, Tables 6 and 7).

There are now additional second-generation, systemically active and highly selective M1 allosteric agonists and PAMs that are serving as important tools for determining the role of selective activation of M1 mAChRs in native tissue preparations and in animal models of cognition, including the M1 allosteric agonists TBPB, which is a selective and potent M1 allosteric agonist in recombinant systems (Jones et al. 2008) (Fig. 6, Tables 6 and 7). Site-directed mutagenesis studies have revealed that point mutations in the ACh binding site that reduce the activity of orthosteric mAChR agonists at M1 produce no change in the response to TBPB. A Schild analysis for the blockade of TBPB effects with the orthosteric mAChR antagonist atropine showed that TBPB interacts with the orthosteric site in a noncompetitive manner (Jones et al. 2008). Based on an allosteric ternary complex model for the actions of two molecules that interact with distinct sites on a receptor, these results collectively suggest that TBPB may act as an allosteric M1 agonist (Christopoulos and Mitchelson 1997; Jacobson et al. 2010). However, further studies are warranted as it cannot be ruled out that TBPB may act as a bi-topic agonist, similar to 77-LH-28-1 (Avlani et al. 2010). In native tissue preparations, TBPB potentiated NMDA receptor currents in CA1 hippocampal pyramindal cells, a function that is thought to contribute to the procognitive effects of mAChR agonists, as described earlier (Jones et al. 2008). In several preclinical models predictive of antipsychotic-like activity, TBPB produced efficacy at doses that do not induce the side effects associated with nonselective stimulation of peripheral mAChRs. More importantly, TBPB reversed apomorphine-induced deficits in PPI of the acoustic startle reflex and scopolamine-induced impairments in the acquisition of a hippocampal working memory task, contextual fear conditioning (Kane 2008). In addition, selective activation of M1 by TBPB increased the non-amyloidogenic processing of the amyloid precursor protein and reduced Aβ formation in vitro, as previously reported with other nonselective mAChR agonists. These data are consistent with the hypothesis that selective activation of M1 mAChRs may provide both enhancement of cognitive functions and potential disease-modifying activity for the treatment of symptoms associated with AD.

Finally, VU0357017 represents a highly potent, selective, and systemically active third-generation M1 allosteric agonist (Lebois et al. 2010) (Fig. 6, Tables 6 and 7). Unlike the other allosteric M1 agonists, VU0357017 activates the M1 mAChR at a novel allosteric site on the third extracellular loop, instead of within the seven transmembrane domain (Lebois et al. 2010). This compound potentiated NMDA receptor currents in slice electrophysiology experiments and blocked scopolamine-induced deficits in contextual fear conditioning (Lebois et al. 2010).

4.3.2 M1 Positive Allosteric Modulators

A major advance in the development of systemically active and selective M1 PAMs was the identification and characterization of benzylquinolone carboxylic acid (BQCA) (Fig. 6). In cell-based systems, BQCA is a potent PAM with a 129-fold leftward shift of the ACh concentration–response curve with high M1 selectivity that lacks agonist, potentiator, or antagonist activity at M2–M5 up to 100 μM (Ma et al. 2009) (Table 6). In addition, BQCA increases the affinity of the M1 mAChR for ACh, but does not bind at the orthosteric ACh binding site. In native tissue, BQCA increased mPFC spontaneous excitatory postsynaptic currents (sEPSCs) and potentiated carbachol-induced effects on sEPSCs frequency, and these effects were absent in M1 mAChR KO mice (Shirey et al. 2009). With in vivo electrophysiological techniques, BQCA was also shown to enhance firing rates of mPFC neurons after systemic administration (Shirey et al. 2009) (Table 7). In animal studies, BQCA reversed scopolamine-induced disruptions of the hippocampus-mediated memory task of contextual fear conditioning, increased wakefulness, decreased delta sleep, and restored deficits in mPFC-dependent discrimination reversal learning in a transgenic mouse that overexpresses a familial AD mutant form of the amyloid precursor protein (Tg2576 mice) (Ma et al. 2009; Shirey et al. 2009) (Table 7). Interestingly, BQCA also increased cortical blood flow, a process previously attributed to M5 mAChR activation based on KO studies (Yamada et al. 2001, 2003). Taken together, studies with M1 allosteric agonists and PAMs have demonstrated that selective activation of M1 produces efficacy in preclinical models of cognitive enhancement similar to the effects observed with other nonselective mAChR agonists, and indicate an important role for M1 activation in prefrontal cortex-dependent synaptic plasticity and learning.

4.3.3 M4 Positive Allosteric Modulators

There have also been recent developments in the identification of systemically active M4 PAMs, including LY2033298 and VU0152100 (Chan et al. 2008; Brady et al. 2008) (see Fig. 6 for chemical structures, and Tables 6 and 7 for in vitro properties and functional effects, respectively). LY2033298 represents a highly selective M4 PAM that robustly potentiates the response of ACh through binding at residue F186 in the third extracellular loop (o3) of the receptor (Nawaratne et al. 2010), but does not directly activate M4 mAChRs. Using rat M4 AChRs (rM4) membranes in cell-based studies, the in vitro potency of LY2033298 for potentiation of [3H]-oxotremorine-M was decreased by fivefold to sixfold in comparison with studies using human M4 AChR (hM4) membranes (hM4 EC50 = 8 nM; see Table 6). Across all in vivo models tested to date, LY2033298 had no effects when administered alone, but potentiated the effects of a subthreshold dose of the nonselective mAChR agonist oxotremorine in the inhibition of conditioned avoidance responding and reversal of apomorphine-induced disruption of the PPI (Chan et al. 2008; Leach et al. 2010; Suratman et al. 2011). The observed lower potency of LY2033298 at the rat M4 mAChR has been postulated to account for the lack of efficacy observed in animal models with the LY2033298 alone.

More recently, another highly selective, systemically active M4 mAChR PAM, VU0152100, with a 30- to 70-fold leftward shift in the ACh response was discovered (Brady et al. 2008) (Fig. 6). VU0152100 exhibits high mAChR subtype selectivity for M4 (see Table 6) relative to the other mAChRs and 15 other GPCRs that are highly expressed in the brain (Brady et al. 2008), and increases M4 mAChR receptor affinity for ACh without competing for the orthosteric ACh binding site (Brady et al. 2008). In preclinical studies, VU0152100 reversed amphetamine-induced hyperlocomotion and disruptions in the acquisition of contextual fear conditioning (Byun et al. 2011). Interestingly, these findings suggest that there is sufficient endogenous ACh tone to potentiate cholinergic responses when VU0152100 is administered alone. Although preliminary, these studies using selective M4 mAChR PAMs indicate that selective activation of M4 mAChRs produces efficacy in preclinical models predictive of antipsychosis-like activity comparable to the effects observed with xanomeline and other mAChR agonists and hint at some potential cognition enhancing effects.

5 Summary

Converging findings with subtype-selective mAChR activators and mAChR antagonists and KO mice are providing important validation for the role of the muscarinic cholinergic system in the modulation of normal cognitive functions and in the potential reversal of cognitive deficits observed in neurologic and psychiatric disorders, including AD and schizophrenia. Discovery of the novel subtype-selective mAChR ligands is also providing critical tools to better understand the relative roles of the mAChR subtypes in the different aspects of cognition and in the observed efficacy with AChEIs and orthosteric mAChR agonists. To date, selective M1 and M4 allosteric agonists and/or PAMs are providing the most promising preclinical data for the potential treatment of cognitive impairments and behavioral disturbance associated with dementia or schizophrenia.