The Autism Spectrum Disorders (ASD), including Autistic Disorder, Asperger’s Disorder and Pervasive Developmental Disorder-Not Otherwise Specified (PDD-NOS) (American Psychiatric Press 1994), are characterized by difficulties in three core areas: social interaction; communication; and repetitive behaviors or circumscribed interests (Rapin 1997). Based on twin and family studies, it appears that ASD’s are largely genetically determined (Bailey et al. 1995; Bolton et al. 1994; Rutter 1999).

Given the complexity of susceptibility factors thought to underlie ASD, examining the determinants of specific behavioral and biological alterations associated with the disorder may be a productive strategy for future research. This focus on specific traits, biomarkers, endophenotypes or intermediate phenotypes should provide measures that are genetically transmitted more simply than the DSM-IV-defined disorder and may point to specific systems playing key roles in autism expression (Gottesman and Gould 2003; Happé et al. 2006; Leboyer et al. 1998; McBride et al. 1996; Skuse 2001; Szatmari et al. 2007).

Parents of individuals with ASD can share specific behavioural and cognitive characteristics with their children. Previously noted findings in parents (on a group basis) include social difficulties, communication impairments and stereotyped, rigid behaviours, as well as cognitive differences (Folstein et al. 1999; Happé 2001; Lainhart et al. 2002; Losh and Piven 2007; Murphy et al. 2000; Piven et al. 1997; Ruser et al. 2008; Stone et al. 2008). It has been suggested that these characteristics constitute a broader phenotype of autism which may represent an expression of the genetic liability for ASD. Several studies have reported biological differences in ASD parents versus controls, including increased whole blood serotonin Leboyer 1999); decreased plasma levels of reelin (Fatemi et al. 2002); diminished size of the left hippocampus (Rojas et al. 2004); altered local gray matter volumes (Peterson et al. 2006) and decreased cortical activation (Baron-Cohen et al. 2006). The biological findings are in need of replication and a structural imaging study not finding differences between parents and controls has been reported (Palmen et al. 2005).

There has been a longstanding interest in the neurotransmitter/neurohormone 5-hydroxytryptamine (5-HT) in autism as a potential intermediate phenotype. Initial interest was stimulated with the early report of increased group mean platelet 5-HT levels in autism (Schain and Freedman 1961). The basic finding has been frequently replicated and platelet hyperserotonemia is now a well-characterized trait in ASD (Anderson 2002; Mulder et al. 2004). However, the mechanism of 5-HT elevation remains unclear. Additional factors contributing to the continuing interest in 5-HT include the compound’s critical role in early neurodevelopment (Janusonis et al. 2004; Polleux and Lauder 2004; Whitaker-Azmitia 2001), the report of reduced 5-HT synthesis in the brains of children with autism (Chandana et al. 2005), and the wide use of 5-HT selective reuptake inhibitors (SSRIs) for treating symptoms associated with autism (Kolevzon et al. 2006).

The critical role of the 5-HT2 receptor in cortical function (Jakab and Goldman-Rakic 1998) provides a theoretical basis for focusing on this receptor and accumulating evidence supports the idea that this system may be altered in autism. In 1989, McBride and colleagues reported that central and peripheral 5-HT2 receptor functioning appeared reduced in autism. A diminished 5-HT2-mediated neuroendocrine response was observed and was paralleled by lower platelet 5-HT2 receptor binding and reduced 5-HT2-mediated platelet aggregation in individuals with autism (McBride et al. 1989). Platelet 5-HT2 binding alterations in relatives of individuals with autism have been reported by Cook et al. (1993); moreover, in both studies 5-HT2 receptor measures were inversely related to platelet 5-HT levels. More recently, Murphy et al. (2006) reported lower 5-HT2A receptor density centrally by in vivo SPECT imaging in eight adults with Asperger’s syndrome. It is worth noting that there have been several negative candidate gene studies of the 5-HT2A receptor gene in ASD (Cho et al. 2008; Herault et al. 1996; Veenstra-Vander-Weele et al. 2002). However, the studies have been limited in size and have not fully studied the genetic variation.

We performed a neuroimaging study examining brain 5-HT2 receptor binding density in parents of two or more children with ASD. Parents from such multiplex families are assumed to have higher genetic loading for ASD and should offer advantages when examining potential intermediate phenotypes associated with the disorder. We hypothesized that parents of probands with ASD would have diminished central 5-HT2 receptor binding density compared to controls. While there are several neuroimaging studies of parents of children with ASD (mentioned above), only the study of Murphy and colleagues (2006) focused on serotonin.

Methods

Subjects

The parents of the children with ASD were volunteers from families participating in our autism research program; controls were recruited as volunteers by advertising at the Hamilton Health Sciences Centre. Potential participants were informed of the study (approved by the Institutional Review Board of McMaster University) and, after giving signed consent, completed an unstructured interview with one of the investigators (JG). The interview was used to exclude a history of neurological and/ or psychiatric disorder during the 6 month period predating the study; subjects with histories of medication and/or substance use that could affect 5-HT levels were similarly excluded.

The “parent” group consisted of 19 parents from 11 multiplex families and was comprised of 8 females and 11 males (2 parents were Hispanic, 2 Asian and 15 Caucasian). The families were unrelated and the marriages were non-consanguineous. The control group consisted of 9 females and 8 males (1 Asian and 16 Caucasian). The mean (±SD) ages of the parent and control groups were similar (44.8 ± 6.2 and 43.6 ± 8.4 years, respectively; F = 0.22, p = 0.65) and there was no significant sex difference between the groups (Fisher’s exact test, p = 0.74).

Proband Assessment

The probands (the children of the parents) did not participate in this investigation; however, they had been diagnosed with ASD following the completion of the Autism Diagnostic Observation Schedule (ADOS) (Gotham et al. 2007) and the Autism Diagnostic Interview Revised (ADI-R) (Lord et al. 1994) with their parent(s) as informant(s). There were 23 affected children with ASD (mean age of 13.1 years, range 5–25 years). Twenty-one children met ADI-R criteria for autism and two met criteria for ASD by having a score one point below the autism ADI-R algorithm cut-off in one domain. The mean IQ of the probands based on the Leiter scale (Levine 1986) was 64.2 ± 31.1; mean adaptive behavior scores in socialization, communication and daily living based on the Vineland Scales (Sparrow et al. 1984) were between 47 and 55.

Parent and Control Psychological Assessment

The parents and controls were assessed with the Family History Interview for Developmental Disorders-shortened version (Folstein and Rutter 1991), as modified by Bolton et al. (1994) and MacLean et al. (1999). The shortened version includes those questions that probe symptoms of the BAP (Bolton et al. 1994), as well as screening questions about other psychiatric disorders. An algorithm constructs a three-factor definition of the BAP corresponding to each domain of the autistic triad. Parents with one social, communication or repetitive activities impairment are considered to have the “broad” variant of BAP and those with two types of impairment, the “narrow” variant.

Measurement of Central 5-HT2 Receptor Density

The [18F]setoperone positron emission tomography (PET) studies were performed at the Hamilton Health Sciences PET Centre between 2 and 6 pm, over a period of 3 years (1998–2001). Prior studies of animals and humans have shown that [18F]setoperone PET is a reliable method for selectively measuring 5-HT2 receptor density in the human cortex (Blin et al. 1990). Because a significant portion of [18F]setoperone in the subcortex is bound to dopamine-type-2 (D2) receptors, subcortical BP’s were not included in the analyses.

[18F]setoperone was prepared from the nitro precursor of setoperone using an adaptation of a method described by Crouzel et al. (1988). The PET imaging was performed with an ECAT 953/31 tomograph (Siemens/CTI) following a bolus injection of 185 MBq of [18F]setoperone in a manner similar to that described by Blin et al. (1990). After infusion of [18F]setoperone, images were obtained for 90 min in a sequence similar to that described by Kapur et al. (1997). Following attenuation correction using a 68Ge rotating pin, images were reconstructed with a Hanning filter at 5 mm FWHM. Regions of interest (ROIs) were drawn for 12 cortical regions in three different planes of the brain and in the cerebellum. Given the very low levels of 5-HT receptors (Pazos et al. 1987) and the very small amount of displaceable 18F-setoperone binding in the cerebellum (Blin et al. 1990), cerebellar activity was used as a reference region for determining non-specific binding and background radiation. It has been previously shown that the ratio (S/C) of cortical (S) activity to cerebellar activity (C), termed the binding potential (BP), is an appropriate index for determining [18F]setoperone binding in the brain (Kapur et al. 1997). Furthermore, Petit-Tabou et al. (1996) have shown that when [18F]setoperone reaches a state of pseudo-equilibrium between 60 and 90 min post injection, the cortex-to-cerebellum ratio is highly correlated with cortical binding density (Bmax/Kd ratio). Cerebellar counts (controlled for age and sex) did not differ significantly across the groups studied (parents 1,916 ± 845, controls 1,427 ± 1,105; t(1) = −1.42, p = 0.34).

The PET system was calibrated approximately every 6 months with variations of around 10% noted. Intra-class correlation coefficients (ICCs) were derived for four scans (two parent and two control) to determine test-retest (1 month apart) and inter-rater reliability for the ROI method of determining BP. ICCs were also calculated for the 12 individual cortical ROIs and for the composite cortical BP (i.e. ROIs combined). Calculated test–retest ICCs for the regional BP’s ranged from 0.80 to 0.98, while an ICC of 0.91 was observed for the composite cortical 5-HT2 BP. The inter-rater ICCs for the regional BP’s ranged from 0.75 to 0.97; the composite BP ICC was 0.92. Cases and controls were intermingled and studied over the same time period in an effort to minimize effects of drift.

Measurement of Whole Blood Serotonin (5-HT)

Blood specimens obtained for the determination of platelet 5-HT were drawn into tubes containing EDTA anticoagulant and initially kept at room temperature. Levels of serotonin are stable for up to 24 h in whole blood samples kept at room temperature (Danaceau et al. 2003). Portions were removed within 1–3 h and stored at −70°C until analyzed. Whole blood 5-HT levels (ng/ml) were determined by high-performance liquid chromatography (HPLC) with fluorometric detection as previously described, with within-assay and assay-to-assay coefficients of variation of 4.6% and 5.9%, respectively (Anderson et al. 1987; Epperson et al. 2001). As over 99% of blood 5-HT is found within the platelet, whole blood measurements reflect the platelet pool of 5-HT.

Statistical Analysis

We used a hierarchical approach to the statistical analysis. In the first set of planned comparisons, individual regional cortical BP’s were combined to obtain a single or composite (mean) measure for the cortex. The composite BP was compared between the parent and control groups; individual regional cortical BP’s were then compared across groups. Due to previously reported age-related changes in 5-HT2 binding (Arranz et al. 1993; Meltzer et al. 1999), marked sex differences in ASD occurrence (Baron-Cohen et al. 2005), and suggested sex differences in human brain 5-HT synthesis (Young et al. 1999), both age and sex were used as covariates in all analyses. However, it should be noted that age and the male-to-female ratio did not differ substantially or significantly between parents and controls (see above). Interactions terms were also added to the model, testing the interaction between age and group and sex and group on BP’s. Relationships between 5-HT2 BP and platelet 5-HT levels were tested by Pearson correlation.

Results

The mean (±SD) parent composite cortical 5-HT2 BP was 2.13 ± 0.34 compared to 2.65 ± 0.75 in the controls (see Fig. 1). Controlling for age and sex, this difference was significant [F(1df) = 6.90, p = 0.013], with a large effect size (0.9; Cohen 1988). Neither of the interaction terms was significant. We observed a trend-level negative correlation (moderate) between cortical 5-HT2 BP and age in the parent group (r = −0.42; p = 0.07), but not in the controls (r = −0.26; p = 0.31).

Fig. 1
figure 1

Comparison of composite cortical 5-HT2 receptor BP in parents (N = 19) of children with autism and controls (N = 17). Group means are indicated by the horizontal bar and the standard deviations by brackets. The group mean composite 5-HT2 BP was significantly lower in the parent group: 2.13 ± 0.34 versus 2.65 ± 0.75; [F(1df) = 7.16, p = 0.010]

Full size image

We next examined the BPs of specific cortical ROIs using ANCOVA, again with age and sex as covariates. These analyses showed that 5-HT2 BP’s were significantly diminished in all cortical regions in parents as compared to controls—based on a nominal p-value of 0.05 (see Table 1). Overall, the individual cortical ROI BP’s of the parents ranged from 77% to 88% of those seen in the controls. There was no effect of sex on BP and no sex by group or age by group interaction for any of the cortical regions.

Table 1 Cortical 5-HT2 binding potentials (BPsa) in cortical regions of controls and parents of ASD offspring
Full size table

There was no significant difference between the platelet 5-HT level of the parents (129 ± 47 ng/ml) and the controls (150 ± 47 ng/ml) (F = 1.87; 2df; p = 0.13). The mean platelet 5-HT level of the parents was negatively correlated with their composite cortical 5-HT2 BP (r = −0.59; p = 0.02). In addition, in eight of the 12 cortical regions the 5-HT2 BP versus platelet 5-HT correlation reached a significance level of 0.05 or less (see Table 2). The calculated r values ranged from −0.52 to −0.65, with the highest correlations seen in the right inferior frontal (r = −0.65, p = 0.01) and right inferior parietal lobes (r = −0.64, p = 0.01). Platelet 5-HT level was not correlated with mean cortical 5-HT2 BP (r = −0.08, p = 0.77) nor with any of the individual cortical regional 5-HT2 BP’s in the controls.

Table 2 Partial correlations for regional 5HT2 BPs and platelet 5-HT (after controlling for age and gender) in parents and controls
Full size table

The Family History Interview indicated that 5 of the 19 parents (26%) met criteria for the broader autism phenotype (+BAP; all due to significant difficulty with social interaction). When parents were divided into +BAP and −BAP groups, no significant difference in 5-HT2 BP was observed (composite cortical BP 2.08 ± 0.57 and 2.15 ± 0.25, respectively; p = 0.32]. This comparison is presented for the sake of completeness and is limited by the small number of +BAP parents.

Discussion

We have presented evidence of lower 5-HT2 receptor density throughout the cerebral cortex of parents with two or more children with ASD—a group presumed to have relatively high genetic loading for ASD. This finding is consistent with a previous report of lower central 5-HT2 responsivity in autism (McBride et al. 1989) and with a recent neuroimaging study that observed lower cortical 5-HT2A receptor density in young adults with Asperger’s syndrome using a different imaging methodology (single photon emission computed tomography) and an alternative 5-HT2A receptor ligand (123I-R91150) (Murphy et al. 2006). It is notable that a neuroendocrine challenge paradigm (McBride et al. 1989) and two neuroimaging studies (Murphy 2006 and ourselves), as well as a study of postmortem brain tissue (G. Blatt, personal communication, 2007), have provided converging evidence of lower central 5-HT2 receptor expression or function associated with autism.

In our study, the 5-HT2 BPs measured in the cortical regions of parents were negatively correlated with platelet 5-HT levels. This inverse relationship is consistent with the reported negative correlation of platelet 5-HT levels and 5-HT2-mediated neuroendocrine response in autism (McBride et al. 1989) and with inverse correlations for platelet 5-HT level and platelet 5-HT2 receptor density in individuals with autism or their relatives (Cook et al. 1993; McBride et al. 1989). While the correlations between platelet 5-HT levels and central or platelet 5-HT2 receptor density are intriguing, the nature of this relationship has yet to be determined. Furthermore, we did not show global differences of platelet 5-HT levels between parents and controls. Notwithstanding these limitations, the inverse relationships observed by us and others do suggest the possibility that some 5-HT2 receptor-related factor may play a role in the platelet hyperserotonemia of autism.

The mechanism of the lower 5-HT2 receptor density in ASD family members is unknown. It is well known that brain 5-HT2 receptors do not follow the classical receptor regulation model: both 5-HT antagonists and agonists have been associated with down-regulation of 5-HT2 receptors, and 5-HT2 receptors do not up-regulate following serotonergic denervation (Eison and Mullins 1996). It should be noted that lower cortical 5-HT2 receptor expression has been reported in schizophrenia (Burnet et al. 1996; Laruelle et al. 1993; Lewis et al. 1999), bipolar disorder (Lopez-Figueroa et al. 2004), depression (Meyer et al. 1999; Minton et al. 2003), and after antidepressant treatment (Yatham et al. 1999). Higher levels have been reported in suicide victims (Mann et al. 2001), in recovered depressed patients (Bhagwagar et al. 2006) and after estrogen treatment (Kugaya et al. 2003).

It is unclear whether the apparent lower 5-HT2 receptor density in cortical regions of parents of children with ASD represents a primary underlying etiological process or whether it represents a secondary phenomenon—as compensation for another more fundamental abnormality in the 5-HT system or elsewhere in the brain. The observed alterations in cortical regions do tend to focus attention on factors involved in cortical development. A longstanding interest in 5-HT’s role in cortical development and in neurodevelopmental disorders has recently increased as patterns of early serotonergic innervation of the cortex have been elaborated (Bonnin et al. 2007; Janusonis et al. 2004). The pattern of decreased variance of 5HT2 BP in parents as compared to controls is intriguing; however, a consideration of the possible causes of differences in the observed variances would be premature in the absence of replication.

The findings of this study provide preliminary evidence of decreased cortical 5HT2 expression in parents of children with ASD and it can be speculated that this might be contributing to previously noted social difficulties in these individuals—particularly in those parents with the BAP. Focusing on the 5-HT system and especially on the 5-HT2 receptor might provide a meaningful intermediate phenotype for biological and genetic studies, and may lead to improved nosology for ASD. Identification of the clinical correlates related to diminished 5HT2 function could lead to greater understanding and improved treatment of ASD and the BAP.

We acknowledge several limitations with the current study. First, the region of interest (ROI) methodology used to define [18F]setoperone activity only included specific predefined regions in the analysis; furthermore, MRI co-registration could have been useful for ROI definition. Second, the S/C ratio (BP) is an indirect estimate of receptor density that is linearly proportional to Bmax/Kd (see Kapur et al. 1997); therefore, the possibility of changes in BP having arisen from changes in Kd (receptor affinity) cannot be ruled out. Third, familial aspects of 5-HT2 receptor expression were not directly examined, as the children (of the parents) did not participate in this study. Fourth, using a more dimensional measure of the broader phenotype—such as, the social responsiveness scale (SRS; Constantino et al. 2003) or the Broader Autism Phenotype Questionnaire (BAPQ; Hurley et al. 2007)—might have been more revealing in correlating phenotype and 5HT2 BP (Duvall et al. 2007). Finally, the requirement of studying drug-free and disorder-free parents (at least by recent history) required us to select an inherently non-representative sample.