Introduction

Whether small, isolated populations have mechanisms in place to deal with unfavourable genetic constituency is of growing interest as human activities diminish natural population sizes. Genetic theories predict that reduced population size and increased isolation can result in pronounced genetic drift and inbreeding depression (ID), ultimately reducing genetic diversity, fitness, and adaptive potential (Edmands 2002; Frankham et al. 2002; Reed and Frankham 2003; Mavárez et al. 2006; Pickup et al. 2013). For such populations, hybridization can be beneficial, as the introduction of new genetic material can counteract ID and increase adaptive potential, thus increasing mean population fitness (Edmands 2007; Pickup et al. 2013; Whiteley et al. 2015; Frankham 2015). Nevertheless, hybridization can also result in outbreeding depression (OD), a loss of fitness due to a reduction of local adaptation in the F1 or the breakdown of co-adapted gene complexes in the F2 (Edmands 1999, 2007). Due to concerns related to OD, hybridization remains a controversial management tool and is rarely used to rescue small populations (Frankham 2010; Frankham et al. 2011; Ralls et al. 2018).

To reduce OD risks, current guidelines suggest that wildlife managers should select source populations that have been isolated from the recipient populations for less than 500 years and that are adapted to similar environments (Frankham et al. 2011). In practice, this may not always be feasible as threatened species are typically isolated and locally adapted. Reduced hybrid fitness is also expected when the geographic distance separating parental populations increases (Gilk et al. 2004) given that ecological and genetic dissimilarity often increases with spatial scale (Edmands 1999, 2002). Accordingly, studies focusing on hybridization between similar populations showed that hybridization effects tended to be beneficial (Frankham et al. 2011; Whiteley et al. 2015). Comparatively, hybridization between geographically, genetically and/or adaptively divergent populations has received less attention (Kronenberger et al. 2017) and previous studies of hybridization between moderately or highly divergent populations resulted in variable fitness effects (Lynch 1991; Edmands 1999, 2002; Willi et al. 2006; Kronenberger et al. 2017).

Another factor thought to influence the effects of hybridization is the contemporary effective population size (Ne) of a population. Small Ne populations (i.e. < 50, Frankham et al. 2013) are expected to have reduced adaptive potential due to low additive genetic variation and the cumulative effects of inbreeding (Reed and Frankham 2003). Therefore, introducing new genetic material to small populations is often thought to be beneficial.

A third and final factor is the degree of environmental dissimilarity or stress experienced by hybrids. Under stressful conditions, the negative fitness effects of ID are expected to amplify (Keller and Waller 2002). In masking deleterious mutations associated with ID, hybridization can therefore increase fitness under environmental stress (Edmands and Deimler 2004; Edmands 2007). However, the relationship between hybrid fitness and environmental stress can also be complex, with some authors reporting hybrid vigor, hybrid break down, and OD becoming more severe as environments become more stressful (Fraser et al. 2008; Prill et al. 2014).

Few studies have investigated how hybrid fitness correlates with measures expected to influence hybridization effects, such as population Ne, geographic distance and genetic divergence between hybridized populations, and environmental stress (Edmands 2002; Ramachandran et al. 2005; Fraser et al. 2010; Pierce et al. 2017; Lemmon and Juenger 2017). As temperature and pH control metabolism and life-history traits (e.g. length, weight, growth rate, survival) of fishes (Beacham and Murray 1985; Marten 1992; Baker et al. 1996), they are particularly relevant parameters to consider when investigating the consequences of environmental dissimilarity on hybrid fitness.

Remarkable population diversity is found within salmonids such as brook trout (Salvelinus fontinalis), and many native populations are experiencing unprecedented population declines (Flebbe et al. 2006; Hudy et al. 2008; Kanno et al. 2011; Wenger et al. 2011). Due to their propensity to occupy diverse habitats and return to their natal areas to spawn (Stabell 1984), they often form genetically distinct, locally adapted populations (Fraser et al. 2011). This, along with their socio-economic, cultural, and keystone species importance, has made them the focus of many breeding and supplementation programs (Araki et al. 2007, 2008; Houde et al. 2011). As wild populations are becoming increasingly small and fragmented, maintaining population distinctiveness and viability is desired by avoiding ID via inter-population hybridization, whilst avoiding OD and the reduction of local adaptation (Araki et al. 2007, 2008; Houde et al. 2010; Fraser et al. 2010; Houde et al. 2011; Robinson et al. 2017).

In a common garden experiment, we investigated the effects of geographic distance and genetic divergence between populations, population size, and environmental dissimilarity on F1 hybridization outcomes among crosses generated from eight isolated brook trout populations from Cape Race, Newfoundland, Canada (Supplementary Fig. A1). Although several of the populations have low genetic diversity and population sizes that are too small to prevent ID (Ne< 50) (Bernos and Fraser 2016), Cape Race populations would not typically be considered for genetic rescue attempts. The populations have persisted naturally since becoming isolated from one another approximately 10,000 years ago (Danzmann et al. 1998; Wood et al. 2014) and remain temporally stable in their genetic population structure (Wood et al. 2014; Bernos and Fraser 2016). In fact, according to the existing guidelines on genetic rescue (Frankham 2015), hybridization between Cape Race populations would likely generate OD as they have been isolated for much more than 500 years and are likely adapted to different environmental conditions. Indeed, multi-year habitat data shows that the streams vary greatly in environmental conditions (Wood et al. 2015), including temperature and pH (Online Appendix A: Table A1). Moreover, the populations vary in morphology, life history, molecular loci linked to functional traits, and quantitative genetic differentiation (QST) in a manner consistent with an adaptive basis (Supplementary Table A2) (Hutchings 1996; Fraser et al. 2014; Wood et al. 2015; Wells et al. 2016; Zastavniouk et al. 2017; Fraser et al. 2019).

Based on the theoretical predictions discussed, hybridization should decrease fitness in Cape Race brook trout populations in all but perhaps the smallest populations. Frequently, individual fitness measurements are pooled in studies to estimate the population-level effects of hybridization, without regard to family-level variation. This pooling assumes that fitness measurements are independent among individuals and that hybridization effects are constant across families. However, numerous studies have demonstrated that the extent of inbreeding depression can vary between family lines within populations (reviewed in Kelly 2005) and that maternal effects can significantly alter the effects of hybridization (Houde et al. 2011; Debes et al. 2013). Therefore, in this study, we evaluated hybridization effects at both the population level and the maternal-family level within populations. Specifically, we expected that relative hybrid fitness (i.e. hybrid versus non-hybrid maternal-family and population levels) would (i) be comparatively greater when outcrossing small populations than large populations; (ii) be greater when genetic divergence (neutral or quantitative) and geographic distance between hybridized populations were smaller; and (iii) increase as environmental dissimilarity (temperature, pH) between the laboratory and wild conditions increased, assuming that hybridization masks deleterious mutations associated with ID under environmental stress.

Materials and methods

Study site

Cape Race is dominated by several low-order streams (0.27–8.10 km in length) inhabited primarily by brook trout. The present study focused on eight populations (Supplementary Fig. A1) varying in Ne and population size (Ne = 44–589; adult census population sizes N = 202–7801). They are isolated, genetically distinct, and have a low genetic diversity at neutral markers (Bernos and Fraser 2016). Of those eight populations, three are likely to experience inbreeding depression (Ne < 50) and two are too small to prevent long-term fitness loss according to existing guidelines (Ne< 100, Frankham et al. 2014). There is no trout stocking at Cape Race, negligible fishing due to the small size of trout (typically 10–15 cm in length) and low human density (Hutchings 1996; Wood et al. 2014). Multi-year data collection showed that the streams vary widely in water velocity, temperature, pH, dissolved oxygen, depth and vegetation cover (Supplementary Table A1; Wood et al. 2014; Bernos and Fraser 2016).

Gamete collection, wild

In October 2014, gametes were collected from seven Cape Race populations: Bob’s Cove (BC), Coquita (CO), Freshwater (FW), Ouananiche Beck (OB), Still There By Chance (STBC), Whale Cove (WC), and Watern Cove (WN). Individuals were collected from previously documented spawning sites (Wood et al. 2014). Backpack electrofishing was used to collect fish, which were then inspected for signs of readiness: a release of sperm for males, and an elongated cloaca/soft belly for females. Readiness was assessed in the days leading up to the expected date of gamete collection and ready fish were held for 24–72 h in flow-through cages before collection. Gamete collection took place between 19h00 and 1h00. Sperm was collected in 1.5 ml micro-centrifuge tubes while eggs were collected in 60 ml opaque plastic containers. Gametes were kept on ice and transported to St. John’s, Newfoundland immediately after collection. They were flown directly to Montreal and crossed at Concordia University within 15 h from the start of gamete collection; shipments occurred on Oct 16th, 21st, and 26th.

Gamete collection, captive

In November/December 2014, gametes were also collected from captive-born individuals produced from wild parents from five Cape Race populations (BC, CC, FW, OB, and WN). These captive-bred individuals were used to bolster the number of crosses produced and included an additional population (CC) which we were unable to obtain enough individuals from the wild in 2014 to produce hybrid crosses. Fish from each population were sampled daily to check for signs of “readiness” (above) once the water temperature reached 7 °C. If enough males and females in two or more populations showed signs of readiness, gametes were immediately collected and crossed on the same day. Fish were maintained in captivity under common environmental conditions for one generation (details in Fraser et al. 2019). They were individually tagged to keep track of the family lines and to avoid crosses between siblings. Crossing events occurred on November 19th, 26th, 28th, and December 4th and 12th. Gamete collection and general protocol followed as above.

Common garden and breeding design

A total of 19 F1 cross-types were conducted (eight non-hybrid and 11 hybrid cross-types), with 265 non-hybrid families and 130 hybrid families across populations (Table 1). We applied a standard, partially crossed North Carolina II design, meaning that most dams were mated to multiple sires and sires to multiple dams to generate several non-hybrid and hybrid families (Lynch and Walsh 1998). Crosses generated from wild or captive trout were reared under a constant, warm temperature regime (7 °C) from October, 2014–April, 2015 that mimicked temperature stress in most of the native streams (Supplementary Table A1; Fig. A2; Wood and Fraser 2015) and more basic pH (7.0). Crosses were randomly assigned to a 5.2 cm diameter flow-through tube within a 1000 L recirculating tank. Most same-dam families were replicated across at least two tanks (70%). Some same-dam families were replicated within the same tank due to limited tank space (9%); remaining same-dam families could not be replicated due to limited gamete quantity. Eggs were left undisturbed until the eyed stage at which time dead eggs were removed daily to reduce fungal Infections. Individual-level variables estimated from each offspring were total length at hatch (± 0.01 mm) and total length at full yolk sac absorption (± 0.01 mm; Beacham and Murray 1985; Einum and Fleming 2000; Araki et al. 2008). For each family, we also estimated overall survival to hatch (number of hatchlings divided by number of eggs).

Table 1 Total number of non-hybrid and hybrid families generated using eight brook trout populations from Cape Race, Newfoundland, Canada for three fitness-related traits (survival to hatch, length at hatch and length at yolk absorption)
Full size table

Predictors of hybridization effects

Populations might be more likely to experience outbreeding depression as the degree of genetic divergence increases between hybridized populations. Three pairwise quantitative genetic differentiation estimates (QST) were taken from Wood et al. (2015) for length at hatch, length post yolk sac absorption, and an average from 15 morphological, life history and behavioural traits as a proxy for survival (Supplementary Table A2). QST estimates were available for all but one population (CO). Although QST is more strongly tied to adaptive divergence (McClelland and Naish 2007; Leinonen et al. 2013), neutral divergence can also contribute to OD risks (Frankham et al. 2011). We therefore also used neutral genetic differentiation estimates (FST) from Fraser et al. (2014) as a predictor of variability in hybridization effects in a supplementary analysis at the family level (Online Appendix B).

As environmental difference might increase with geographic distance, populations might also be more likely to experience outbreeding depression as geographic distance increases between hybridized populations. The distance between streams was calculated using geographical coordinates of stream mouths which, once inputted to the “distVincentyEllipsoid” function in the “geosphere” package (Hijmans et al. 2015), calculated the shortest distance (greatest circle distance) between two stream mouths (Supplementary Table A2).

Populations with a small Ne might benefit more from hybridization than those with a large Ne as the former are more likely to experience ID. We used the effective number of breeders (Nb, the effective number of reproducing individuals in a single breeding cohort) as an analogue of contemporary Ne because the two parameters are strongly correlated in Cape Race populations (Bernos and Fraser 2016) (Supplementary Table A1). As preliminary investigations revealed that Nb and Ho were not correlated in this study (p = 0.43), we also conducted the analyses with Ho instead of Nb at the family level (Online Appendix C).

We expect hybridization benefits to decrease as the degree of environmental dissimilarity between the wild and the laboratory increases. Environmental dissimilarity was measured by examining the degree of change between wild streams and in-lab temperature and pH (temperature = 7.0 °C, pH 7.0). For Cape Race populations, 7 °C was a thermal stress at the incubation stage (Wood and Fraser 2015), while a pH of 7.0 was higher than all mean spawning-period pH measurements (Supplementary Table A1). Each population’s peak spawning date was calculated using data acquired in that year’s spawning ground assays (Wood and Fraser 2015). From these unique starting dates, using previously documented data on the number of degree days to hatch for each population (Wood and Fraser 2015), the average temperature and pH was taken for each stream over the incubation stage in the wild and compared to the constant temperature and pH experienced by crosses in the lab setting.

Statistical analyses

Consistency of hybridization effects

We used linear general and generalized mixed-effect models (LMM and GLMM, respectively) to test whether traits (individual length at hatch and length at yolk sac absorption, family survival to hatch) differed between non-hybrid and their hybrid cross-types. For each of the eight non-hybrid cross-types, we fitted LMMs including individual length at hatch and at yolk sac absorption (respectively) as a response variable and cross-type as a fixed effect. We specified a random intercept for family nested within dam to account for maternal-family effects (i.e. dams were mated to multiple sires and individual length was measured for each offspring). We also included a fixed effect for tank to account for potential covariance, except for the BC models because all hybrids were contained within one tank (BC). We evaluated differences in survival to hatch between hybrids and non-hybrids using GLMMs with a binomial distribution and a logit link function. Model structure was as previously described. To test for significant differences amongst cross-types, we compared the models with the fixed effect to a reduced model using Wald F-tests based on the Kenward–Roger method (Kenward and Roger 1997) for LMMs, and likelihood ratio tests for GLMMs. We also used marginal and conditional R2 to evaluate the variance explained by cross-type and maternal effects in all the models (Nakagawa and Schielzeth 2013). Statistical analyses were conducted with the lme4 package (Bates et al. 2015) in R (R Development Core Team 2008). When significant differences amongst groups were found, we used lsmeans (Lenth 2015) to conduct multiple comparisons among cross-types using a Bonferroni correction.

Family-level analysis of variables influencing the effects of hybridization

At the family level, we used LMMs to evaluate the effect of maternal Nb, QST or geographic distance (km) between hybridized populations and environmental dissimilarity on the fitness traits for hybrids relative to their non-hybrid crosses. Environmental dissimilarity was the difference in incubation temperature or pH between the maternal stream and lab environment. Domestication selection is common in captive environments and can occur after the first generation of captivity (Fraser et al. 2019); such adaptation might subsequently affect fitness in hybrid crosses. We investigated potential differences in relative fitness between hybrids generated from wild or captive mothers by including origin (wild or captive) as a fixed effect in the models. We ran similar models with FST instead of QST (see supplementary materials). Maternal-family level relative hybrid fitness metrics were estimated from the difference in lengths between hybrid and non-hybrid individuals from the same dam (Kelly 2005), and the ratio of survival to hatch between hybrid and non-hybrid families from the same dam. The assumption here was that at early life-history stages, larger fish are more likely to outcompete smaller fishes for better resources such as higher quality food and habitat or survive their first winter (Quinn and Peterson 1996; Bashey 2008; Huss et al. 2008; Pess et al. 2011). Standardizing at the maternal-family level allowed us to quantify hybridization effects at the family level while simultaneously controlling for maternal effects.

As log-transformed relative risks approximate the normal distribution (Oehlert 1992), we used log-transformed relative hybrid survival and untransformed relative hybrid lengths as response variables in separate models. In all models, the explanatory variables were centered and scaled prior to the analysis and we included a random intercept by non-hybrid family of reference (dams often produced multiple non-hybrid and hybrid families, and hybrid families were compared to each of the non-hybrids) to account for covariance in our dataset. Model selection was as previously described. Statistical analyses were conducted with the lme4 package (Bates et al. 2015) in R (R Development Core Team 2008). Variance inflation factor analyses were conducted in all models to check for multicollinearity between explanatory variables. As relative length and survival were averaged within each family (pooled across tanks), tanks were not included in the models. There were two hybrid families and one non-hybrid family (CC-WN, FW-WN, and WN non-hybrid) where no eggs survived to hatch: these were excluded from the analyses as the resulting relative survival ratios were meaningless.

Population-level analysis of variables influencing the effects of hybridization

We also evaluated the effect of Nb, QST, geographic distance, and environmental dissimilarity between hybridized populations on relative hybrid fitness at the population-level. As elsewhere (Frankham et al. 2011), we quantified relative hybrid fitness as the ratio between each hybrid/non-hybrid population for each of the fitness traits (i.e. BCOB hybrid cross-type would be compared to non-hybrid BC and OB). Log-transformed relative hybrid fitness was used as a response variable. QST or FST (respectively), geographic distance, Nb, and environmental dissimilarity were centered, scaled, and used as fixed effects in two different models. As a proxy for total fitness, we first used relative hybrid fitness for all traits as a response variable in a LMM and included a categorical fixed effect for trait to account for basal differences. None of the predictors explained the variability in total relative fitness (results not shown). There was, however, significant variation in relative fitness amongst traits: relative survival to hatch was significantly different from relative length at hatch and yolk sac absorption (Supplementary Table A3). We therefore used log-transformed relative length metrics and survival as response variables in two separate models (length at hatch and yolk sac absorption were combined). Although we initially included a random intercept by non-hybrid cross-type of reference to account for covariance in both models, it was subsequently dropped from the relative length model (and kept in the survival model) as it led to singularity. Model selection was as previously described.

Results

Consistency of hybridization effects

When taking maternal effects into account (family-level analyses), there was no significant evidence for outbreeding depression, and other hybridization effects (beneficial or neutral) were inconsistent among traits. We found significant differences in mean fitness traits between cross-types for 7 of the 24 LMMs and GLMMs (i.e. eight cross-types × three traits; Fig. 1, Supplementary Table A4; final model estimates not presented). Of the 66 pairwise comparisons between hybrid and non-hybrid cross-types (i.e. twenty-two hybrid/non-hybrid pairs × three traits), six pairwise comparisons, spread across four populations and the three traits, showed support for the occurrence of heterosis; the remaining 60 non-hybrid vs. hybrid pairwise comparisons did not significantly differ in fitness traits (Supplementary Table A5). There was also some variability in fitness traits between hybrids at 9 of the pairwise comparisons between hybrid populations. In all models, the proportion of the variance cumulatively explained by the fixed effect (tank and cross-type) was minor (median = 4.0%, range < 1–27%) relative to the variance explained by the random maternal effects only (median = 52.5%, range 17–72%) (Supplementary Table A6). When maternal effects were unaccounted for and fitness was pooled across individuals within cross-types (population-level analyses) (Supplementary Table A7), cross-types tended to benefit from hybridization at two traits (length at hatch and yolk sac absorption); survival to hatch, however, was lower in hybrids. The effects of hybridization were not consistent across traits within cross-types.

Fig. 1
figure 1

Length at hatch, length at yolk sac absorption, and survival to hatch for 11 hybrid and 8 non-hybrid cross-types for brook trout. Dots represent individual measurements, boxplots indicate means and standard deviations, and whiskers show the range. Significant pairwise contrasts showing evidence for heterosis are indicated by stars and dotted lines. Non-hybrid cross-types including fish of captive origin (Table 1 for proportions) are bolded and italicised

Full size image

Predictors of hybridization effects at the maternal-family level

The best-fit model describing relative length at hatch at the family level included maternal Nb, pH and temperature dissimilarity, and QST of the hybridized populations (Table 2, A8; Fig. 2). The best-fit model describing relative survival to hatch included maternal Nb, pH and temperature dissimilarity, and origin of the population. None of the predictors were significant in the best-fit model describing relative length at yolk sac absorption. For relative length at hatch and yolk sac absorption, most of the variance in hybridization effects could be explained by the non-hybrid family of reference (45.3% and 85%, respectively); the fixed effects, cumulatively, explained little variance (6.2%) in the relative length at hatch model. For survival to hatch, most of the variance in hybridization effects remained unexplained; the cumulative fixed effects and the non-hybrid family of reference respectively explained 6.4% and 5.8% of the variance. Supplementary analysis with Ho instead of Nb revealed that Ho was not a better predictor of relative hybrid fitness (Online Appendix B). Indeed, while relative length at hatch tended to be greater when outcrossing more heterozygous populations, Ho explained very little variance in hybridization outcomes (< 2.2%) and was not significant in the other models. In the supplementary analysis including the extra population (CO) and FST instead of QST, we found that FST had no significant effect on fitness (Online Appendix C).

Table 2 General mixed-effect modeling of various predictors of relative hybrid fitness in brook trout for three, fitness-related early life history traits
Full size table
Fig. 2
figure 2

Significant relationships estimated from general mixed-effect models between some of the explanatory variables on a relative length at hatch and b relative survival to hatch for hybrid compared to non-hybrid maternal families. Remaining comparisons (10 of 18) of life history trait expression between hybrid and non-hybrid families versus explanatory variables were not significantly correlated and are therefore not represented. Positive relative fitness values indicate that hybrids performed better than non-hybrid maternal families. Negative temperature or pH dissimilarity indicate that laboratory conditions were warmer and more basic than in wild maternal streams

Full size image

As expected, relative length at hatch was greatest when outcrossing small Nb populations (Fig. 2). Conversely, relative survival to hatch was lower when outcrossing small Nb than large Nb populations, while maternal Nb had no significant effect on relative length at yolk sac absorption. Relative length at hatch also increased as QST increased, although QST estimates were not that varied across populations and did not have a significant effect on fitness in the other models. Relative length at hatch also decreased as temperature and pH dissimilarity increased—however, relative survival to hatch increased as temperature and pH dissimilarity increased. Finally, relative survival was lower when hybrids were generated from a captive compared to a wild female (Fig. 2).

Predictors of hybridization effects at the population level

At the population level, the best-fit model describing relative lengths (hatch and yolk sac absorption combined) included hybridized populations’ QST, Nb and pH dissimilarity, and explained 44.1% of the variance in relative hybrid length (Table 3, A9; Fig. 3). The best-fit model describing survival to hatch included Nb and temperature dissimilarity: the fixed and random effects explained, respectively 5.3% and 87.6% of the variance in relative hybrid survival. In contrast to our predictions and the results of the family-level analysis, relative hybrid fitness was greater when outcrossing large Nb populations in both length and survival models; relative hybrid length decreased as temperature dissimilarity between the lab and the wild population decreased, and relative survival decreased as pH dissimilarity decreased (Fig. 3).

Table 3 Linear and general linear mixed-effect modeling of various predictors of relative hybrid fitness at the population-level for, respectively, length (at hatch and yolk sac absorption combined) and survival
Full size table
Fig. 3
figure 3

Significant relationships estimated from linear and general linear mixed-effect models between some of the explanatory variables on a relative length (hatch and yolk sac absorption combined) and b relative survival to hatch for hybrid versus non-hybrid population crosses. Remaining comparisons (7 of 12) of life history trait expression between hybrid and non-hybrid populations versus explanatory variables were not significantly correlated and are therefore not represented. Positive relative fitness values indicate that hybrids performed better than non-hybrid maternal families. Negative temperature or pH dissimilarity indicate that laboratory conditions were warmer and more basic than in wild maternal streams

Full size image

Discussion

Our objective was to evaluate how the fitness effects of hybridization vary between hybrid and non-hybrid crosses using eight isolated, highly divergent brook trout populations raised in a common garden experiment. A salient feature of our work is that, despite the long-term isolation (> 10,000 years) and different ecological conditions experienced by Cape Race brook trout, we found no consistent evidence for outbreeding depression at fitness-related life-history traits. However, the effects of hybridization were influenced by maternal-family effects that needed to be incorporated in our analyses to evaluate potential differences in population-level and environmentally-dependent hybridization effects.

Hybridization generally had neutral effects on three early life history fitness-related traits. These results are surprising given that recent reviews suggested high OD risks if populations are isolated for more than 500 years and inhabit different environments (Frankham et al. 2011). Conversely, three of the studied populations might have been expected to benefit more from outbreeding in having Ne of ~ 50 or slightly less (e.g. STBC, CO, WC; Bernos and Fraser 2016). While there were some indications of small Ne populations benefitting from hybridization, signs of hybrid vigor were generally weak as we only detected significant fitness gains for one of those populations (WC) at one trait. Consistent with several other studies on Cape Race brook trout, these results suggest that the trout populations might be less susceptible to inbreeding depression, presumably owing to purging, the selective removal of deleterious alleles when in a homozygous state (Crnokrak and Barrett 2002; Leberg and Firmin 2008). For instance, despite having lower neutral genetic diversity (Wood et al. 2014; Fraser et al. 2014), small Cape Race brook trout populations retain similar quantitative genetic variation and trait differentiation (VA, QST; Wood et al. 2015), responses to acute thermal stress (Wells et al. 2016), and phenotypic plasticity (Wood and Fraser 2015) relative to large populations.

It is also clear from the results of our study that the ability to detect heterosis or outbreeding depression will be influenced by the traits studied. When maternal-family effects were accounted for, some of the variance (6.2–6.4%) in relative hybrid fitness was cumulatively explained by the factors assessed for two of the traits, length at hatch and survival to hatch; however, trends were inconsistent. For instance, hybrids tended to be the largest at hatch when dams came from a small Nb population. This is consistent with heterosis and could be due to the alleviation of genetic load by masking deleterious alleles or increasing heterozygosity (Charlesworth and Willis 2009). On the contrary, survival to hatch tended to be lower when dams came from a small Nb population, suggesting that other mechanisms (e.g. genetic incompatibilities) might concurrently mediate hybridization outcomes. Similarly, relative hybrid length at hatch decreased but survival to hatch increased when environmental dissimilarity increased (i.e. when laboratory temperature and pH conditions were warmer and more basic than in maternal streams). These results imply that hybridization effects could also be environmentally-dependent (Edmands 2007; Fraser et al. 2008; Prill et al. 2014), with traits being differentially affected. Overall, we found little support for the hypothesis that increased divergence between parental populations would increase OD risks. Indeed, relative hybrid length at hatch increased with increasing genetic divergence at quantitative traits (QST) between parental populations. Furthermore, QST had no effects on relative hybrid fitness in the other models, and genetic divergence at neutral makers (FST) as well as geographic distance had no effects on relative hybrid fitness.

Inferences on the predictors of hybridization effects also changed depending on whether family-effects were considered in the models, suggesting important family-level correlations that could be overlooked. For example, we found opposite directions for the relationship between several predictors (Ne, environmental stress) and hybridization effects between the family- and population-level analyses for relative length (at hatch or yolk sac absorption). This seemingly counterintuitive result can be explained by the observation that, within a population, families respond differently to hybridization in such a way that some of them benefit from hybridization and others do not. In this study, the number of offspring produced by each dam varied widely (mean = 29.7, range 1–100) and maternal families varied in their expression of hybridization effects, leading to some females having a disproportionate influence on mean population fitness when measurements were pooled across families.

Although this could be seen as an artifact of an imbalanced dataset, it could also have important ramifications for conservation given that several species are characterized by large variance in family size in nature and that variance in reproductive success can be further increased by supplementation (Araki et al. 2007). The main implications of this result are two-fold. First, it raises the possibility that considering family-level predictors (e.g. maternal characteristics) as well as population-level predictors may help predict hybridization effects. Indeed, in our family-level models, most of the variability in hybridization effects remained unaccounted for by population-level predictors. Second, variation at the family level could also affect genetic rescue outcomes. For instance, if family-level variability in hybridization effects persist through generations, the effects of hybridization on population fitness could be buffered if some families experience OD and others heterosis.

Study caveats

Several factors may explain why the relationships between relative hybrid fitness and factors thought to influence hybridization effects were weak in our study. First, high levels of plasticity in brook trout (Wood and Fraser 2015) and their partially duplicated genome (Thorgaard et al. 1983) may buffer them from environmental stresses and fitness effects of ID and outbreeding. In plants, for example, polyploidy can reduce the effects of ID (Husband and Schemske 1997; Grindeland 2008) and the benefits of outcrossing (Pickup and Young 2008). Second, while our study is large for a vertebrate species, it was constrained to the F1 generation of hybridization at an early life history stage due to space and resource requirements. Hence our results do not provide a multi-generational perspective on the effects of hybridization; disruption of co-adapted gene complexes occurs in F2 or later generations, and F1 hybrid fitness does not always predict introgression (Edmands and Deimler 2004; Willett 2012). Similarly, relative hybrid fitness may change at later life stages; non-hybrid crosses of two populations in this study (STBC, BC) in particular have very poor lifetime survival in the same laboratory rearing environment (Fraser et al. 2019). Third, our study is unique in its examination of long-isolated populations that may have been strongly influenced by bouts of inbreeding and purging. If this is the case, we may in fact be examining a subset of the former populations which have persisted after these events. Finally, our study was conducted in a laboratory setting: other biotic and abiotic environmental factors might differentially influence fitness-related traits in the wild, and domestication selection may have influenced fitness in some of our crosses. Although the current study is limited in its ability to extrapolate its findings to the fitness of brook trout in the wild, it offers that hybrid fitness cannot, generally, be predicted from properties of the parental populations (see also Debes et al. 2013, 2014).

Conservation implications

We expected to observe a fitness decrease in outcrossing brook trout populations whose divergence started approximatively 10,000 years ago and that experience different environmental conditions. However, we found limited support for fitness losses following hybridization; when accounting for family effects, fitness effects of hybridization were mostly neutral, and some positive. From these results, it is therefore also unclear whether hybridization always benefits very small populations or populations with limited neutral genetic diversity. Cape Race populations are however unique in that their long-term persistence as small isolated populations, in stark contrast to human-driven population declines over a shorter time scale, might have allowed purging to occur. Perhaps only smaller populations, or populations isolated more recently, may consistently respond to hybridization as a means of genetic rescue (see Robinson et al. 2017). It remains, however, that our work adds to a growing body of literature suggesting that, under certain conditions, current guidelines on genetic rescue might be too conservative (Ralls et al. 2017). Furthermore, our study reiterates that predicting hybridization effects remains complex (Tallmon et al. 2004; Houde et al. 2011; Debes et al. 2013, 2014). Indeed, our results show that OD, heterosis, and neutral effects are not consistently expressed across families and individual traits and can therefore co-occur within a population, thus making predictions challenging at the population level. We believe that accounting for differences among family lines might allow researchers to formulate family-specific inferences, which has obvious implications for the management of small populations where family numbers are usually limited. Lastly, our work identifies a need to evaluate predictors at the family level to improve our ability to predict hybridization effects.