Introduction

One wonders whether Gene Glass had the slightest idea of what scientific impacts meta-analysis would eventually have, when this educational psychologist coined the term ‘meta-analysis’ by formalizing a statistical methodology for combining the results on differences between treatment and control groups across studies (Glass 1976). The term ‘meta-analysis’ is now commonly used to refer to a range of statistical procedures, which synthesize results from different studies on the same or similar topics via standardized effect size statistics such as r, d and odds ratio (Hedges and Olkin 1985; see next section for what meta-analysis actually is). Such procedures are now routinely used not only in social, medical and clinical sciences, where meta-analysis flourished initially (Egger et al. 2001; Cooper et al. 2009), but also in biological and environmental sciences encompassing different areas of ecology, evolution and conservation (Koricheva et al. 2013).

Arnqvist and Wooster (1995) wrote the influential review outlining the roles and potentials of meta-analysis in ecology and evolution. According to their review, Jarvinen’s (1991) work, which investigated the effect of female age on laying date and clutch size in two bird species, was the first ‘evolutionary ecology’ meta-analysis (but see an attempt to synthesize selection coefficients in the book by Endler 1986). After more than two decades of a growing meta-analytic culture in our fields, it seems the time is ripe to reflect on how meta-analysis has contributed to different areas of ecology and evolution. Accordingly, we have organised this special issue of Evolutionary Ecology, titled “Meta-analytic insights into evolutionary ecology”. The special issue comprises nine contributed papers covering various topics in relation to meta-analysis: (1) natural selection (Kingsolver et al. 2012), (2) sexual selection (Jennions et al. 2012), (3) community ecology (Cadotte et al. 2012), (4) host-parasite interactions (Poulin and Forbes 2012), (5) plant evolutionary ecology (Castellanos and Verdú 2012), (6) social behaviour (Majolo et al. 2012), (7) behavioural syndromes (Garamszegi et al. 2012), (8) conservation biology (Côté and Reynolds 2012), and (9) methodological advances (Nakagawa and Santos 2012). In addition to these contributions, the special issue ends with a short note announcing the launch of an online repository of meta-analytic data (Garamszegi and Nunn 2012). Although our coverage of topics is far from exhaustive, the included topics, we believe, are representative examples. More importantly, these contributions elucidate the roles of meta-analysis in advancing knowledge of each respective area, providing an overall as well as concrete sense of what meta-analysis has done for evolutionary ecology.

In this paper, we first present contrasts between meta-analysis and traditional ‘narrative’ reviews to highlight the unique properties of meta-analysis. Then, we summarise each contribution focusing on emerging insights on both scientific discoveries and scientific practices, which two decades of meta-analyses in evolutionary ecology have produced. We finish by looking into future issues and directions facing meta-analysis, by referring especially to significant developments in research synthesis from medical and social sciences.

Meta-analysis and narrative reviews

Summarizing the current state of knowledge

Whether quantitative or qualitative, the main role of a review piece is to provide an up-to-date overview of the state of knowledge in an area of study. Qualitative reviews (i.e. narrative reviews) have dominated ecology and evolution until recently and will not be replaced for summarizing theoretical, technical and methodological developments (e.g. Nakagawa and Santos 2012). However, a steady increase in the use of quantitative reviews (i.e. meta-analysis) in different fields of ecology and evolution (see Stewart 2010; Cadotte et al. 2012; Jennions et al. 2012; Poulin and Forbes 2012; Koricheva et al. 2013) suggests to us that meta-analysis has been superseding narrative reviews in synthesizing primary empirical studies. There are a number of reasons why this should be the case. Table 1 summarises key functional differences (or similarities) between narrative reviews and quantitative reviews. Here, we briefly highlight such differences. In-depth discussions of comparisons between qualitative and quantitative reviews can be found elsewhere (Cooper et al. 2009; Koricheva et al. 2013).

Table 1 A summary of capability differences between narrative reviews and meta-analysis in four different functions related to reviewing primary research studies (see the text for details)
Full size table

Testing hypotheses

Strictly speaking, a narrative review cannot be used to test a hypothesis. However, authors of such reviews have often attempted hypothesis testing by tallying up significant (positive/supportive) and non-significant (negative/unsupportive) studies, an approach known as ‘vote-counting’. Early advocates of meta-analysis severely criticized vote-counting mainly because it completely ignores study quality in terms of sample sizes and methodologies, often leading to erroneous conclusions (e.g. Bushman and Wang 2009; Stewart 2010; for the case for non-meta-analytic approaches, see Fletcher and Dixon 2012). Meta-analysis, in contrast, can be used for testing hypotheses in as rigorous a way as empirical studies. The ability to detect small effects (sensu Cohen 1988), which any single study cannot reliably detect, is probably its most obvious strength. For example, the first ever meta-analysis published in Evolutionary Ecology (Arnqvist et al. 1996) investigated a general trend in assortative mating by size among 45 populations of closely related water strider species. Their work revealed a weak but significantly positive trend towards assortative mating by size, which challenged the conclusions of earlier primary studies. The kinds of questions that one can ask with meta-analysis can be extended by the use of meta-regression, which is basically meta-analysis with predictors (often called moderators in the meta-analytic literature; reviewed in Thompson and Higgins 2002). Meta-regression is usually applied once meta-analysis detects statistically significant heterogeneity (see below; for example, variation in assortative mating among different populations may be due to differences in resource availability between these populations). It is worth noting that questions posed through meta-analysis are usually higher-order ones, investigating generalities across studies, populations and/or species. Obviously, meta-analysis is not suitable for appraising new hypotheses, for which no or few empirical studies exist (although meta-analysis can be performed with just two effect sizes; see Littell et al. 2008).

Identifying heterogeneity and bias

A key role of any scientific review of empirical studies must be the identification and description of consistency and inconsistency (heterogeneity) among findings within a topic. Narrative reviews can perform such tasks reasonably well. It is traditionally thought that a major role of narrative reviews is to reconcile apparent heterogeneity among studies (Petticrew and Robert 2006). An obvious weakness, however, is that narrative reviews often rely on the statistical significance of each primary study rather than actual effect sizes observed in each study (e.g. Hedges and Olkin 1985; Arnqvist and Wooster 1995; Nakagawa and Cuthill 2007). In contrast, meta-analysis is equipped with statistical methods to scrutinise heterogeneity such as Cochran’s Q or I 2, which statistically quantify the degree of inconsistency among studies (Higgins and Thompson 2002; Higgins et al. 2003). Publication bias, whereby statistically non-significant results are unlikely to be published (Rosenthal 1979; Møller and Jennions 2001), is a pervasive problem for all forms of literature reviews compromising the reliability of narrative reviews and meta-analysis alike. The clearest capability difference between the two types of reviews may be that meta-analysis comes with a range of tools that can detect and even correct publication bias (reviewed in Rothstein et al. 2005; Jennions et al. 2013; Table 1). Despite this advantage, Nakagawa and Santos (2012) report that ecologists and evolutionary biologists often do not perform the analyses required to deal with publication bias. Such poor practice clearly needs to change in the future.

Generating hypotheses and future directions

Narrative reviews or ‘opinion’ articles have always been primary platforms for the presentation of new hypotheses and future directions in given research areas. We argue, however, that meta-analysis is more than capable of doing the same, or even a better, job of generating new hypotheses and future directions than narrative pieces, although we feel that, in our field, this functional aspect of meta-analysis is underutilized. Majolo et al. (2012) discuss the following intriguing case where a meta-analysis generated a hypothesis, which was subsequently tested by empirical work (for another interesting example, see Ord and Stamp 2009).

Arnqvist and Kirkpatrick (2005) conducted the meta-analysis of the rate of extra-pair paternity and female costs associated with extra-pair mating. In this study, they arrived at the hypothesis that genetic correlations between two types of female mating behaviour can maintain an apparently maladaptive female extra-pair mating behaviour (referred to as the within-sex genetic correlation hypothesis). Forstmeier et al. (2011) tested two different hypotheses on the genetic correlations relating to female promiscuous behaviour using a captive population of zebra finches (Taeniopygia guttata). Their study failed to support the within-sex genetic correlation hypothesis by Arnqvist and Kirkpatrick (2005). Instead, Forstmeier and colleagues provided a strong support for the genetic correlation between female and male mating behaviour, i.e. the between-sex genetic correlation hypothesis, which had been proposed in the opinion article by Halliday and Arnold (1987).

In the next section, we condense the essence of the nine articles contributing to this special issue, which are mainly qualitative reviews about meta-analytic studies and methods published in the fields of ecology and evolution to date. Some contributions also include actual meta-analyses and one even features a meta-analysis of many meta-analyses, or a second-order meta-analysis (i.e., Castellanos and Verdú 2012).

Meta-analytic insights

Emerging insights

Before looking into each contribution, we highlight three general and important insights, which emerged from and are repeated in the nine contributions. First, meta-analyses are able not only to confirm a general pattern across populations and species, but also to resolve controversies arising from competing hypotheses. In some cases, meta-analyses revealed unexpected patterns overturning conclusions from earlier high profile and influential papers. Second, meta-analyses (more specfically meta-regression) have pinpointed the sources of heterogenetiy (e.g. different types of traits or methodologies) among seemingly inconsistent results from primary studies. Identification of important moderators sometimes led to refinements in theory and experiments. Third, meta-analyses often revealed that effect sizes for hypothesized relationships and differences are very small. Such findings meant that primary studies were generally underpowered to reach adequate conclusions whereas meta-analysis is an ideal alternative in such cases. Importantly, meta-analysis in the field of ecology and evoltuon has started to foster a scientific culture where each primary study, regardless of journal prestige, is re-examined collectively to draw the best current evidence for a topic in question.

Natural selection

Kingsolver et al. (2012) take a meta-analytic look at phenotypic selection in natural populations, i.e. differences in fitness among individuals associated with phenotypic variation. For meta-analytic purposes, they equate linear and quadratic selection gradients to effect sizes, since these estimate selection on the trait of interest, controlling statistically (via partial regression coefficients in a multiple regression) for the influence of other traits. Kingsolver and co-authors perform a meta-analysis on 1396 linear selection gradients and 686 quadratic selection gradients, using all suitable published estimates compiled by earlier meta-analyses. The results generally confirm those of earlier meta-analyses with regards to differences among types of traits or taxa. In particular, their meta-analysis suggests that stabilising selection is far from widespread. The authors then present an in-depth discussion of the influence of several factors, such as the choice of traits or fitness measures, the standardization of selection gradients, and sampling error, on the interpretation of meta-analyses of natural selection.

Sexual selection

Jennions et al. (2012) present a historical and prospective overview of meta-analyses of sexual selection. Surprisingly, the number of meta-analytic papers per year in sexual selection has not increased since the first paper appeared in 1996 (Arnqvist et al. 1996). Jennions and co-authors compiled and present a comprehensive table of 94 meta-analyses on sexual selection and related topics. Based on the information acquired from this table, they identified an interesting fact: 82 % of 179 different researchers, whose names are associated with these studies, have only co-authored a single meta-analysis, indicating that only a handful of researchers carry out meta-analyses as a part of their normal research toolkit (for the possible reasons, see Majolo et al. 2012). They highlighted the fact that testing hypotheses regarding sexual selection often required meta-analyses due to small effects. Most usefully, Jennions and co-authors not only provide tips for how one could conceive an idea for meta-analysis, but also list many fruitful but neglected topics for future meta-analyses on sexual selection (e.g. intra-sexual selection, the relationship between inbreeding and mate choice, and the relationship between phenotypic plasticity and sexual selection).

Community ecology

Echoing other contributors, Cadotte et al. (2012) show how the use of meta-analysis has increased in community ecology, and catalysed the search for general mechanisms structuring communities. From their investigation of 240 ecological meta-analyses, Cadotte and co-authors found that those published recently include many more taxa and datasets, the latter spanning more years, than earlier meta-analyses. These attributes of more comprehensive meta-analyses are generally positively associated with their citation rate. Importantly, meta-analyses of broad scope, including large numbers of species and datasets, also have more authors. Cadotte and co-authors highlight the crucial role played by synthetic centres and collaborative networks in expanding the scope and impact of meta-analyses. They finish by focusing on three case studies: (1) meta-analyses of the strength of competitive interactions, (2) the importance of diversity for ecosystem function, and (3) the effects of global warming on communities.

Host-parasite interactions

Poulin and Forbes (2012) show that after leading the way 15–20 years ago with the application of meta-analysis to ecological and evolutionary questions, researchers working on host-parasite interactions have kept pace with other evolutionary ecologists in their use of meta-analytic approaches. From a compilation of 40 meta-analyses on host-parasite interactions, Poulin and Forbes distinguish between meta-analyses that provide global assessment of empirical support for theory or for relationships expected from physiological mechanisms, those that play a deciding role in a debate between competing hypotheses, and those that reveal previously unexpected patterns. They also propose aspects of the relationship between hosts and parasites that are ripe for new meta-analyses, being both theoretically mature and rich with empirical studies. These include the interspecific relationship between host ecological features and parasite species richness, and the link between host resistance against parasites and the latter’s host specificity.

Plant evolutionary ecology

Castellanos and Verdú (2012) conduct the first ever second-order meta-analysis in evolutionary ecology. A second-order meta-analysis is a ‘meta-analysis of meta-analyses’ where a quantitative synthesis is conducted upon meta-analytic results rather than primary research papers. Second-order meta-analyses are reasonably common in medical and clinical sciences where these sorts of meta-analyses are also referred to as umbrella reviews (Caldwell et al. 2010; Ioannidis 2009). By gathering approximately 200 meta-analyses, Castellanos and Verdú showed that the strength of response to selection in plants depends more on the type of selection pressures (biotic or abiotic) than the type of traits (related to fitness or not); biotic selection pressures exerted greater responses than abiotic ones, while fitness and non-fitness traits, unexpectedly, differed little in response to selection. This first second-order meta-analysis in evolutionary ecology can be seen as a milestone and a manifestation of the maturation of the meta-analytic culture in ecology and evolution.

Social behaviour

Majolo et al. (2012) review meta-analytic insights into animal social behaviour, and identify six sub-topics containing 33 meta-analytic studies: (1) socio-ecology, (2) communication, (3) cooperation, (4) dominance, (5) sexual behaviour, and (6) parental investment. Among these topics, they state, meta-analysis has probably had its largest impact in the area of cooperation. Two significant insights were achieved: (a) supporting Hamilton’s rule in cooperative vertebrate species through investigating relations among degrees of helping, kin discrimination and dispersal in meta-analytic frameworks (Griffin and West 2003; Cornwallis et al. 2009); and also (b) providing strong evidence for one of the rare cases in which reciprocity works, via meta-analysing the exchange of grooming in primate communities (Schino 2001, 2007; Schino and Aureli 2008, 2010). They also propose three reasons why meta-analyses are still underused despite their undeniably powerful and useful nature: (1) small numbers of studies on exactly the same topics, (2) seemingly unequivocal empirical evidence for study topics, and (3) the lack of statistical information extractable for meta-analysis in the primary literature.

Behavioural syndromes

Investigation into animal personality (also know as behavioural syndromes) is an area of accelerating growth in interest (e.g. Réale et al. 2010). Garamszegi et al. (2012) conduct a meta-analysis on phenotypic correlations between behavioural traits (or personality traits such as activity, aggression, exploration and risk-taking). Their meta-analysis collates over 100 studies to test the generality of such phenotypic correlations (i.e. behavioural syndromes). Indeed, Garamszegi and co-authors find a positive but weak overall correlation (i.e. meta-analytic mean: r = 0.2) among personality traits. They importantly point out that it usually requires nearly 200 individuals to detect this magnitude of effect and that almost all the empirical studies had much smaller sample sizes. Also, they find that repeatability (or consistency) of personality traits is an important and positive predictor of phenotypic correlations. By using meta-regression, they also report other important moderators of the strength of phenotypic correlations; for example, males had higher overall correlations than females, and ectothermal vertebrates had higher correlations than endothermal ones.

Conservation biology

Côté and Reynolds (2012) summarise how the use of meta-analysis has contributed to the integration of concepts from evolutionary ecology into conservation science. Finding only 23 published meta-analyses dealing with conservation from an explicitly evolutionary perspective, they sorted them into four general areas: (1) relationships between genetic diversity, traits and fitness, (2) small population size, (3) changes over time, and (4) local adaptation. Although these meta-analyses generally confirmed a priori expectations, they have sometimes revealed weaker effects than previously assumed. For instance, despite genetic diversity being widely believed to co-vary with individual fitness, several meta-analyses of heterozygosity-fitness correlations have indicated weak and variable overall relationships. Also, the more primary studies that are included in a meta-analysis, the lower the magnitude of the resulting overall effect size linking genetic diversity with fitness. Côté and Reynolds then proceed to highlight the disconnection between the interests of researchers performing meta-analyses at the intersection of evolutionary ecology and conservation biology, and the needs of conservation managers, and end by suggesting possible remedies.

Methodological issues

Nakagawa and Santos (2012) review methodological advancements in meta-analytic methods, especially contributions made by ecologists and evolutionary biologists. They show that new meta-analytic methods, which combine phylogenetic comparative analysis and mixed-effects modelling, have resolved a number of problems posed by meta-analytic data typical of ecology and evolution (Hadfield and Nakagawa 2010); such data have multiple correlated structures due to temporal and spatial replications, as well as phylogenetic relatedness (i.e. statistical non-independence). Nakagawa and Santos advocate the advantages of multivariate meta-analysis in which there are more than one set of effect sizes (Jackson et al. 2011) and also of what they term ‘within-study’ meta-analysis whereby effect sizes from one study are appropriately combined to produce a meta-analytic mean. They conclude that the statistical difficulties in ecological meta-analysis listed by Gurevitch and Hedges (1999) may have been largely resolved.

Future issues and directions

Unresolved matters

Although the last two decades may have identified and resolved many technical matters relevant to meta-analysis in ecology and evolution (Gurevitch and Hedges 1999; Nakagawa and Santos 2012; for the advances in the medical literature, see also Sutton and Higgins 2008), there probably remain as many or more unresolved ‘meta-analytic’ issues. It is repeatedly mentioned by the critics of meta-analysis (e.g. Whittaker 2010) and also in the contributions of this special issue that we do not have objective ways of deciding on the quality of each empirical study independently of its sample size (e.g. Poulin and Forbes 2012). Medical and clinical sciences, on the other hand, already have working point-system protocols to rank primary studies (e.g. by the use of double-blind experimental setups; Egger et al. 2001; Higgins and Green 2008; Valentine 2009). We see the development of such a protocol to be a challenge for our fields because a meta-analysis in evolutionary ecology usually represents a heterogeneous collection of studies often carried out in uncontrolled environments, sometimes with multiple populations and species (Hadfield and Nakagawa 2010; Lajeunesse 2010). Also, most consistent pleas from meta-analysts ask for empiricists to report detailed enough results to be translated into effect size statistics, or to report such effect size statistics directly (e.g. Castellanos and Verdú 2012; Kingsolver et al. 2012; Majolo et al. 2012). This persistent issue perhaps needs to be resolved by a collective action of journals in ecology and evolution imposing clear guidelines of statistical reporting, just as many psychological journals follow the guidelines of statistical presentation endorsed by the American Psychological Association (e.g., they mandate reports of effect size statistics for main results; Wilkinson and The Task Force on Statistical Inference 1999).

Recently, a group of researchers have expressed their concerns regarding current citation practices of meta-analyses in ecology and evolution, because empirical studies used in actual meta-analyses are not cited in the main article but in its (electronic) supplementary material, which will not be indexed by ISI Web of Science or other citation databases (Kueffer et al. 2011). This ‘missing citation’ problem certainly needs to be rectified, and Kueffer and co-authors suggest that journals should allow extra pages for reference sections for the meta-analytic studies. Additionally, they propose close collaborative efforts between meta-analysts and empiricists. Such collaborations are required to achieve true synthesis, since meta-analysts often lack a strong biological basis on the original systems that empiricists study. We also remind readers that meta-analysis is not a panacea for salvaging the lack of statistical power and/or resolving equivocal results; meta-analysis also suffers from Type I and II errors and can lead to misleading and erroneous conclusions, as discussed by many (reviewed in Ioannidis 2010 and contributions in this issue).

Systematic reviews: the future of research synthesis?

Researchers within ecology and evolution may take the term ‘systematic review’ as synonymous with meta-analysis. However, a systematic review does not necessarily entail a meta-analysis, although it most often does (Petticrew and Robert 2006; Higgins and Green 2008). A systematic review represents a comprehensive literature review following strict protocols, which facilitates replicable and upgradable research syntheses, for example, most famously the PRISMA statement (Liberati et al. 2009; Moher et al. (2009); Table 2). The PRISMA statement aims to increase transparency in the process of literature synthesis, containing a checklist of 27 items along with a flow diagram, which visualizes procedures from database searching to inclusion decisions (see Fig. 1). The main difference in scientific practice we see between medical and social sciences, and fields within ecology and evolutionary biology, is that the former fields have collaborative networks, namely the Cochrane Collaboration (medical sciences) and the Campbell Collaboration (social sciences). These organisations oversee the standards of systematic reviews and meta-analyses; Table 2 lists establishments and publications, which promote rigorous practices of systematic reviews and meta-analyses. In our fields, institutions such as NESCent (National Evolutionary Synthesis Center, USA), NCEAS (National Center for Ecological Analysis and Synthesis, USA), and CEBC (Centre for Evidence-Based Conservation, UK) have promoted systematic reviews and meta-analyses, as pointed out by Cadotte et al. (2012) and Côté and Reynolds (2012). Furthermore, environmental sciences and conservation biology have similar collaborations, such as the Collaboration for Environmental Evidence and Conservation Evidence, which focus on aiding policymaking (Table 2). Despite the existence of these organisations, no ‘blue-sky’ areas of ecology and evolution have collaborative networks equivalent to the Cochrane and Campbell Collaborations.

Table 2 A list of organisations, publications and their online links relevant to systematic reviews and meta-analyses
Full size table
Fig. 1
figure 1

PRISMA flow diagram with the four stages: identification, screening, eligibility and included data. Redrawn from Moher et al. (2009)

Full size image

We believe that the PRISMA statement, which has been developed for the health sciences, can potentially be adopted for meta-analyses in ecology and evolution, and that it should be routinely used. Also, a collaborative site for the fields of ecology and evolution (especially topics not directly related to political decision making), akin to the Cochrane and Campbell Collaborations, needs to be developed and actively utilized by ecologists and evolutionary biologists to achieve more rigorous and integrated research synthesis. Such a collaborative site can play a role in: (1) facilitating the use of stringent protocols, (2) dealing with prospective registrations of meta-analytic topics like the one carried out by PROSPERO (Booth et al. 2012; Table 2), especially to avoid different research groups or individuals independently and simultaneously conducting meta-analyses on the same/similar topics, and also (3) organizing task groups for resolutions of the issues listed above.

Concluding remarks

We are entering into an unprecedented data-rich era where terabytes of relevant data are accessible from one’s own laptop, as featured by opinion articles in the early 2011 Science special issue, “Dealing with data” (e.g., Richman et al. 2011). In our special issue, Kingsolver et al. (2012) point out that reanalysis of published data, i.e. ‘secondary’ analysis, (sensu Glass 1976) and amalgamation of secondary analyses will become a feasible and practical option via public data repositories such as Dryad, a member of the DataONE federation (http://www.dataone.org/). Such direct reanalysis approaches could potentially be more powerful than meta-analysis (and second-order meta-analysis), although the heterogeneous nature of ecological and evolutionary datasets will continue to challenge researchers (Richman et al. 2011). We imagine that formal systematic reviews encompassing meta-analyses will become increasingly important and play a central role in research and data syntheses among many methods of synthesis in evolutionary ecology and other areas of research alike. As we have outlined above, and as can be seen in this special issue, meta-analyses have already provided us with vast arrays of new insights into evolutionary ecology over the last 20 years. Now, our hope is that ecological and evolutionary meta-analyses soon evolve into systematic reviews, which employ transparent and replicable protocols supported by a collaborative network of ecologists and evolutionary biologists (see Garamszegi and Nunn 2012).