Introduction

A recent analysis by Møller and Jennions (2002) suggested that ecologists using statistical models are explaining little of the variability in ecological studies. They paint a bleak picture by suggesting that the mean amount of variance in a measured trait explained by the main factor of interest is between 2.5% and 5.42%. This conclusion was based on a meta-analysis of 43 published meta-analyses in physiology, ecology and evolutionary biology. Additionally, the authors suggest sample sizes of greater than 100 are necessary to adequately assess significance of effects, with 80% power.

Ecologists have the challenging task of explaining and predicting the complex natural world. Ideally, ecologists would like to explain 100% of the variability in a system. However, random processes, inherent complexity, measurement accuracy, individual behavior, genetic variability, evolutionary history and the interaction of these factors complicate this explanatory process (Mangel et al. 2001; Møller and Jennions 2002). Nevertheless, ecologists often employ a number of statistical models in an attempt to quantify relationships between the most important dependent and independent variables when examining natural phenomena (Hilborn and Mangel 1997). Many of these statistical models rely on variance techniques to quantify the portion of variance in dependent variables that is explainable by independent variables. Other unmeasured independent variables and the factors listed above are lumped into residual error, or unexplained variance. These statistical models are largely without mechanism, but quantify whether one variable changes in concert with other variables (Mangel et al. 2001).

Although we agree that there is considerable variability in ecological systems, we disagree with the approach that Møller and Jennions (2002) used to draw their conclusion and believe that their research findings are misdirected. By performing a meta-analysis of meta-analyses, they (1) combine studies at different temporal and spatial scales (Osenberg et al. 1999); (2) cannot account for changes in the relationship between ecological variables in different study organisms or locations (Arnqvist and Wooster 1995); and (3) suggest large sample sizes for ecological studies when their calculation of sample size is based on correlations twice removed from the original studies. We argue that the strength of relationships in ecological studies are better assessed using the original statistical models as is done with a meta-analysis rather than with a meta-analysis of meta-analyses, and we hypothesize that a much greater portion of variability is explained than proposed by Møller and Jennions (2002).

Instead of asking "How much variation can be explained by ecologists?" as was done by Møller and Jennions (2002), we posed the question: "How much variation in relationships between variables is not explained?" This approach allowed us to examine the amount of variation not explained with statistical models relative to the total amount of variation in the data set. In effect we assessed the importance of random processes, inherent complexity, measurement accuracy, individual behavior and evolutionary history relative to the relationships between measured variables in a given study. Furthermore, we hypothesized that the variability explained using statistical models would be greatest for experimental studies in controlled environments (e.g. greenhouse, laboratory or growth-chamber) and least for observational field studies as randomness and noise would often be less under more controlled laboratory conditions.

Materials and methods

We collected data from published studies in the journals Behavioral Ecology and Sociobiology, Ecology, Ecosystems, Evolution and Oecologia in the year 2001. We chose studies in these five journals to represent the wide range of numerous sub-disciplines of ecology (Table 1) and assumed that the power of the statistical analyses published in 2001 and in these journals would be similar to those over a broader range of years and publications. All of the factors included in a statistical analysis were included regardless of their significance; this reflected the investigator's attempt at looking at the "important" factors in a given system.

Table 1. Numbers of surveyed papers by ecological disciplines
Full size table

In statistical models, the proportion of variability (η2) in a dependent variable attributed to an effect can be expressed as the ratio of sum of squares of that effect (SSeffect) to the total sum of squares (SStotal) (Pearson 1911; Fisher 1928). We modified η2 from the proportion of variability in a dependent variable attributed to an effect to quantify the proportion of variability not explained (i.e. residual or experimental errors) by experimental factors as:

$${{\rm{\eta }}^{{\rm{2}}} {\rm{ = }}{{{\rm{SS}}_{{{\rm{error}}}} } \over {{\rm{SS}}_{{{\rm{total}}}} }}}$$
(1)

Thus, percent variability explained by factors would be calculated as 100×(1−η2). The statistic we used, η2, limited the type of data we could include in our data set to linear regression, fixed effect ANOVAs and fixed effect ANCOVAs (see Tabachnick and Fidell 1983). We were also restricted to studies that published enough information to obtain the degrees of freedom, error and total sums of squares or r 2 values. Since the addition of more factors in a given study will explain more of the variance, regardless of the significance of the factors, we calculated the weighed mean of η2 using total degrees of freedom as the weight factor.

We found a total of 1,137 error terms in 181 published studies that met our criteria (see Appendix). Of the 1,137 error terms, 166 were controlled experimental studies, 376 were field experimental studies, 67 were controlled observational studies and 528 were field observational studies. Multiple η2 values per study were treated as non-independent samples. To account for this non-independence, we randomly chose one analysis per study, calculated η2 per study and the weighted mean η2 across all 181 studies; this was repeated 1,000 times. Means and 95% confidence intervals (CI) were obtained from the 1,000 iterations of this randomization procedure. Means of η2 with non-overlapping confidence intervals were considered significantly different.

Results and discussion

The proportion of variability not explained by experimental factors (η2) estimated from our analysis of all combined ecological studies (Table 1) had a 95% confidence interval of 63%–45%. Thus, ecologists using statistical models are explaining roughly half of the variability in dependent variables in their studies. When further dividing these studies into experimental or observational, we found similar mean unexplained variance (Table 2). Controlled and experimental studies had the smallest unexplained variance, roughly 47%, but none of the comparisons were statistically significant (Table 2). Consistent with our hypothesis but not statistically significant, controlled studies that were experimental in design had the lowest unexplained variance (43%) when compared to experimental field or controlled or field observational studies (Fig. 1). Observational field studies, with little to no control over factors influencing variance, still explained on average 51% of the variation in response variables.

Table 2. Main effects of variance not explained in ecological studies, with comparisons between different locations (controlled environments or field studies) and different types of design (observational or experimental studies)
Full size table
Fig. 1.
figure 1

Percentage of variance attributed to error in ecological studies. Data were collected from papers containing simple fixed effect ANOVAs, ANCOVAs and simple linear regressions. Studies were categorized into controlled environments (solid bars, e.g. greenhouse, growth chamber) or field environments (hatched bars) and further divided into experimental (colorless bars) or observational (gray bars) studies

Full size image

Our results show that ecological studies evaluated with statistical models can account for considerably more variation than Møller and Jennions (2002) would suggest. This is in large part due to a fundamental difference in how Møller and Jennions (2002) viewed ecological relationships as compared to the approach we now present. Their meta-analysis of meta-analyses looked at linear response to single factors (e.g. elevated CO2, mating success, sexual selection) across a wide range of studies that may have come from entirely different systems. Such an approach fails to appreciate that non-linear complexity and threshold responses are ubiquitous across ecological systems (Romme et al. 1998; Maurer 1999), and that statistical models are generally specific to a location and system (Mangel et al. 2001). We believe what Møller and Jennions (2002) measured was how universal the relationships are between variables across multiple ecological systems, not how much variability in these relationships can be explained by ecologists. For example, a study on the effects of elevated CO2 on plants (Curtis and Wang 1998) was included in Møller and Jennions' meta-analysis. In that study, CO2 effects were examined across numerous plant types from coniferous forests to tropical trees and from arid systems to more mesic deciduous forests. By combining all of these studies into one meta-analysis, one is in effect putting a large degree of variability into the data with little more than a hope to get a correct sign (positive or negative) in the relationship between CO2 and plants, not explain variability. In addition, Møller and Jennions (2002) introduce another level of variability by combining numerous meta-analyses that span several spatial scales (Osenberg et al. 1999) from the individual (Thornhill and Møller 1998) to the ecosystem (Brett and Goldman 1996), and temporal scales including generational times (Järvinen 1991) and single field seasons (Xiong and Nilsson 1999).

We also feel that Møller and Jennions (2002), in their retrospective power analysis, greatly over-estimate the sample sizes (122–396) necessary for adequate statistical power to measure significant effects. Because they performed a meta-analysis of meta-analyses, their suggested sample sizes should reflect the number of meta-analyses needed to obtain overall significance of effects, not the number of samples necessary in a single study. The sample sizes reported from our sampled studies reflect a wide range from a minimum of five replicates to sample sizes greater than 4,000, with a median of 25. Although many of the studies found had sample sizes suggested by Møller and Jennions (2002), many still had much lower sample sizes with adequate statistical power. This is an important consideration as allocation of limited resources to increase replication is of critical importance when designing ecological studies.

In conclusion, we feel that ecologists using statistical models are explaining much more of the variation in relationships between important variables in ecological studies than suggested by Møller and Jennions (2002) and are currently using adequate sample sizes. Our results may also suffer from the same limitations as meta-analyses, such as publication bias (Cooper and Hedges 1993) and combining studies across spatial and temporal scales (Gurevitch et al 2001). Nevertheless, explained variance in the individual studies we examined is vastly different from the 2.5%–5.42% reported by Møller and Jennions (2002). Our analysis suggests, when combined with that of Møller and Jennions (2002), that the scope of inference for single-factor effects is small in scale and that extrapolations of single-factor effects across vastly different systems may be difficult. This conclusion, however, does not suggest that ecologists are inherently limited in their ability to quantify interactions occurring in ecological systems. We believe this is a potentially dangerous perspective given the importance of ecological science in public policy and decision-making. It also highlights the need for more mechanistic studies that do not rely on correlative analyses which can be subject to spurious relationships, in conjunction with more advanced/predictive modeling that clarifies those mechanisms to elucidate the important ecological patterns and processes (Hilborn and Mangel 1997). While we feel meta-analyses are important syntheses, they should be confined to assessing the consistency in response of single-factor effects across systems or environmental conditions and should not be used to quantify the portion of variability explained (Gurevitch et al. 1992; Osenberg and Mittelbach 1989). Additionally, the conflicting results between our study and that of Møller and Jennions (2002) illustrates the difficulty in finding emergent properties or fundamental ecological laws from independent studies that result from the complex nature of ecological systems (Brown 1999).