Introduction

Although the term ‘niche’ was coined at the beginning of the twentieth century (Grinnell 1904), the modern concept is that of Hutchinson (1957): the niche as that volume in multidimensional hyperspace in which species can maintain a viable population. One concept that arises is that of niche overlap between species. We suggest here that present indices for measuring the degree of niche overlap are inadequate in many contexts. We propose a new approach.

In the Hutchinsonian niche concept, each axis of the hypervolume can be an environmental condition (e.g. salinity, temperature) or a resource (e.g. prey type, soil nitrogen) (Schoener 1989). Niche overlap in environmental axes can indicate the extent to which species will be found in the same locale, for example in the same range of temperatures and salinities. This is an overlap in beta (conditions) niche (Pickett and Bazzaz 1978; Wilson 1999). Species can also overlap in alpha (resources) niche. Recently Tokeshi (1999) has introduced a refinement of Hutchinson’s concept—the ‘utilitarian niche’—and Rosenfeld (2002) that of the ‘functional niche’. In both, the hyperspace axes are functions or processes defined by functional traits of species.

Overlap in this hyperspace is crucial to several aspects of ecological theory: (1) Low overlap in alpha niche is a major explanation for the continued co-existence of species (Mookerji et al. 2004; Silvertown 2004); (2) Niche overlap is basic to seeking assembly rules in ecological communities, especially in testing the theory of limiting similarity, which states that species with high mutual niche overlap will be unable to coexist (MacArthur and Levins 1967); (3) Redundancy, an important concept in discussions on diversity/ecosystem-functioning relationships, is basically niche overlap (Loreau 2004), and (4) The driving force behind character displacement and adaptive radiation is prior niche overlap (Day and Young 2004).

For all these purposes, an index is needed to quantify niche overlap. Most existing indices assume that the species are described by their occupation of a number of discrete categories (e.g. prey types; Pianka 1973). However, often our description of the niche of a species is based on data that are continuous (e.g. abundance at each altitude, or frequency of use of prey where prey size is measured on a continuous scale). An index of the Pianka type simply cannot be applied directly. In order to use it, the niche axis must be split into categories, and information is lost in this process. Here, we propose the application of a new estimator that is free from these problems. It is non-parametric to avoid constraints of distributions and resulting bias. We illustrate its use with examples from lagoon fishes and terrestrial plants.

Materials and methods

Overlap between two normal distributions

Several methods have been proposed for estimating the niche overlap among species. Schatzmann et al. (1986) reviewed 13 such indices. However, all of these indices assume that the resources apportioned by species are in discrete categories (e.g. the prey type consumed by each species). When resources are estimated on a continuous scale or when ecological studies deal with quantitative functional traits, we need to measure overlap in a different way. The same calculations would apply to any quantitative variables, but for niche overlap work, traits that are apparently functional would be chosen.

If we assume that the niche-use spectrum of a species follows a normal curve, the niche overlap between species i and species j can be measured by the usual formula for the overlap of two normal curves:

$${\text{NO}}_{\text{N}} {\left({i,j} \right)} ={\text{e}}^{{\frac{{ - {\left( {\mu _{i} - \mu _{j} } \right)}^{2} }} {{{\left( {2{\left( {\sigma ^{2}_{i} + \sigma ^{2}_{j} } \right)}} \right)}}}}} ,$$
(1)
μ i :

mean (= mode = median) of the distribution of species i,

μ j :

mean of the distribution of species j,

σ 2 i :

variance of the distribution of species i,

σ 2 j :

variance of the distribution of species j

This origin of this measure remains unclear, though it was attributed to MacArthur and Levins (1967) by Manly (1994) and was used by Cody (1975).

This overlap index has two major problems. Firstly, it assumes that the quantitative traits used to measure overlap are normally distributed. Morphological characters, including functional traits, are often not normally distributed (e.g. Stehlikova et al. 2003) and physiological ones are probably not either. To avoid this problem, some authors have proposed versions of the index with different distributions of the trait, such as the Weibull distribution (Manly and Patterson 1984), but the restriction to a particular distribution remains. Secondly, the overlap that we aim to estimate is the area under the two distributions for species i and j (Fig. 1). If we consider the example where distributions for species i and j are normal distributions with the same mean but different variances, the true overlap (i.e. area under both curves) is clearly <1 whereas the index value given by Eq. 1 is 1 because μ i  − μ j  = 0, and thus NON(i, j)=1 (Fig. 1).

Fig. 1
figure 1

Two normal distributions with the same mean (10) and two different variances (4 and 9 respectively). The area under each of the distributions is 1

Full size image

We therefore do not advise using index NON.

Overlap between two kernel distributions

To avoid the constraint of assuming a particular distribution (often normal) of the traits used to estimate overlap, we propose to use an unbiased overlap index, already introduced by Stine and Heyse’s (2001), which is based on kernel distributions and estimated by integral functions. It is non-parametric, and can be used when the distribution of the trait (i.e. its density function) is unknown. These estimates are based on Stine and Heyse’s work (2001), replacing the overall unknown density function by a set of normal density functions (Fig. 2). Given n data from a sample of a continuous trait x: X 1, X 2,..., X n , the kernel density is:

$$\ifmmode\expandafter\hat\else\expandafter\^\fi{f}_{h} {\left(x \right)} = \frac{1}{n}{\sum\limits_{i = 1}^n {\frac{1}{h}K{\left({\frac{{x - X_{i}}}{h}} \right)}}}.$$
(2)
Fig. 2
figure 2

Kernel density estimate for the distribution of a data set with five values. The population density is the sum of five kernel normal distributions for each observed data

Full size image

The non-parametric estimator of the density is calculated with the standard normal density function K, named kernel:

$$K{\left( {\frac{{x - X_{i} }} {h}} \right)} = \frac{{{\text{e}}^{{\raise0.7ex\hbox{${ - {\left( {\frac{{x - X_{i} }} {h}} \right)}^{2} }$} \!\mathord{\left/ {\vphantom {{ - {\left( {\frac{{x - X_{i} }} {h}} \right)}^{2} } 2}}\right.\kern-\nulldelimiterspace} \!\lower0.7ex\hbox{$2$}}} }} {{{\sqrt {2\pi } }}} .$$
(3)

These kernels are centred on each data point (Fig. 2) and integrate to 1.0. Each kernel is divided by n to estimate the total kernel density (formula 2) so that each area is 1/n and the area under the population density function is 1.0. h is the bandwidth which controls the smoothness of the estimator.

The use of a normal density function to estimate the density does not assume that the sample is normal (Stine and Heyse 2001). Moreover, simulations have shown little impact of the distribution of the kernel on the estimate of overlap (Stine and Heyse 2001).

Once we obtain a non-parametric kernel density function for a quantitative functional trait (t) for each of two species i and j, we can estimate their niche overlap as the area under the smaller of the two population density functions:

$${\text{NO}}_{{\text{K}}t} (i,j) = {\int {\min {\left({f_{it} (x),f_{jt} (x)} \right)}{\rm d}x}},$$
(4)

where f it and f jt are the kernel population density functions for species i and j and functional trait t. NOKt (i, j) tends to 0 when the two distributions are disjoint and is 1 when they are perfectly similar. Following Stine and Heyse (2001), the overlap can alternatively be estimated as:

$${\text{NO}}_{{\text{K}}t} (i,j) = 1 - \frac{1}{2}{\int {{\left| {f_{it} (x) - f_{jt} (x)} \right|}{\rm d}x}}.$$
(5)

Because:

$${\int {f_{it} {\left(x \right)}}}{\rm d}x + {\int {f_{jt} {\left(x \right)}{\rm d}x}} = 1 + 1 = 2 = {\int {{\left| {f_{it} (x) - f_{jt} (x)} \right|}{\rm d}x}} + 2{\int {\min {\left({f_{it} (x),f_{jt} (x)} \right)}{\rm d}x}}.$$

This estimator should consistently estimate the true overlap between two species for one functional trait without assumptions about shape of the distribution.

The value of the bandwidth h is the main critical issue because the estimate of overlap is quite sensitive to it (Stine and Heyse 2001). In general, the smaller the bandwidth, the smaller the estimated overlap because, as the bandwidth shrinks, the kernel functions become more narrow and concentrated around the observed data leading to low overlap estimation. Conversely, as the bandwidths grow, the overlap increases because large values of h produce a very smooth kernel density (formula 2) and a wide kernel leading to more overlap between the distributions (Stine and Heyse 2001).

In our article we used an optimal bandwidth proposed by Silverman (1986) when normal kernel density functions are applied to normal data with variance σ2 to approximate their distribution: this estimate of the bandwidth is given by h=1.06σn −1/5 (Stine and Heyse 2001). So, h depends on the population only through the standard deviation σ of the dataset and its size n. This critical point has to be discussed further in this article.

The use of several traits, and weighted average niche overlap

When several quantitative traits are considered, we can estimate the multivariate overlap between species in either of two ways: the average niche overlap over the traits or the overlap in multidimensional space. The latter seems more attractive because this overlap is basically the intersection of species in the multidimensional space. However, when the number of quantitative traits increases, the multidimensional overlap can only decrease or stay constant. For instance, if we measure two functional traits on species, one related to the diet and the other to the reproduction, we would obtain a zero/null overlap for two species sharing exactly the same resources but with totally different reproductive life-history traits despite the obvious competition and thus overlap between these species if they co-occur. This example is caricatured, and ecological interactions are often much more complex and in some cases the multidimensional overlap is certainly appropriate. However, overlap estimation in a multidimensional space was not considered in our article, and we present only the calculation of the average overlap when several quantitative traits are considered.

Having calculated the overlap between two species i and j from formula 5 for each trait t, NOK (i,j,t), we calculate the mean overlap for all T quantitative functional traits:

$${\text{NO}}_{\text{K}} (i,j) = \frac{1}{T}{\sum\limits_{t = 1}^T {{\text{NO}}_{\text{K}} (i,j,t)}}.$$
(6)

The associated variance is:

$$s^{2}_{{{\text{NO}}_{\text{K}} (i,j)}} = \frac{{{\sum\limits_{i = 1}^T {{\left({{\text{NO}}_{\text{K}} (i,j,t) - {\text{NO}}_{\text{K}} (i,j)} \right)}^{2}}}}}{T}.$$
(7)

Nevertheless, one could argue that when the aim is to estimate the mean overlap between species, and their traits are statistically dependent, we should discount the degree of independence of each trait by downweighting highly correlated traits. The logic is that such traits are related to similar functions and are thus redundant, whereas independent traits add new functional information. For this, we propose the use of the ‘estimator in a dependent sample’ (EDS) of Kark et al. (2002). In this variant of our index, the weighted average niche overlap between two species i and j is given by:

$${\text{NO}}_{\text{Kw}} (i,j) = \frac{1}{{{\sum\limits_{t = 1}^T {w_{t}}}}}{\sum\limits_{t = 1}^T {w_{t} {\text{NO}}_{K} (i,j,t)}}.$$
(8)

The associated variance is:

$$s^{2}_{{{\text{NO}}_{\text{Kw}} (i,j)}} = \frac{{{\sum\limits_{i = 1}^T {w_{t} {\left({{\text{NO}}_{\text{K}} (i,j,t) - {\text{NO}}_{\text{Kw}} (i,j)} \right)}^{2}}}}}{{{\sum\limits_{t = 1}^T {w_{t}}}}}.$$
(9)

As in formula (6), NOK(i, j, t) is the niche overlap between species i and j for the trait t, and T is the number of functional traits. w t is the weighting parameter, i.e. the degree of independence for trait t:

$$w_{t} = \frac{1}{2} + {\sum\limits_{l = 1}^T {{\left({1 - \frac{{r_{tl} ^{2}}}{2}} \right)}}}$$
(10)

r tl is the correlation coefficient (Pearson in our study) between traits t and l over all the species considered.

If one trait t is totally independent from the others (i.e. r tl =0 for all traits l except r tt =1), we obtain w t  = 1/2 + (1-1/2) + (T−1) (1) = 1 + T − 1 = T which is the maximum weighting value with T traits. This procedure gives an estimate of the average overlap between two species based on a set of multiple-correlated traits.

In our examples below, we use the weighted version of the mean overlap, but some ecologists may prefer the unweighted version (formulae 6, 7) because when two traits are strongly related it does not necessarily imply that the functions they represent are functionally redundant. For example, when functional traits of fishes are not standardized they depend mainly on the fish size, and so are highly related, but they may be linked to different functions [e.g. size of the mouth and size of the eye in Dumay et al. (2004)].

Average niche overlap for a community

When several species are considered, i.e. when a community is sampled, and when each species has several functional traits (T), we advocate calculating a new mean niche overlap for the whole community. With S sampled species we obtain S(S−1)/2 pairs of niche overlap values between species. Thus, for a community with S species and T traits, the average niche overlap using the weighted version is:

$${\text{NO}}_{\text{K,community}} = \frac{{{\sum\limits_{i = 1}^{S - 1} {{\left({{\sum\limits_{j = i + 1}^S {{\text{NO}}_{\text{Kw}} (i,j)}}} \right)}}}}}{{\frac{{S(S - 1)}}{2}}}$$
(11)

and its associated variance:

$$s^{2}_{{{\text{NO}}_{{{\text{K,community}}}}}} = \frac{{{\sum\limits_{i = 1}^{S - 1} {{\left({{\sum\limits_{j = i + 1}^S {{\left({{\text{NO}}_{\text{Kw}} (i,j) - {\text{NO}}_{{{\text{K,community}}}}} \right)}^{2}}}} \right)}}}}}{{\frac{{S(S - 1)}}{2}}},$$
(12)

where NOKw(i, j) is the weighted average niche overlap between species i and j (formula 8). For the unweighted version, NOKw(i, j) is replaced with NOK(i, j) (formula 6).

All the niche overlap indices and their variances can be computed with an S-plus script (Venables and Ripley 2002). This script uses S-plus functions for the kernel and integration (formula 5) calculations.

Coastal lagoon fishes

To illustrate the overlap indices, we use data from five fish species from west Mediterranean lagoons: Atherina boyeri (Risso), Syngnathus abaster (Risso), Salaria pavo (Risso), Chelon labrosus (Risso) and Pomatoschitus microps (Krøyer), with respectively 42, 51, 3, 5 and 50 sampled individuals. These fishes were caught in Thau (43°24′N–3°36′E) and Mauguio (43°34′N–4°03′E) lagoons using a draw net in spring 2003. A. boyeri, S. abaster and P. microps are small-body species and the most common fishes in these lagoons while C. labrosus and S. pavo are more difficult to catch with this sampling method. To characterize the functional niche of each species, eight functional traits were measured, bringing information on three factors: diet, prey capture and position in the water column (Table 1) (Dumay et al. 2004).

Table 1 Parameters measured on fish and their functional importance
Full size table

Because of size differences, fish species traits were standardized (Adite and Winemiller 1997; Winemiller 1991). The standardization was by biomass because recent studies have highlighted the robust relationship between morphological or metabolic rates and body mass (Enquist and Niklas 2001; Niklas and Enquist 2001; West et al. 1997). The exception to this was gut length which was standardized by the standard length according to Cleveland and Montgomery (2003) and Kramer and Bryant (1995). If the allometric relationship between a trait (X) and mass (M) is X=aM b and the exponent coefficient is invariant between scales or species, [ln(X+1)][ln(M+1)]−1 could be expected to be constant or robust for the same population. So, this transformation was chosen.

Plant communities

To illustrate the overlap indices among plant species, data from nine plant species found within a sand dune community at Kaitorete Spit on the east coast of South Island, New Zealand (43°50′S 172°35′E) were used. These species were Acaena agnipila Gand., Einadia triandra (Forster f.) A. J. Scott, Calystegia soldanella (L.) R. Br., Carmichaelia appressa G. Simpson, Desmoschoenus spiralis (A. Rich.) Hook. f., Hypochaeris radicata L., Lagarus ovatus L., Muehlenbeckia complexa (A. cunn.) Meissn. and Rumex acetosella L. The species composition was determined by sampling 20 randomised 0.5×0.5 m quadrats within each of nine 50 m2 areas, and measurements were made on individuals randomly selected across these areas (see Stubbs and Wilson 2004, for further details; Table 3).

To characterize the functional niche of each species, five functional traits were chosen to reflect the ways in which the potentially important resources (water, nutrients, light etc.) are acquired and retained by plants: number of leaves on the terminal shoot [a terminal shoot was defined as an entire tiller for grasses and graminoids, and the shoot distal to the lowest leaf remaining on the main stem for shrubs and forbs (Wilson et al. 1994)]; leaf inclination from the horizontal; specific leaf area (SLA = leaf area/leaf mass); support fraction [non-photosynthetic tissue mass as a proportion of the total terminal shoot (Wilson et al. 1994)] and total chlorophyll content per unit fresh leaf weight [by extraction in N,N-dimethlyformamide and spectrophotmetric measurement at 663.8 nm and 646.8 nm (Porra et al. 1989)]. Each of the first four characters was measured on ten individuals of each species, while total chlorophyll content was only measured on six individuals of each species because of its expensiveness.

Results

Overlap estimation among species using fish morphology data

The Anderson-Darling normality test showed significant departures from a normal distribution for 80% of the 40 distributions (8 traits × 5 fish species).

Among the ten species pairs (Table 2), the lowest NOK value (0.02) was for S. abaster and C. labrosus. These species have distinct functional niches; the former is the black-striped pipefish, a fusiform zooplanktonophagous species with a large eye diameter, whereas the latter is a detritivorous/herbivorous Mugilidae species with a long gut length and a long protrusion (Dumay et al. 2004). The highest overlap index (0.53) was between A. boyeri and P. microps. These species were not placed in the same functional group by Dumay et al. (2004), but they have some similarities such as feeding on small invertebrates.

Table 2 Average weighted niche overlap, using the kernel-based index NOKw (formula 8), for the eight functional traits considered (Table 1) between each pair of fish species: mean (SD)
Full size table

Niche overlap estimates obtained for our data set using the kernel method (NOK) were significantly lower than those obtained with the existing normal-distribution-based NON (t-test on 80 NON estimates vs the corresponding 80 NOK estimates, P<0.05). Thus the overlap estimation using the normal distribution appears to be overestimating the ‘true’ overlap (as estimated using the kernel method).

This general result is illustrated in Fig. 3 for a particular case. If we consider the overlap between P. microps and A. boyeri for the standardized eye diameter functional trait (Table 1), its value estimated with NOK was 0.53, compared to 0.92 with NON. The distribution of this functional trait was far from being normal (P<0.001 with the Anderson-Darling normality test), being highly asymmetric. Thus the area under the two distributions varies considerably depending on whether we consider a normal or a kernel distribution.

Fig. 3
figure 3

Example of overlap between two fish species (A. boyeri (solid line) and Pomatoschistus microps (dotted line)) with respect to the eye diameter functional trait, using the kernel method (a) and Cody’s method assuming a normal distribution of data (b)

Full size image

The distributions of several of the species’ functional traits were of shapes similar to this, resulting in similar spurious overlap indices when normal distributions were assumed.

Overlap estimation among species using plant data

The Anderson-Darling normality test showed significant departures from a normal distribution for only 27% of the 45 distributions (5 traits × 9 plant species).

Using the five functional traits, overlap values ranged between 0 and 0.78 (Table 4). For four of the five traits the minimum overlap was 0.00—the number of leaves on a terminal shoot, support fraction, leaf area and total leaf chlorophyll—and for all these traits at least some of the zero overlaps were with the cyperaceous dune builder D. spiralis. However, for leaf chlorophyll six of the seven zero overlaps involved the rosaceous Australian herb A. agnipila. The minimum overlap in leaf inclination was 0.1.

The maximum overlaps were between various species, but involved the native bindweed C. soldanella for the number of leaves, leaf area and leaf chlorophyll, the European forb H. radicata for support fraction, leaf chlorophyll and leaf inclination, and A. agnipila for the number of leaves and leaf inclination. If we consider the functional overlap in terms of a weighted average over the five traits, the three most similar species are H. radicata, R. acetosella and E. triandra (overlap >0.6 for all pairs) whereas the most dissimilar pair of species was D. spiralis Lagurus ovatus (0.11). D. spiralis had low weighted average functional overlap values with all the species examined in this study.

At the community level, the estimated overlap varied between areas in the range 0.26–0.37 (Table 3).

Table 3 Plant species presence and weighted average overlap (SD is associated Standard Deviation) for each sampled area
Full size table

Discussion

Even though the principal aim of this study was to propose a new method for estimating niche overlap on quantitative traits, the ecological results we have obtained are interesting.

Using lagoon fish species, we found a maximum weighted average overlap of 0.53 (A. boyeri and P. microps) but the fish overlaps are generally low (<0.4) suggesting large differences in functional attributes (Table 2). The five species considered are all from different functional groups (Dumay et al. 2004), but S. abaster stands far apart from the other non- Syngnathidae species because of its unusual pipefish characteristics. Its colouration and long, slender body mimics the aquatic plants amongst which they live. It did not show protrusion, but its tubelike mouth allows it to ingest tiny prey from some distance, which may compensate for its slow speed, and its large eyes are well adapted to locate small prey precisely. Thus we are not surprised to find low average functional overlaps between S. abaster and other species (0.02–0.30).

It is interesting to note that of the nine plant species compared with NOK, D. spiralis and L. ovatus were the two most dissimilar (Table 4). Both of these species have a graminoid form, and in a simplistic classification, of the kind frequently used in functional-type/ecosystem-function studies, they would be placed in the same functional group. This highlights the importance of using a wide variety of functional characters. We should allow the plants to indicate which of these are the important characters when searching for evidence of assembly rules within plant communities, rather than allowing our preconceptions to bias the results. Very little difference was observed in the community overlap values from the nine areas examined at Kaitorete Spit. This is perhaps not surprising as the species composition within most of these areas was very similar, with four of the nine species occurring in all of the areas, and a fifth species occurring in all bar one area. However the remaining four species were scattered across the nine areas and were all fairly dissimilar to one another. The values of community niche overlap were relatively low (Table 3), supporting the previous conclusion that limiting similarity was operating at this site (Stubbs and Wilson 2004).

Table 4 Weighted niche overlap, using the kernel-based index NOKw (formula 8), for each of the five functional traits considered for plants between each species pair
Full size table

In this study, we have pointed out that the index which we name NON, used by MacArthur and Levins (1967), Manly (1994) and Cody (1975) to estimate overlap between two continuous distributions, is biased (Fig. 1). Moreover it assumes that the ecological traits used to estimate overlap are normally distributed which, as in the present study, is not always the case. Given the wide use of niche overlap indices in ecological studies (related to evolutionary process (e.g. Day and Young 2004), to coexistence rules (Kingston et al. 2000; Sugihara et al. 2003; Winemiller and Kelso-Winemiller 2003) and to functional diversity (e.g. Diaz and Cabido 2001), it is extremely important to derive an unbiased measure which can be used on data from a continuous distribution. The NOK family of indices that we introduce here, being based on a kernel distribution estimator, are independent of the particular form the distribution of a trait takes. These indices can be calculated at either the species or the community level, and we have provided also an associated variance with each index.

As it was claimed by Stine and Heyse (2001), the choice of the bandwidth is still a critical issue when applying kernel density estimators to estimate overlap between two distributions because this choice influences the final result and their no consensus about the ‘correct’ bandwidth to adopt in all cases. In our article we used an optimal bandwidth proposed by Silverman (1986) in the case of normal distributions but more statistical investigations would be necessary to evaluate the bandwidth influence on our non-parametric indices of niche overlap. Moreover some sample sizes used in this study are really low with 6–10 individuals for plants and 3–5 individuals for some fishes. We are aware that some ‘strange’ (multimodal or strongly asymmetric) kernel distributions can emerge from these small samples but any kernel distribution is certainly a better approximation than a normal one in order to evaluate an overlap because its estimation using the normal distribution tends to be an overestimation. In addition we did not detect an extreme or particular overlap value (or variance value) between the two rarest fish species whatever the traits considered (S. pavo and C. labrosus). Nevertheless we do not recommend estimation of overlap for too small sample sizes and further investigations would be useful to detect the influence of sample size (number of individuals) on overlap estimations using the non-parametric indices at the species and the community levels.

With these new indices, niche overlap studies can be extended from the categorical traits such as diet (e.g. Declerck et al. 2002; Albrecht and Gotelli 2001; Shine et al. 2002) to include continuous variables such as functional traits, morphological attributes or environmental conditions, be it in plant or animal ecology. As a consequence, this new family of indices seems suitable to estimate overlap in the beta niche (any environmental filter such as the salinity tolerance), in the alpha niche (any resource acquisition variable such as the trophic level), in the ‘utilitarian’ or the functional niche (any functional trait such as the swimming performance).