1 Introduction

The evolution of a population establishes a link between selected individual characteristics and the environment in which the population lives. Quantifying how the impact of the environment varies along evolutionary trajectories is an important question. Here, we aim at considering how other species interact with the population of interest. These different species compose an ecological community in which each population has a specific role: parasites, predators, resources, etc... The evolution of the different species then modifies the complete interaction network, continuously redefining the selective environment acting on the considered population. The coevolution of different species therefore allows us to consider the feedback loop that links phenotype distributions to environmental variations (Ferrière et al. 2004).

In the present paper, we focus on the case of prey-predator communities evolving on similar time scales. As far as ecological dynamics are concerned, there exists an important literature on such predator-prey interactions. In the 1920’s, Lotka (1926) and Volterra (1926) independently proposed a dynamical system for the ecological dynamics of prey and predators which was then extensively studied (see Takeuchi and Adachi 1983; Hofbauer and Sigmund 1998; Murray 2002). More recently Dieckmann et al. (1995), Marrow et al. (1992, (1996) tackled the question of how natural selection affected the dynamics of such interactions. In the adaptive dynamics framework introduced by Metz et al. (1992), Dieckmann and Law (1996), these authors developed heuristic tools to study the phenotypic coevolution of monomorphic prey and predator populations and its impact on the network. The survival of prey and predators is strongly conditioned on their respective abilities to defend and hunt. As a result, the understanding of the variety of defense traits and of behavioral and morphological adaptation of predators to these defensive mechanisms has become an important focus for evolutionary ecology (see among others Strauss et al. 2002; Müller-Schärer et al. 2004; Lind et al. 2013; Courtois et al. 2012). Considering such coevolutionary dynamics brings up new questions regarding the structure of ecological networks, their stability and the consequences of evolution on their emergent properties (e.g. Loeuille 2010; Dercole et al. 2006). For instance, it has been shown that predator-prey coevolution may yield food-web architectures that resemble the ones observed in empirical datasets (Loeuille and Loreau 2005; Rossberg et al. 2006; Caldarelli et al. 1998; Drossel et al. 2001). Coevolution of predator-prey interactions may also erode the regulating role of predation (Loeuille and Loreau 2004) and change the overall distribution of energy within the community (Loeuille and Loreau 2006). Further models suggest that evolution can select ecological dynamics that are inherently less stable (Loeuille 2010; Ferriere and Gatto 1993; Doebeli and Koella 1995) or more stable (Abrams 2000; Abrams and Matsuda 1997) than initial systems. It is important to note that the importance of coevolution for ecological network dynamics is not restricted to the realm of mathematical models. Indeed, some of the implications of defense evolution in prey for the stability of ecological dynamics have been reproduced experimentally (Yoshida et al. 2003; Meyer et al. 2006). Evolutionary dynamics have also been experimentally reproduced in plant-herbivore systems (Agrawal et al. 2012). Because the importance of eco-evolutionary dynamics of predator-prey interactions now relies on a strong theoretical background and complementary empirical observations or experimental works, evolution is nowadays largely used in terms of applications. To give just an example, the implications of plant-enemy coevolution for the management of agricultural production has been stressed by many (Denison et al. 2003; Thrall et al. 2011; Loeuille et al. 2013).

In a mathematical setting, Durrett and Mayberry (2010) looked into a specific prey-predator community and considered the phenotypic evolution of prey in a fixed community of predators and vice versa under the assumptions of adaptive dynamics (large population, rare and small mutations). They consider a probabilistic microscopic model of the community, following the rigourous approch developed by Champagnat (2006), Champagnat et al. (2006), Champagnat and Méléard (2011) for the eco-evolutionary dynamics of a population with logistic competition.

In this article, we present a stochastic individual-based model for the predator-prey community that evolves as a multi-type birth and death process. The phenotype of an individual is transmitted to its offspring after a potential mutation. The prey phenotypes constrain their defense abilities and influence their reproduction, mortality rate and competition ability. We also consider the evolution of predator phenotypes and model its impact on the predation intensity. We give an example of prey and predator phenotypes in Sect. 2.2 and we illustrate our results with exact simulations of the individual-based process.

We study the stochastic prey-predator community process in different scalings corresponding to the assumptions of adaptive dynamics: large population, rare mutations and mutations of small impact. Since we assume that mutations are rare, it is important to understand the behavior of the community between two mutations. Therefore we study the evolution of a prey-predator community composed of \(d\) prey sub-populations and \(m\) predator sub-populations (Sect. 2). The main question is the composition of this community in a long time scale corresponding to the scale where mutations occur. In the large population limit, the dynamics of the prey-predator community is well approximated by a system of differential equations. In Sect. 3, we study the long time behavior of this deterministic system. In particular, we introduce conditions for the existence and uniqueness of a globally asymptotically stable equilibrium. These conditions rely on specific matrices for the interaction between the species. We improve here a result of Goh, Takeuchi and Adachi (see Goh 1978; Takeuchi and Adachi 1983) in our specific setting. The existence of globally stable equilibria is related to optimization problems called Linear complementarity problems. We consider a class of these problems related to the augmented problems (see Cottle et al. 1992) and extend existing results to our framework.

Then we prove in Sect. 4 that the individual-based stochastic process also converges to this equilibrium in finite time and remains close to this equilibrium on a long time scale. In particular we give a result on the exit time of an attractive domain which remains true even for a perturbed process. Our result is obtained using the properties of the Lyapunov function associated with the deterministic system as in the work of Champagnat et al. (2014). The interest is to highlight the time scale separation between competition phases and mutation occurences. Between two mutations, we can thus characterize the resident prey-predator community.

In Sect. 5, we study the impact of rare mutations on the community. The rare mutation framework was first formalized by Champagnat (2006) for the phenotypic evolution of a population with logistic competition. At each reproduction event, the phenotype of the newborn can be altered by a mutation. We consider the successive invasions of mutants and characterize the survival probability of a mutant trait in a given community. In the mutation scale, we prove that the process jumps from a deterministic equilibrium to another one according to the successive mutant invasions. This jump process extends the polymorphic evolutionary sequence to a co-evolutionary framework.

Finally, we consider the case where mutations have a small impact on phenotypes. Combining these three assumptions (large population, rare mutations and small mutation jumps), we derive a couple of canonical equations describing the coevolution of the prey and predator traits (Champagnat and Méléard 2011; Marrow et al. 1992, 1996).

2 The model

2.1 The microscopic model

We consider an asexual prey-predator community in which each individual is characterized by its phenotypic traits. At each reproduction event the trait of the parent is transmitted to its offspring.

The interest of this work is the coevolution of prey and predator traits that affects the predation. The phenotype \(x\in \fancyscript{X}\) of a prey individual describes its ability to defend itself against predation. We assume that this trait has an effect on the predation intensity that the prey individual undergoes, but also on its reproduction rate, intrinsic death rate, and ability to compete with other prey individuals. Such costs may emerge because the energy allocated to defense is diverted from other functions such as growth, maintenance or reproduction (e.g. Herms and Mattson 1992; Agrawal et al. 2012; Lind et al. 2013). The phenotype \(y\in \fancyscript{Y}\) of a predator characterizes its prey consumption rate. This trait affects the predation exerted on prey but also the death rate of the predator. Again, such costs may be explained by differential allocation among life-history traits, but also by behavioral constraints. For instance, increased consumption rate requiring a larger time investment in resource acquisition, it may decrease the vigilance of the predator against its own enemies, creating a mortality cost (see Illius and Fitzgibbon 1994; Trussell et al. 2006). The trait spaces \(\fancyscript{X}\) and \(\fancyscript{Y}\) are assumed to be compact subsets of \(\mathbb {R}^p\) and \(\mathbb {R}^P\) respectively.

The community is composed of \(d\) prey types \(x_1,\dots ,x_d\) and \(m\) predator types \(y_1,\dots ,y_m\). The state of the community is described by the vector of the sub-population sizes. We introduce a parameter \(K\) scaling these sub-population sizes (as in Fournier and Méléard 2004; Champagnat et al. 2006). To ease the distinction between prey and predator populations we denote by \(N^K_i\) the number of prey individuals with trait \(x_i\), for \(1\le i\le d\), and by \(H^K_l\) the number of predators with trait \(y_l\), for \(1\le l\le m\). Finally the community is represented by the vector

$$\begin{aligned} \mathbf {Z}^K=\frac{1}{K}\bigl (N_1^K,\dots ,N_d^K,H_1^K,\dots ,H^K_m\bigr ), \end{aligned}$$
(1)

of the rescaled numbers of individuals holding the different traits.

The dynamics of the community follows a continuous time multi-type birth and death process. We first describe the behavior of the prey population. Each prey individual with trait \(x\) gives birth to an offspring at rate \(b(x)\). The newborn holds the same trait as its parent. The death rate of a prey individual holding trait \(x\) is given by

$$\begin{aligned} \lambda (x,\mathbf {Z}^K)=d(x)+\sum _{i=1}^d \frac{c(x,x_i)}{K}N^K_i+\sum _{l=1}^m \frac{B(x,y_l)}{K}H^K_l, \end{aligned}$$

where \(d(x)\) is the intrinsic death rate of a prey individual with trait \(x\), \(c(x,x^{\prime })\) the competition exerted by a prey individual with trait \(x^{\prime }\) on the prey individual with trait \(x\) and \(B(x,y)\) the intensity of the predation exerted by a predator holding trait \(y\) on the prey individual with trait \(x\). In the absence of predators, the prey population evolves as a birth and death process with logistic competition whose behavior was extensively studied by Champagnat (2006), Champagnat et al. (2006), Champagnat and Méléard (2011).

For the predator population, each predator holding trait \(y\) gives birth to a new predator at rate

$$\begin{aligned} r\sum _{i=1}^d \frac{B(x_i,y)}{K}N^K_i, \end{aligned}$$

proportional to the predation pressure it exerts on the prey population. The parameter \(r\) can be seen as the conversion efficiency of prey biomass into predator biomass. We assume in the following that \(r<1\). In the absence of prey, the predators are unable to reproduce and their population will become extinct rapidly. Each predator holding trait \(y\) dies at rate \(D(y)\). The competition between different predators is taken into account through the prey consumption.

The interaction between prey and predators affects the prey death rate and the predator birth rate. This interaction benefits predators but penalizes prey. It creates an asymmetry in the community process and makes it difficult to study: comparisons between two processes whose rates are close, are not possible on a long time scale. We will see in the following how to circumvent this difficulty.

2.2 An example introducing two types of defenses

The diversity of defense strategies observed in nature is overwhelming and the maintenance of such a diversity of strategies is an important focus of evolutionary ecology (Ehrlich and Raven 1964). Just focusing on one type of consumption interaction, namely plant-herbivore interactions, strategies of defense are morphological (e.g., through spines or trichomes/hair Zhang et al. 2012), chemical (e.g., the productions of phenols and tannins Becerra et al. 2009) or through the attraction of enemies of herbivores (“crying for help” Kessler and Baldwin 2001). Even when focusing on one defense mechanism, e.g. chemical, the diversity of compounds that are used for defense is very high, not only in total, but even within species (Poelman et al. 2008). Modelling such a diversity is challenging and a broad categorization is necessary. Here, based on previous empirical or experimental works (see Strauss et al. 2002; Müller-Schärer et al. 2004), we propose to consider two major classes of defenses, based on their action mode and on the costs they incur: quantitative defenses and qualitative defenses.

Quantitative defenses correspond to phenotypes that are efficient against a vast number of enemies, but that incur a direct cost in terms of growth or reproduction (Müller-Schärer et al. 2004). Typical examples include structural defenses such as increased toughness (Poorter and Jong 1999), production of morphological defenses (trichomes, spines) (e.g. Agren and Schemske 1994; Mauricio and Rausher 1997) or production of digestibility reducing compounds (Baldwin 1998). In the present work, we assume that the cost of quantitative defenses affects reproduction (cf. Müller-Schärer et al. 2004; Strauss et al. 2002; Lind et al. 2013).

Conversely, qualitative defenses correspond to phenotypes that alleviate consumption by some of the enemies, but incur a cost through another ecological interaction (eg, increased consumption by other enemies or reduced benefits from mutualists Müller-Schärer et al. 2004 ; “ecological costs” sensu Strauss et al. 2002). For instance, alkaloid defenses in plants are efficient against generalist herbivores, but may attract specialists that have evolved to tolerate them or even to use them against their own predators (Müller-Schärer et al. 2004). Other chemical defenses (eg, nicotine) affect the quality of nectar, reducing pollination opportunities (Adler et al. 2012). Floral traits such as color or corolla size may reduce the attraction of herbivores, but at the expense of pollinator visitation (Strauss 1997). In the present work, qualitative defenses allow a reduction in the effect of one predator, but increase the vulnerability to another predator. Because such defense strategies largely impact the similarity of prey niches regarding their enemies (Robinson et al. 2012), we here make the hypothesis that individuals that are closer in terms of qualitative defenses \(x\) have a stronger interference competition. Such an hypothesis is justified by experimental observations (Agrawal et al. 2012), and coherent with the fact that closely related or trait-similar species usually compete more strongly (see Abrams 1983; Burns and Strauss 2011).

We take these two types of defenses into account by associating each prey with a two-dimensional trait \(x=(q_n,q_a)\) where \(q_n\in \mathbb {R}_+\) is the quantity of quantitative defense produced by the prey and \(q_a\in \mathbb {R}\) represents its qualitative defense. The allocative trade-off induced by the quantitative defense \(q_n\) is represented by an exponential decrease of both the prey birth rate and the predation intensity, at speed \(\alpha _n\) and \(\beta _n\) respectively. In simulations, we chose a weak allocative trade-off with \(\alpha _n=1{/}10\) and \(\beta _n=2\): prey can increase their production of defenses without being too penalized.

The predator ability to consume the different qualitative defenses of prey individuals is characterized by two parameters: their preferred qualitative defense \(\rho \), and their degree of generalism \(\sigma \). Specialists predators have a small range \(\sigma \) and exert an important predation pressure on the prey populations holding traits close to their preference, while generalist predators (\(\sigma \) large) consume a large range of qualitative defenses but with less efficiency. Each predator is then represented by the couple \(y=(\rho ,\sigma )\in \mathbb {R}\times ]0,+\infty [\). The predation intensity decreases with the difference \(|\rho -q_a|\) between the preference of predators and the prey qualitative defense. Note that higher generalism incurs a cost in terms of interaction efficiency, as the maximal predation rate is of order \(1{/}\sigma \).

Fig. 1
figure 1

We represent the evolution through time of the respective sizes of the prey sub-populations with trait \(x_1=(0,0.8)\) (

figure a
), \(x_2=(0,1.7)\) (
figure b
) and of the predator population with trait \((\rho , 0.6)\) for different choices of \(\rho \) (
figure c
). Other parameters are \(K=500\), \(b_0=2.5\), \(d_0=0\), \(c_0=1.5\), \(D=0.5\), \(r=0.8\), \(\alpha _n=0.1\), \(\beta _n=2\) (color figure online)

In the simulations, we used the following rate functions: for \((q_n,q_a)\in [0,+\infty [\times \mathbb {R}\) and \((\rho ,\sigma )\in \mathbb {R}\times ]0,+\infty [\):

$$\begin{aligned} \begin{aligned} b(q_n,q_a)&\,=b_0\exp (- \alpha _n q_n), \quad d(q_a,q_n)=d_0,\\ c(q_n,q_a,q_n^{\prime },q_a^{\prime })&=c_0\exp \left( -\frac{(q_a-q_a^{\prime })^2}{2}\right) ,\\ B(q_n,q_a,\rho ,\sigma )&=\exp (- \beta _n q_n)\frac{1}{\sigma }\exp \left( -\frac{(q_a-\rho )^2}{2\sigma ^2} \right) ,\\ D(\rho ,\sigma )&=D. \end{aligned} \end{aligned}$$
(2)

We illustrate this example with exact simulations of the birth and death process introduced above. We are interested in the impact of predators on a prey population using two different qualitative defenses and no quantitative defense: the different prey traits are \(x_1=(0,0.8)\) and \(x_2=(0,1.7)\). We represent on Fig. 1, the evolution through time of the respective sizes of the prey sub-populations with trait \(x_1\) (in green

figure d

), \(x_2\) (in red

figure e

) and of the predator population holding a trait \((\rho , 0.6)\) for different choices of \(\rho \) (in blue

figure f

).

When the predator preference differs too much from the prey defense, their population dies out and the two prey populations coexist. In the sequel, we are interested in the cases where the predator population survives. We observe three different behaviors. In Fig. 1a, the preference of predators is \(\rho =0.2\). The three populations coexist on a long time scale. The prey population holding trait \(x_2\) has more individuals than the prey population with trait \(x_1\) since predation is less important on \(x_2\). In Fig. 1b, the preference of predators is \(\rho =0.7\): predators are well adapted to the trait \(x_1\). The predation intensity is so strong on prey holding trait \(x_1\) that their population die out. However both populations of predators and prey with trait \(x_2\) survive. In Fig. 1c, the preference of predators is \(\rho =1.26\): they consume both prey populations similarly. We observe that the three populations coexist and that both prey sub-populations have similar small size.

As the parameter \(\rho \) increases further, we first observe the extinction of the prey population holding trait \(x_2\). This is the symmetrical case to (b). Then, we observe similarly to case (a) that the three populations coexist.

2.3 Existence of the process and uniform bounds of the community size

The prey-predator community process \(\mathbf {Z}^K=\frac{1}{K}(N^K_1,\ldots ,N^K_d,H^K_1,\ldots ,H^K_d)\) introduced above is a Markov process on \((\mathbb {N}{/}K)^{d+m}\). Its transition rates (or jump rates) are given by the birth and death rates of individuals.

A trajectory of the prey-predator community process can be constructed as solution of a stochastic differential equation driven by Poisson point measures (see Fournier and Méléard 2004; Champagnat et al. 2006). This construction is given in Appendix A.

The community process is well defined up to the explosion of the number of individuals. We denote by \(N^K=\sum _{i=1}^d N_i^K\) the total prey number and by \(H^K=\sum _{l=1}^mH_l^K\) the total number of predators. The prey population size \(N^K\) jumps of \(+1\) each time a prey individual is born and of \(-1\) each time a prey individual dies; the predator population size evolves similarly.

In the sequel we make the following assumptions:

Assumption A

The rate functions \(b\), \(d\), \(c\), \(B\) and \(D\) are continuous, positive and bounded respectively by \(\bar{b}\), \(\bar{d}\), \(\bar{c}\), \(\bar{B}\) and \(\bar{D}\). Moreover the functions \(c\), \(B\) and \(D\) are bounded below by positive real numbers \(\underline{c}\), \(\underline{B}\) and \(\underline{D}\).

Assumption B

The initial condition satisfies \( \sup _K\mathbb {E}\left( \left( \frac{N^K(0)}{K}\right) ^3\!+\!\left( \frac{H^K(0)}{K}\right) ^3\right) {<}\infty \).

The next proposition gives moment properties of the community process and states that the expected population size remains bounded uniformly in \(K\) and \(t\).

Proposition 2.1

Under Assumptions A and B

(i):

For every \(T>0\),

$$\begin{aligned} \sup _K\mathbb {E}\left( \sup _{t\in [0,T]} \left( \frac{N^K(t)}{K}\right) ^3+\left( \frac{H^K(t)}{K} \right) ^3 \right) <\infty . \end{aligned}$$
(ii):

Moreover

$$\begin{aligned} \sup _K\sup _{t\ge 0}\mathbb {E}\left( \left( \frac{N^K(t)}{K}+\frac{H^K(t)}{K}\right) ^2\right) <\infty . \end{aligned}$$

Point (i) justifies the existence of the process \(\mathbf {Z}^K\) for all times and point (ii) will be used to justify convergence results on long time scales. The proof of the Proposition is given in Appendix B.

2.4 Limit in large population

In this section we study the behavior of the community in a large population limit (\(K\rightarrow \infty \)). We use the same scaling for both populations and establish that the stochastic process \(\mathbf {Z}^K\) can be approximated by the solution of a deterministic system of differential equations.

For \(\mathbf {x}=(x_1,\dots ,x_d)\in \fancyscript{X}^d\) and \(\mathbf {y}=(y_1,\dots ,y_m)\in \fancyscript{Y}^m\) we denote by \(LVP(\mathbf {x},\mathbf {y})\) the differential system

$$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle \frac{dn_i(t)}{dt}= n_i(t) \left( b(x_i)-d(x_i)-\sum _{j=1}^dc(x_i,x_j)n_j(t) -\sum _{l=1}^mB(x_i,y_l) h_l(t) \right) ,&{}\quad \forall \, 1\le i\le d,\\ \displaystyle \frac{dh_l(t)}{dt}= h_l(t)\left( r \sum _{i=1}^dB(x_i,y_l) n_i(t)-D(y_l) \right) ,&{}\quad \forall \,1\le l\le m. \end{array}\right. \end{aligned}$$
(3)

A solution of this system is a vector \(\mathbf {z}=(n_1,\dots ,n_d,h_1,\dots ,h_m)\).

Proposition 2.2

Under Assumptions A and B and assuming that the sequence of initial conditions \((\mathbf {Z}^K(0))_K\) converges in probability toward a deterministic vector \(\mathbf {z}(0)\in [0,\infty )^{d+m}\), then for every \(T>0\) the sequence of processes \((\mathbf {Z}^K(t),t\in [0,T])_K\) converges in law in the Skorohod space \(\mathbb {D}([0,T],(\mathbb {R}_+)^{d+m})\) toward the unique function \((\mathbf {z}(t),t\in [0,T])\) solution of the system \(LVP(\mathbf {x},\mathbf {y})\) with initial condition \(\mathbf {z}_0\) and satisfying \(\sup \nolimits _{t\in [0,T]} ||\mathbf {z}(t)|| <\infty \).

The proof follows a classical compactness-uniqueness method developed by Fournier and Méléard (2004, Theorem 5.3). First we prove using Proposition 2.1(i) that the sequence \((\mathbf {Z}^K(t),t\in [0,T])_K\) is tight. Then we identify the limit as the unique solution of the system of differential equations \(LVP(\mathbf {x},\mathbf {y})\).

Remark 1

The extinction of the predator population is not possible in finite time for the solutions of the differential system \(LVP(\mathbf {x},\mathbf {y})\). Indeed, if there exists \(1\le l\le m\) such that \(h_l(0)>0\), then for every \(t\ge 0\),

$$\begin{aligned} \frac{d}{dt} h_l(t) \ge -D(y_l)h_l(t). \end{aligned}$$

Thus \(h_l(t)\ge h_l(0)\exp (-D(y_l)t)>0\).

Conversely, if there is no predator at time \(t=0\), i.e. \(\mathbf {z}(0)=(\mathbf {n}(0),0)\), then the stochastic process \(\mathbf {Z}^K\) converges toward the solution of a competitive Lotka-Volterra system (denoted by \(LVC(\mathbf {x})\)) given by:

$$\begin{aligned} \frac{dn_i(t)}{dt}= n_i(t) \left( b(x_i)-d(x_i)-\sum _{j=1}^dc(x_i,x_j)n_j(t) \right) ,\quad \forall \, 1\le i\le d. \end{aligned}$$
(4)

3 Long time behavior of the solutions of the deterministic system \(LVP\)

In this section we study the long time behavior of the solutions to the \(LVP(\mathbf {x},\mathbf {y})\) system for fixed \(\mathbf {x}=(x_1,\dots ,x_d)\in \fancyscript{X}^d\) and \(\mathbf {y}=(y_1,\dots ,y_m)\in \fancyscript{Y}^m\). To simplify notation, we forget the dependence on traits for the parameters and only use subscripts: for example \(B_{il}=B(x_i,y_l)\).

We are interested in the equilibria of the dynamical system (3). Hofbauer et Sigmund proved (Section 5.4, p. 47 Hofbauer and Sigmund 1998) that the \(LVP(\mathbf {x},\mathbf {y})\) systems satisfy the competitive exclusion principle. This ecological principle states that \(m\) different species cannot survive on fewer than \(m\) different resources (or in less than \(m\) different niches) (see Armstrong and McGehee 1980). An important consequence is that every asymptotically stable equilibrium \(\mathbf {z}^*\) of the \(LVP(\mathbf {x},\mathbf {y})\) system contains no fewer prey sub-populations than of predators:

$$\begin{aligned} \#\{ 1\le i\le d, n^*_i>0\} \ge \#\{ 1\le l\le m, h^*_l>0\}. \end{aligned}$$

Therefore the diversity among predators is limited by the diversity among prey.

In Sect. 3.1, we introduce conditions for an equilibrium to be globally asymptotically stable, (i.e. every solution of the system with positive initial condition converges when \(t\) goes to \(\infty \) toward this equilibrium). This strong notion of stability entails that such an equilibrium is unique. Numerous authors, notably Goh (1978), Takeuchi and Adachi (1983) have already studied this question. We develop here a different approach by improving the Lyapunov function introduced by these authors. The interest of this approach is to obtain quantitative information on the behavior of the stochastic process close to the deterministic equilibrium (see Sect. 4). Then in Sect. 3.2, we study the existence of globally asymptotically stable equilibria. This question is related to the existence of solutions to Linear Complementarity Problems. Combining these two results, we derive conditions that ensure the existence of a unique globally asymptotically stable equilibrium for the \(LVP\) systems.

3.1 Condition for global asymptotic stability

We assume the existence of a non-negative equilibrium \(\mathbf {z}^*=(n^*_1,\dots n^*_d,h^*_1,\dots ,h^*_m)\) of the \(LVP(\mathbf {x},\mathbf {y})\) system defined in (3). We seek conditions on this equilibrium to be globally asymptotically stable. The global stability relies on the properties of the interaction matrix of the system \(LVP\):

$$\begin{aligned} I=\left( \begin{array}{l@{\quad }l} C&{}B\\ -rB^T&{}0 \end{array} \right) , \end{aligned}$$
(5)

where \(C=(c_{ij})_{1\le i,j\le d}\) and \(B=(B_{il})_{1\le i\le d,1\le l\le m}\). We introduce two assumptions on the differential system:

Assumption C

  • C1 For every \(d\in \mathbb {N}\) and almost every \((x_1,\ldots ,x_d)\in \fancyscript{X}^d\), the matrix of the competition among prey \(C(\mathbf {x})=(c(x_i,x_j))_{1\le i,j\le d}\) satisfies that \(C(\mathbf {x})+C(\mathbf {x})^T\) is positive definite.

  • C2 Let \(d,m\in \mathbb {N}\), \(\mathbf {x}=(x_1,\ldots , x_d)\in \fancyscript{X}^d\), and \(\mathbf {y}=(y_1,\ldots , y_m)\in \fancyscript{Y}^m\). Every subsystem of the system \(LVP(\mathbf {x},\mathbf {y})\) is non degenerate.

Assumption C1 allows us to define a Lyapunov function for the system \(LVP\). As an example, Assumption C1 is satisfied for matrices \(C=(c_{ij})\) symmetric and strictly diagonally dominant ( \(|c_{ii}|>\sum _{j\ne i}|c_{ij}|\)). Remark that the competition matrix is symmetric when the competition among preys only depends on the distance between their phenotypes. This is often the case when individuals pay a cost in phenotype matching (Yoder and Nuismer 2010; Burns and Strauss 2011). Strictly diagonally dominant matrices arise when the competition within the sub-populations is more important than the competition with the other sub-populations. This assumption reflects the impact of the similarity of niches of individuals with close phenotypes (Robinson et al. 2012; Burns and Strauss 2011).

Assumption C2 allows to characterize the different equilibria of the \(LVP(\mathbf {x},\mathbf {y})\) system with their null and positive components. This assumption reflects that every sub-population plays a different role in the prey-predator community (and in any sub-commnity).

We associate with the equilibrium \(\mathbf {z}^*\) two subsets containing the subscripts of the traits that disappear in the equilibrium for the prey and predator populations respectively:

$$\begin{aligned} P=\{1\le i\le d, n^*_i=0\}\quad \text { and } \quad Q=\{1\le l\le m, h^*_l=0\}. \end{aligned}$$
(6)

The following proposition states conditions for the global asymptotic stability of an equilibrium.

Proposition 3.1

Let us assume Assumption C and the existence of an equilibrium \(\mathbf {z}^*\) of the system \(LVP(\mathbf {x},\mathbf {y})\) such that

$$\begin{aligned} \left\{ \begin{aligned}&\forall i\in P,\quad b_i-d_i-\sum _{j=1}^dc_{ij}n_j^* -\sum _{l=1}^mB_{il} h_l^*< 0,\\&\forall l\in Q,\quad r \sum _{i=1}^dB_{il} n_i^*-D_l< 0, \end{aligned} \right. \end{aligned}$$
(7)

then this equilibrium is globally asymptotically stable. Moreover such an equilibrium is unique.

Conditions (7) ensure that the equilibrium \(\mathbf {z}^*\) is asymptotically stable. This can be easily obtained by computing the eigenvalues of the Jacobian matrix of the system.

Proof

We define the function

$$\begin{aligned} V(\mathbf {z})=\sum _{i=1}^d r(n_i-n_i^*\log (n_i))+\sum _{l=1}^m (h_l-h_l^*\log (h_l)). \end{aligned}$$
(8)

Using the fact that \(\mathbf {z}^*\) is an equilibrium of the system \(LVP(\mathbf {x},\mathbf {y})\), the derivative of \(V\) along a solution equals

$$\begin{aligned} \begin{aligned} \frac{d}{dt}V(\mathbf {z}(t))&\,=-\frac{r}{2}(\mathbf {n}-\mathbf {n}^*)^T(C+C^T)(\mathbf {n}-\mathbf {n}^*)\\&\quad +r\sum _{i\in P} n_i(b_i-d_i-\sum _{j=1}^dc_{ij}n_j^* -\sum _{l=1}^mB_{il} h_l^*)+\sum _{l\in Q} h_l\left( \sum _{j=1}^drB_{jl} n_j^*-D_l\right) . \end{aligned} \end{aligned}$$
(9)

Since \(\mathbf {z}^*\) satisfies (7) and by C1, the derivative \(\frac{d}{dt}V(\mathbf {z}(t)\) is nonpositive, and vanishes at points \(\bar{z}=(\bar{n}_1,\cdots , \bar{n}_d, \bar{h}_1,\cdots \bar{h_m})\) such that

$$\begin{aligned} \begin{aligned}&\bar{n}_i=n_i^*,\quad \forall \, 1\le i\le d,\\&h_l=0=h^*_l,\quad \forall \, l\in Q. \end{aligned}\end{aligned}$$
(10)

Thus the derivative vanishes not only at point \(z^*\). In the following we search for a function \(W\) and \(\gamma >0\) such that

$$\begin{aligned} L(\mathbf {z})=V(\mathbf {z})+\gamma W(\mathbf {z}) \end{aligned}$$
(11)

is a Lyapunov function for the system: for every solution \((\mathbf {z}(t); t\ge 0)\), the function \(L(\mathbf {z}(t))\) decreases with time and reaches its only minimum at \(\mathbf {z}^*\). We set

$$\begin{aligned} W(\mathbf {z})=\sum _{l=1}^m(h_l-h_l^*)\sum _{i=1}^dB_{il}(n_i-n_i^*). \end{aligned}$$
(12)

Its derivative along a solution is given by:

$$\begin{aligned} \begin{aligned} \frac{d}{dt}{W}(\mathbf {z}(t))&\,= \sum _{l=1}^m h_l r\left( \sum _{i=1}^d B_{il}(n_i-n_i^*) \right) ^2\\&\quad + \sum _{l\in Q} h_l \left( r \sum _{i=1}^d B_{il}n^*_i-D_l\right) \left( \sum _{j=1}^dB_{jl}(n_j-n_j^*)\right) \\&\quad - \sum _{i=1}^d n_i\left( \sum _{l=1}^m B_{il}(h_l-h_l^*) \right) ^2\\&\quad + \sum _{i\in P} n_i \left( b_i-d_i-\sum _{j=1}^dc_{ij}n_j^* -\sum _{l=1}^m B_{il} h_l^* \right) \left( \sum _{k=1}^m B_{ik}(h_k-h_k^*)\right) \\&\quad - \sum _{i=1}^d n_i\sum _{j=1}^d c_{ij}(n_j-n_j^*)\left( \sum _{l=1}^m B_{il}(h_l-h_l^*) \right) . \end{aligned} \end{aligned}$$

The second, third and forth terms are bounded because the solutions of the system are bounded as well. The last term can be bounded by :

$$\begin{aligned} \sum _{i=1}^d n_i\sum _{j=1}^d c_{ij}(n_j-n_j^*)\sum _{l=1}^m B_{il}(h_l-h_l^*)\le & {} \sum _{i=1}^d n_i \left( \frac{\left( \sum _{j=1}^d c_{ij}(n_j-n_j^*)\right) ^2}{\varGamma }\right. \\&\left. \qquad + \varGamma \left( \sum _{l=1}^m B_{il}(h_l-h_l^*) \right) ^2 \right) , \end{aligned}$$

where \(\varGamma \) will be chosen afterwards. Together with Eq. (9) we can upper bound the derivative of \(L\):

$$\begin{aligned} \begin{aligned} \frac{d}{dt}L(\mathbf {z}(t))&\le -(\mathbf {n}-\mathbf {n}^*)^T(U+U^T)(\mathbf {n}-\mathbf {n}^*)- \gamma (1-\varGamma ) \sum _{i=1}^d n_i \left( \sum _{k=1}^m B_{ik}(h_k-h_k^*) \right) ^2,\\&\quad +\sum _{i\in P} n_i\left( b_i-d_i-\sum _{j=1}^dc_{ij}n_j^* -\sum _{l=1}^mB_{il} h_l^*\right) \left( 1+\gamma \sum _{k=1}^m B_{ik}(h_k-h_k^*) \right) \\&\quad +\sum _{l\in Q} h_k\left( \sum _{j=1}^dr B_{jl} n_j^*-D_l\right) \left( 1+\gamma \sum _{i=1}^dB_{il}(n_i-n_i^*)\right) \end{aligned} \end{aligned}$$
(13)

where \(U=(c_{ij}+ \frac{\gamma }{\varGamma }c_{ij}\sum _{u=1}^d n_u+\gamma \sum _{l=1}^mh_lB_{il} B_{jl} )_{1\le i, j\le d}\).

It remains to choose \(\varGamma \) and \(\gamma \). We set \(\varGamma <1\). Since the solution \(\mathbf {z}\) is bounded, it is possible to choose the constant \(\gamma \) such that the matrix \(U+U^T\) is positive definite and

$$\begin{aligned}&1+\gamma \sum _{i=1}^dB_{ik}(n_i-n_i^*)>0 ,\quad \forall 1\le k\le m, \quad \text { and } \\&\quad 1+\gamma \sum _{k=1}^m B_{ik}(h_k-h_k^*)>0, \quad \forall \, 1\le i\le d, \end{aligned}$$

The derivative of \(L( \mathbf {z}(t))\) is then non positive and null for the vectors \((u_1,\dots ,u_d, v_1,\dots ,v_m)\) such that:

$$\begin{aligned} \left\{ \begin{array}{ll} \forall i\in \{1,\ldots ,d\},&{}\quad u_i=n_i^*,\\ \forall l\in Q,&{}\quad v_l=h_l^*=0,\\ \forall i\in \{1,\ldots ,d\},&{} \quad \sum \limits _{l=1}^m B_{il}(v_l-h_l^*) =0.\end{array}\right. \end{aligned}$$

Since \(\mathbf {z}^*\) is an equilibrium, these conditions are equivalent to

$$\begin{aligned} \left\{ \begin{array}{ll} \forall i\in \{1,\ldots ,d\},&{}\quad u_i=n_i^*,\\ \forall l\in Q,&{}\quad v_l=h_l^*=0,\\ \forall i\notin P,&{}\quad b_i-d_i-\sum \limits _{j=1}^d c_{ij}n^*_j -\sum \limits _{l=1}^m B_{il} v_l=0,\end{array}\right. \end{aligned}$$

The vector \((\mathbf {u},\mathbf {v})\) is then an equilibrium \(LVP\) having the same null components as \(\mathbf {z}^*\). Assumption C2 ensures that \((\mathbf {u},\mathbf {v})=\mathbf {z}^*\).

3.2 Existence of globally asymptotically stable equilibria for the system \(LVP\)

The existence of equilibria of the system \(LVP(\mathbf {x},\mathbf {y})\) satisfying (7) is related to the existence of solutions to specific optimization problems called linear complementarity problems (LCP) (see Takeuchi and Adachi 1982).

Definition 1

(Cottle et al. 1992) Given \(M\in \mathbb {R}^{u\times u}\) and \(q\in \mathbb {R}^u\), the linear complementarity problem associated with \((M,q)\) (denoted by \(LCP(M,q)\)) seeks a vector \(z\in \mathbb {R}^u\) satisfying

$$\begin{aligned} \begin{aligned} \forall 1\le j\le u, \quad&z_j\ge 0 \quad \text {and }\quad (Mz+q)_j\ge 0,\\&(Mz+q)^T\cdot z=0. \end{aligned} \end{aligned}$$
(14)

Note that the last condition can be written \((Mz+q)_j z_j=0\), \(\forall 1\le j\le u\).

Let us remark that every equilibrium \(\mathbf {z}^*\in (\mathbb {R}_+)^{d+m}\) of the system \(LVP(\mathbf {x},\mathbf {y})\) satisfying (7) is a solution of \(LCP(I,R)\) where \(u=d+m\), \(I\) is the interaction matrix introduced in (5) and \(R=(-(b_1-d_1),\cdots ,-(b_d-d_d),D_1,\dots ,D_m))^T\) is the vector of the growth rates of the sub-populations. Actually, an equilibrium of the system \(LVP(\mathbf {x},\mathbf {y})\) satisfying (7) is also a solution to \(LCP(\widetilde{I},\widetilde{R})\) where

(15)

We therefore consider a specific range of LCP related to the shape of the interaction matrix \(\widetilde{I}\) which presents a null sub-matrix. The following result derives easily from existing results (see Cottle et al. 1992). We detail the proof in Appendix C.

Theorem 3.2

Let \(M\in \mathbb {R}^{d\times d}\) and \(q\in \mathbb {R}^d\). For every matrix \(B\in (\mathbb {R}_+)^{d\times m}\) and every non-negative vector \(D\in \mathbb {R}^{m}\) we define

(16)

The problem \(LCP(\tilde{M},\tilde{q})\) admits a solution.

Note that a solution \((\mathbf {n},\mathbf {h})\) of \(LCP(\widetilde{I},\widetilde{R})\) is an equilibrium of the \(LVP\) system such that

$$\begin{aligned} \left\{ \begin{array}{ll} \forall 1\le i\le d,&{} \quad \text { if }\, n_i=0 \text { then } b_i-d_i-\sum \limits _{j=1}^dc_{ij}\,n_j -\sum \limits _{l=1}^mB_{il} h_l\le 0,\\ \forall 1\le l\le m,&{}\quad \text { if }\, h_l=0 \text { then }r \sum \limits _{i=1}^dB_{il} n_i-D_l\le 0. \end{array}\right. \end{aligned}$$
(17)

These conditions are similar to conditions (7) for the global asymptotic stability of an equilibrium, but contain large inequalities. Therefore to obtain the existence of globally asymptotically stable equilibria of the \(LVP\) systems we introduce an additional assumption that prevents the quantities involved in conditions (7) and (17) from vanishing. These quantities correspond to the growth rates of prey individuals holding trait \(x_i\) and of predators holding trait \(y_l\) in a community described by the vector \(\mathbf {z}^*\). In ecology these quantities are referred to as invasion fitness. We denote the invasion fitness of a prey individual holding trait \(x\) in a community \(\mathbf {z}^*\) by

$$\begin{aligned} s(x;\mathbf {z}^*)= b(x)-d(x)-\sum _{i=1}^dc(x,x_i)n^*_i-\sum _{k=1}^m B(x,y_k)h^*_k, \quad \forall \, x\in \fancyscript{X}, \end{aligned}$$
(18)

and invasion fitness of a predator holding trait \(y\) in a community \(\mathbf {z}^*\) by

$$\begin{aligned} F(y;\mathbf {z}^*)=\sum _{j=1}^drB(x_j,y) n^*_j-D(y) ,\quad \forall \, y\in Yc. \end{aligned}$$
(19)

Assumption D

For every \((\mathbf {x},\mathbf {y})\in \fancyscript{X}^d\times \fancyscript{Y}^m\), and every vector \((\mathbf {n},\mathbf {h})\) solution of \(LCP(I,R)\), the sets \( \{x^{\prime }\in \fancyscript{X}, s(x^{\prime };(\mathbf {n},\mathbf {h}))=0\} \) and \( \{y^{\prime }\in \fancyscript{Y}, F(y^{\prime };(\mathbf {n},\mathbf {h})) =0\} \) have null Lebesgue measure.

In the following we prove that conditions for survival of a small population can be expressed thanks to the fitness functions \(s\) and \(F\) (we will be interested in the survival of a mutant population). More precisely if a population has a non positive fitness, then it becomes extinct quickly. Otherwise, the population has a chance to invade the resident community. Therefore these fitness functions measure the selective advantage of a trait value in a given community. Assumption D is equivalent to assume that every possible trait has either an advantage or a disadvantage in every stable equilibria of the \(LVP\) system.

Combining Proposition 3.1 and Theorem 3.2 we establish that

Theorem 3.3

Under Assumptions C and D, for almost every \((\mathbf {x},\mathbf {y})\in \fancyscript{X}^d\times \fancyscript{Y}^m\) there exists a unique globally asymptotically stable \(LVP(\mathbf {x},\mathbf {y})\). Moreover this equilibrium satisfies (7).

In the sequel we denote by \(\mathbf {z}^*(\mathbf {x},\mathbf {y})=(\mathbf {n}^*(\mathbf {x},\mathbf {y}),\mathbf {h}^*(\mathbf {x},\mathbf {y}))\) the unique globally asymptotically stable equilibrium of the \(LVP(\mathbf {x},\mathbf {y})\) system. Under the same assumptions we can also establish the existence of a unique globally asymptotically stable equilibrium of the \(LVC\) system introduced in (4). We denote by \(\bar{\mathbf {n}}(\mathbf {x})\) this equilibrium.

4 Consequence for the long time behavior of the stochastic process

Let us fix \(\mathbf {x}\in \fancyscript{X}^d\) and \(\mathbf {y}\in \fancyscript{Y}^m\) and denote by \(\mathbf {z}^*=\mathbf {z}^*(\mathbf {x},\mathbf {y})\) the unique globally asymptotically stable equilibrium of the system \(LVP(\mathbf {x},\mathbf {y})\). In this section we study the long time behavior of the prey-predator community process \(\mathbf {Z}^K\) defined in (1). In Proposition 2.2, we compare the stochastic process with its deterministic approximation on a finite time interval \([0,T]\), however, on longer time scales the stochastic process may exit the neighbourhood of this approximation. We first prove that \(\mathbf {Z}^K\) enters in finite time in a neighbourhood of \(\mathbf {z}^*\). Then, using a probabilistic argument of large deviation, we prove that the trajectory remains in a neighbourhood of \(\mathbf {z}^*\) during a time of order \(\exp (KV)\) for \(V>0\). Finally we study the extinction time of small populations which are not adapted in the community.

For every \(\varepsilon >0\), we denote by \(\fancyscript{B}_{\varepsilon }\) the \(\mathbb {R}^{d+m}\) sphere of radius \(\varepsilon \) centred in \(\mathbf {z}^*\).

Proposition 4.1

Let us assume Assumptions A and B and that the sequence of initial conditions \(\mathbf {Z}^K(0)\) converges in probability toward a deterministic vector \(\mathbf {z}(0)\), then for every \(\varepsilon >0\), there exists \(t_{\varepsilon }>0\) such that

$$\begin{aligned} \lim _{K\rightarrow \infty } \mathbb {P}( \mathbf {Z}^K_{t_{\varepsilon }} \in \fancyscript{B}_{\varepsilon })=1. \end{aligned}$$

Proof

To prove this result we use classical techniques developed in Ethier and Kurtz (1986, Chapter 11, Theorem 2.1) to obtain the convergence in probability uniformly on a time interval of the process \(\mathbf {Z}^K\): \(\forall T>0\), \(\forall \varepsilon >0\)

$$\begin{aligned} \lim _{K\rightarrow \infty } \mathbb {P}\left( \sup _{t\in [0,T]} ||\mathbf {Z}^K(t)-\mathbf {z}(t)||<\varepsilon \right) =1, \end{aligned}$$

where \(\mathbf {z}(t)\) is the solution of \(LVP(\mathbf {x},\mathbf {y})\). The difficulty relies in the fact that the birth and death rates are only locally Lipschitz functions of the state of the process. However, as the limit function \(\mathbf {z}(t)\) takes values in a compact set of \(\mathbb {R}^{d+m}\), we overcome this difficulty by regularizing the birth and death rates outside a sufficient large compact set.

Moreover there exists a compact set \(C\) containing the sequence of initial conditions \((\mathbf {Z}^K(0))_{K\ge 0}\) with probability converging to \(1\). We set for every initial condition \(z_0\in C\) the last time \(t_{\varepsilon }(z_0)\) where the deterministic solution \(\mathbf {z}(t)\) enters \(\fancyscript{B}_{\varepsilon }\). This time is finite according to Theorem 3.3. Since the solutions of the \(LVP(\mathbf {x},\mathbf {y})\) system are continuous with respect to their initial condition, the time \(t_{\varepsilon }=\sup _{z_0\in C} t_{\varepsilon }(z_0)\) is finite and satisfies that \( \forall t>t_{\varepsilon }\), \(\sup _{\{z_0\in C\}} ||\mathbf {z}(t)-z^* ||<\varepsilon . \) Combining these two results, we conclude the proof of Proposition 4.1.

We then study the time spent by \(\mathbf {Z}^K\) in the neighbourhood of \(\mathbf {z}^*\). The estimate of the exit time of an attractive neighbourhood gives a good scaling for the introduction of rare mutations in the next section. This result relies usually on the large deviation theory. However, classical techniques cannot be applied in our setting since the birth and death rates of \(\mathbf {Z}^K\) are not bounded uniformly away from zero. We introduce here a different method which allows to extend the result to perturbations of the process \(\mathbf {Z}^K\). In particular, we aim at considering small mutant populations that interact with the process \(\mathbf {Z}^K\) or at modifying the birth and death rates introduced in Sect. 2. Another interest in considering perturbations of the process is the study of the stability or resilience of this prey-predator network (see the seminal work of May 2001 or Thébault and Fontaine 2010; Ives and Carpenter 2007 for more recent references).

We define a perturbation \(\fancyscript{Z}^K=(\fancyscript{N}^K_1,\ldots ,\fancyscript{N}^K_d,\fancyscript{H}^K_1,\ldots ,\fancyscript{H}^K_m)\) of the process \(\mathbf {Z}^K\) by \(2\) families of \(d+m\) real-valued random processes \((u^K_i)_{1\le i \le d+m}\) and \((v^K_i)_{1\le i \le d+m}\) predictable with respect to the filtration \(\fancyscript{F}_t\) generated by the sequence of processes \(\mathbf {Z}^K\). The sequence \((u^K_i)_{1\le i \le d+m}\) describes the modifications of the birth rates of the prey and the predator populations while the sequence \((v^K_i)_{1\le i \le d+m}\) gives the modifications of the death rates. The modified process evolves as follows:

  • For \(1\le i\le d\), the perturbed prey population \(\fancyscript{N}^K_i\) evolves as a birth and death process with individual birth rate \(b(x_i)+u_i^K(t)\) and individual death rate \(\lambda (x_i,\fancyscript{Z}^K(t))+v^K_i(t)\) at time \(t\).

  • For \(1\le l\le m\), the perturbed predator population \(\fancyscript{H}^K_l\) evolves as a birth and death process with individual birth rate \(r\sum _{i=1}^dB(x_i,y_l)\fancyscript{N}^K_i+u_{d+l}^K(t)\) and individual death rate \(D(y_l)+v^K_{d+l}(t)\) at time \(t\).

In the case where \(u^K_i=v^K_i=0\) for all \(1\le i\le d+m\) the process \(\fancyscript{Z}^K\) is the prey-predator community process \(Z^K\).

We assume that the processes \((u^K_i)_{1\le i \le d+m}\) and \((v^K_i)_{1\le i \le d+m}\) are uniformly bounded by \(\kappa \).

Theorem 4.2

For every \(\varepsilon \) small enough, there exist a constant \(V_{\varepsilon }>0\) and \(\varepsilon ^{\prime \prime }<\varepsilon \) such that if \(\kappa \) is small enough and \(\fancyscript{Z}^K(0)\in \fancyscript{B}_{\varepsilon ^{\prime \prime }}\), then the probability that the process \( (\fancyscript{Z}^K(t); t\ge 0)\) exits the neighbourhood \(\fancyscript{B}_{\varepsilon }\) after a time \(e^{V_{\varepsilon }K}\) converges to \(1\) as \(K\rightarrow \infty \).

The results is obtained using the method developed by Champagnat et al. (2014, Proposition 4.2). We detail the proof in Appendix D and give hereby the main ideas in the non perturbed setting.

Proof

(Ideas of the proof) We recall the definition of \(P\) and \(Q\) in (6) and set

$$\begin{aligned} ||\mathbf {z}-\mathbf {z}^*||_{PQ}=\sum _{i\notin P}|n_i-n_i^*|^2 +\sum _{i\in P}|n_i|+\sum _{l\notin Q}|h_l-h_l^*| +\sum _{l\in Q}|h_l|. \end{aligned}$$

The Lyapunov function \(L\) for the system (3) defined by (11) with an appropriate choice of \(\gamma \) is smooth in the neighbourhood of \(\mathbf {z}^*\). In particular we can define three non negative constants \(C\), \(C^{\prime }\) and \(C^{\prime \prime }\) such that

$$\begin{aligned} \begin{aligned} ||\mathbf {z}-\mathbf {z}^*||^2&\le ||\mathbf {z}-\mathbf {z}^*||_{PQ}\le C\left( L(\mathbf {z})-L(\mathbf {z}^*)\right) \le CC^{\prime } ||\mathbf {z}-\mathbf {z}^*||_{PQ}, \end{aligned} \end{aligned}$$
(20)

and

$$\begin{aligned} \frac{d}{dt}L(\mathbf {z}(t)) \le -C^{\prime \prime } ||\mathbf {z}-\mathbf {z}^*||^2, \end{aligned}$$
(21)

We introduce the stopping time \(\tau ^K_{\varepsilon } =\inf \{t\ge 0, \mathbf {Z}^K\notin \fancyscript{B}_{\varepsilon }\}\). Let \(T\) be a positive time to be chosen afterwards. Thanks to the semi-martingale decomposition of the process \(L(\mathbf {Z}^K(t))\) we can prove that for every \(K\) large enough, there exists \(C^{\prime \prime \prime }>0\) such that for all \( t\le T\wedge \tau _{\varepsilon }^K\):

$$\begin{aligned} ||\mathbf {Z}^K(t)-\mathbf {z}^*||^2\le & {} C \left[ C^{\prime }||\mathbf {Z}^K(0)-\mathbf {z}^*||_{PQ}+ \sup _{[0,T]}|M^K_t|\right. \nonumber \\&\left. -\,C^{\prime \prime }\int _0^t||\mathbf {Z}^K(s)-\mathbf {z}^*||^2-C^{\prime \prime \prime }\frac{1}{K} ds\right] , \end{aligned}$$
(22)

where \(M^K_t\) is a local martingale with zero mean which can be written explicitly using compensated Poisson point measures (see Appendix D).

We define for every \(\kappa >1/K\), \(S_{\kappa }=\inf \{t\ge 0, ||\mathbf {Z}^K(t)-\mathbf {z}^*||^2\le 2C^{\prime \prime \prime }\kappa \}\) and introduce

$$\begin{aligned} T_{\kappa }=\frac{C^{\prime }(||\mathbf {Z}^K(0)-\mathbf {z}^*||_{PQ})+ \sup _{[0,T]}|M^K(t)|}{C^{\prime \prime }C^{\prime \prime \prime }\kappa }, \end{aligned}$$
(23)

which represents the maximal time that the process \(||\mathbf {Z}^K-\mathbf {z}^*||^2\) can spend above the threshold \(2C^{\prime \prime \prime }\kappa \) before the time \(T\wedge \tau ^K_{\varepsilon }\). The inequality (22) becomes for all \(t\le S_{\kappa }\wedge T \wedge \tau _{\varepsilon }^K\)

$$\begin{aligned} ||\mathbf {Z}^K(t)-\mathbf {z}^*||^2\le CC^{\prime \prime }C^{\prime \prime \prime }\kappa T_{\kappa }. \end{aligned}$$

This equation connects the time spent by the process outside a ball, with the values taken by \(||\mathbf {Z}^K-\mathbf {z}^*||^2\) during this time interval. Therefore if we bound the values of \(T_{\kappa }\), we control the process \(||\mathbf {Z}^K-\mathbf {z}^*||^2\) and consequently the exit time \(\tau ^K_{\varepsilon }\). To estimate \(T_{\kappa }\) we need to control exponentially the values of the martingale \(M^K_t\) uniformly on a time interval. To this aim, we use the following lemma.

Lemma 1

(Graham and Méléard 1997, Proposition 4.1) For every \(\alpha >0\) and \(T>0\) there exists a constant \(V_{\alpha ,T}\) satisfying that for all \(K\) large enough:

$$\begin{aligned} \mathbb {P}\left( \sup _{[0,T\wedge \tau _{\varepsilon }^K]} |M^K_t| >\alpha \right) \le \exp (-KV_{\alpha ,T}) \end{aligned}$$

With this result and (22) we study for \(\varepsilon ^{\prime \prime }<\varepsilon ^{\prime }<\varepsilon \), the number of back and forth, \(k_{\varepsilon }\) between the balls \(\fancyscript{B}_{\varepsilon ^{\prime \prime }}\) and \(\fancyscript{B}_{\varepsilon ^{\prime }}\) before the exit of \(\fancyscript{B}_{\varepsilon }\). With an appropriate choice of the parameters \(\varepsilon ^{\prime }\) and \(T\), we establish that \(k_{\varepsilon }\) is smaller than a geometric random variable with parameter \(\exp (-KV)\), thus

$$\begin{aligned} \mathbb {P}(k_{\varepsilon }>\exp (KV/2))=1-(1-\exp (-KV))^{\exp (KV/2)} \underset{K\rightarrow \infty }{\longrightarrow } 1 \end{aligned}$$

To conclude it remains to show, using (22) again, that these back and forth require a time of order 1.

Finally we study the behavior of the process while it remains close to the equilibrium \(\mathbf {z}^*\). The equilibrium \(\mathbf {z}^*\) can have zero components and we establish that the associated stochastic sub-populations become extinct in a time of order \(\log K\). We introduce the stopping time

$$\begin{aligned} S^K_{ext}=\inf \{t\ge 0, \forall i\in P, N^K_i(t)=0 \quad \text { and }\quad \forall \, l\in Q, H^K_l(t)=0\}, \end{aligned}$$

and set \(S^K_{ext}=0\) if both \(P\) and \(Q\) are empty.

Proposition 4.3

Let \(\varepsilon >\varepsilon ^{\prime \prime }>0\) small enough. If the initial condition \(\mathbf {Z}^K(0)\in \fancyscript{B}_{\varepsilon ^{\prime \prime }}\),then there exists \(a>0\) such that

$$\begin{aligned}\lim _{k\rightarrow \infty } \mathbb {P}(S^K_{ext}\le a\log K) =1.\end{aligned}$$

Proof

Fix \(l\in Q\). We prove the result for the predator population holding trait \(y_l\), and the same reasoning can be applied to a prey population holding trait \(x_i\), for \(i\in P\) (see Theorem 4 in Champagnat 2006).

Theorem 3.3 ensures that the fitness \(F(y_l;\mathbf {z}^*)\) is negative. We define the constant \(V_{\varepsilon }\) associated by Theorem 4.2 to the exit time \(\tau ^K_{\varepsilon }\) of the ball \(\fancyscript{B}_{\varepsilon }\). For every \(t\le \tau ^K_{\varepsilon }\), the number of predators \(H^K_l(t)\) is bounded from below by a continuous time birth and death process \(H\) with birth rate \(\lambda =r\sum _{i=1}^dB(x_i,y_l) (n^*_i+\varepsilon )\), death rate \(\mu =D(y_l)\) and initial condition \(H^K_l(0)\le K\varepsilon ^{\prime \prime }\). We choose \(\varepsilon \) small enough for the process \(H\) to be sub-critical: \(\varepsilon < -F(y_l,\mathbf {z}^*)/(r\sum _{i=1}^dB(x_i,y_l))\).

From classical results on branching processes (see Athreya and Ney 2004, p. 109), we obtain that

$$\begin{aligned} \mathbb {P}(H(t)=0|H(0)=1 )=1-\frac{\mu -\lambda }{\mu \exp (-(\lambda -\mu )t) + \lambda }. \end{aligned}$$

Since \(\forall h_0\in \mathbb {N}\), \(\mathbb {P}(H(t)>0|H(0)=h_0)=1-\mathbb {P}(H(t)=0|H(0)=1)^{h_0}\), we deduce that for every initial condition \(0\le h_0\le K\varepsilon ^{\prime \prime }\),

$$\begin{aligned} \mathbb {P}(H(t)>0|H(0)=h_0)\le 1- \left( 1-\frac{\mu -\lambda }{\mu \exp (-(\lambda -\mu )t) + \lambda }\right) ^{K\varepsilon ^{\prime \prime }}. \end{aligned}$$

We set \(1>\delta >0\) and apply the previous inequality to the positive time \(t_K^l= (\frac{\delta -1}{\lambda -\mu })\log (K)\). We obtain that \(\forall 0\le h_0\le K\varepsilon ^{\prime \prime }\),

$$\begin{aligned} \mathbb {P}(H(t_K^l)>0|H(0)=h_0)\le 1- \left( 1-\frac{\mu -\lambda }{\mu K^{1-\delta } + \lambda }\right) ^{K\varepsilon ^{\prime \prime }}\underset{K\rightarrow \infty }{\longrightarrow } 0. \end{aligned}$$

We conclude the proof by choosing \(a\log K\) as the maximal \(t^l_K\) for \(l\in Q\cup P\).

5 Evolution of the process in a rare mutation time scale

In this section, mutations happen during the prey and predator reproduction events. We observe their impact on the dynamics of the community. The coevolution of the traits depends on the occurrence of mutations and the invasion of the mutant population. We seek conditions for the survival of a mutant population and study the consequences of the fixation of a mutation for the prey-predator community.

The individual birth and death rates are defined as in Sect. 2. The mutation events are added as follows

  • when a prey individual with trait \(x\) gives birth, the trait of its offspring is affected by a mutation with probability \(u_Kp(x)\). The newborn holds a trait \(x+l\) where \(l\) is distributed according to \(\pi (x,l)dl\). Otherwise (with probability \(1-u_Kp(x)\)) the newborn inherits its parent trait \(x\).

  • Similarly for each predator holding a trait \(y\). At each reproduction event, with probability \(u_KP(y)\) the trait of the offspring is affected by a mutation: it holds the trait \(y+l\) where \(l\) is distributed according to \(\varPi (y,l)dl\). Otherwise the newborn inherits its parent trait \(y\).

The same parameter \(u_K\) scales the mutation frequencies in both prey and predator populations. This assumption is consistent with the fact that the demographic dynamics of both populations happens on the same time scale (Sect. 4). When the parameter \(u_K\) is small, the mutations are rare. We assume in the sequel that \(Ku_K\rightarrow 0\) as \(K\rightarrow \infty \). This assumption measures the rarity of the mutations and is consistent with the theory of adaptive dynamics (Metz et al. 1992; Dieckmann and Law 1996).

In Sect. 5.1 we illustrate the impact of mutations on the example introduced in Sect. 2.2. In Sect. 5.2 we consider the limit of the community process under the assumptions of infinite population and rare mutations. We extend the results obtained by Champagnat (2006) to the prey-predator coevolution. Finally in Sect. 5.3 we consider a limit when the mutation steps are small. We prove that the coevolution of the prey and predator traits can be described by the deterministic coupled system of differential equations introduced by Marrow et al. (1996). This system extends the canonical equation of adaptive dynamics to the coevolution of a prey-predator interaction.

5.1 Simulations

Let us consider again the example introduced in Sect. 2.2 in which prey individuals are characterized by a trait \(x=(q_n,q_a)\) where \(q_n\) is the quantity of quantitative defenses they produce and \(q_a\) the type of qualitative defense they use. The predators are characterized by \(y=(\rho ,\sigma )\) where \(\rho \) reflects the qualitative value they prefer and \(\sigma \) is their range. The mutations are distributed according to gaussian distributions, centred in the trait of the parent with covariance matrices \(\gamma \) and \(\varGamma \) for prey and predators respectively.

We illustrate in different cases the impact of mutations on the community. We will observe the convergence on the rare mutation scale toward a pure jump process taking values in the set of couples of finite measures on the trait spaces \(\fancyscript{X}\) and \(\fancyscript{Y}\) respectively.

Fig. 2
figure 2

a Represent the traits \(q_a\) (

figure g
,
figure h
,
figure i
,
figure j
) and \(\rho \) (\(\times \)) present in the community through time. b Gives the dynamics of the number of predators on the time interval \([0:800]\) and c gives the dynamics of size of the prey populations holding trait \((0.3,0.4)\) in green, \((0.3,0.664)\) in blue and \((0.3,1.285)\) in pink (the same colors are associated on a). The vertical line corresponds to the extinction time of predators. The other parameters are \(K=1000\), \(u_K=5\times 10^{-5}\), \(p=1\), \(P=0\), \(\pi (q_a,l)\sim \fancyscript{N}(q_a,0.1)\) \(b_0=2\), \(d_0=0\), \(c_0=1.5\), \(D=0.5\), \(r=0.8\), \(\alpha _n=0.1\), \(\beta _n=2\) (color figure online)

Fig. 3
figure 3

a Represents the traits \(q_a\) (

figure k
) and \(\rho \) (\(\times \),
figure l
,
figure m
,
figure n
) present in the community through time. b Gives the dynamics of the rescaled number of predators holding trait \((0.2,0.6)\) in black, \((0.339,0.6)\) in green and \((0.531,0.6)\) in pink and \((0.597,0.6)\) in blue. c Represents the rescaled size of the prey population through time. The other parameters are \(K=1000\), \(u_K=1\cdot 10^{-4}\), \(p=0\), \(P=1\), \(\varPi (\rho ,l)\sim \fancyscript{N}(\rho , 0.01)\), \(b_0=2\), \(d_0=0\), \(c_0=1.5\), \(D=0.5\), \(r=0.8\), \(\alpha _n=0.1\), \(\beta _n=2\) (color figure online)

Fig. 4
figure 4

Both Figures represent the traits \(q_a\) (

figure o
) and \(\rho \) (\(\times \)) present in the community through time. The mutation probabilities vary: on a \(p=P=1\), \(\pi (q_a,\cdot )\sim \fancyscript{N}(q_a,0.1)\), \(\varPi (\rho ,\cdot )\sim \fancyscript{N}(\rho ,0.1)\), and on b \(P=5\), \(p=1\) \(\pi (q_a,\cdot )\sim \fancyscript{N}(q_a,0.01)\), \(\varPi (\rho ,\cdot )\sim \fancyscript{N}(\rho ,0.01)\). The other parameters are \(K=1000\), \(u_K=10^{-4}\), \(b_0=2\), \(d_0=0\), \(c_0=1.5\), \(D=0.5\), \(r=0.8\), \(\alpha _n=0.1\), \(\beta _n=2\) (color figure online)

5.1.1 Co-evolution of the qualitative defense \(q_A\) and the predator preference \(\rho \)

We first consider the coevolution of the prey trait \(q_a\) and of the predator trait \(\rho \). Both traits are associated through the predation function \(B\), and the defense trait \(q_a\) influences the competition among prey. In these simulations we assume that mutations do not affect the prey trait \(q_n\) and the predator trait \(\sigma \). We consider three cases: first we assume that no mutation occurs in the predator population (Fig. 2), then the opposite case where mutations only occur in the predator population (Fig. 3), finally we study the coevolution of the traits (Fig. 4).

In the first case we assume that no mutation occurs in the predator population: \(P=0\). The initial community is composed of \(K\) prey individuals holding trait \(x=(0.3,0.4)\) and \(K\) predators holding trait \(y=(0.2,0.6)\). The mutation probability \(u_K=5 \times 10^{-5}\) is small. Figure 2a gives the different values of \(q_a\) carried by prey and of \(\rho \) carried by predators for all times. We observe that natural selection favours the values of \(q_a\) far from \(\rho \). The predator population dies out when the defense \(q_a\) gets to far away from their preference. The extinction time is represented by a vertical line on the three graphs. As long as predators are present in the community, we observe that the prey traits are concentrated in a single value: the prey population remains monomorphic.

In the other graphs, we focus on the demographic dynamics. Figure 2b gives the dynamics of the number of predators through time. On Fig. 2c we represent the size of the prey sub-populations with the following traits: the initial trait value \((0.3,0.4)\) in green, \((0.3,0.664)\) in blue, and \((0.3,1.285)\) in pink (the same colors are used on Fig. 2a). On these graphs we observe the impact of the mutations on the community. The mutation \((0.3, 0.664)\) is the first to invade the initial community and to replace the resident prey holding trait \((0.3,0.4)\). We observe that before the appearance of this mutation the respective numbers of predators and prey \((0.3,0.4)\) remain stationary. Some mutations have appeared but their population remained small (less than 10 individuals). This phenomenon illustrates the stationarity of the prey and predator population sizes near the deterministic equilibria of the \(LVP\) system stated in Theorem 4.2.

The invasion of the mutation \((0.3,0.664)\) is characterized by a fast extinction of the resident prey population and a fast growth of the mutant population. Meanwhile, the number of predators diminishes to another stationary value. The extinction speed of the resident population is given by Proposition 4.3.

The invasion of a mutant prey holding trait \((0.3,1.285)\) in the resident community composed of prey holding trait \((0.3,0.664)\) and predators, drives the predators to extinction. The extinction of predators is a direct consequence of the prey phenotypic evolution: it is called an evolutionary murder (see Dercole et al. 2006). Afterwards both prey populations survive. Note that their respective population sizes are similar: they have indeed the same natural birth and death rates and similar ability for competition. In this simulation, the prey population remains dimorphic after the predator extinction and both traits are driven apart by the competition.

This simulation is characteristic of the behavior of the process when the population is large and mutations are rare. As introduced by Champagnat (2006) there exist two phases: a long phase where the sizes of the sub-populations remain stable, close to the equilibrium values of the deterministic system; a short phase corresponding to the invasion of a mutant trait in the resident population. The successive mutant invasions induce jumps in the traits present in the community as well as in respective sizes of each sub-population. We describe this jump process in Sect. 5.2

We then consider the opposite case where mutations only affect the predator preference \(\rho \) and not the prey population (see Fig. 3). As before, Fig. 3a represents the traits \(q_a\) and \(\rho \) in the population. Figure 3b corresponds to the rescaled number of predators holding the traits \((0.2,0.6)\) in black, \((0.339,0.6)\) in green, \((0.531,0.6)\) in pink and \((0.597,0.6)\) in blue (represented with the same colors on Fig. 3a). The rescaled size of the prey population is drawn on Fig. 3c. The initial population is composed of \(K\) prey individuals with trait \((0.3,0.6)\) and \(K\) predators with trait \((0.2,0.6)\). We recall that the predator preference corresponds to the value of the qualitative defense that they can avoid or the prey type that they are specifically able to consume (see Müller-Schärer et al. 2004; Courtois et al. 2012). Predators whose preference \(\rho \) is closer to the prey qualitative defense \(q_a=0.6\) have an advantage in terms of relative fitness. We observe that the predator population remains monomorphic and that the trait jumps closer to \(q_a\) accordingly to the successive invasions of mutants. At each invasion, the sizes of the prey and predator populations jump to the stable equilibrium of the associated \(LVP\) system. The last invasion phase is very slow (see Fig. 3b). It is due to a very slow convergence toward the equilibrium, of the solutions to the \(LVP\) system associated with the traits \(x=(0.3,0.6)\), \(y_1=(0.531,0.6)\) and \(y_2=(0.597,0.6)\). We observe in a general manner that the invasion times of successive mutations increase as \(\rho \) comes closer to \(q_a\). This reflects the flatenning of the fitness landscape for predators: through time, advantageous mutations become less beneficial with respect to the resident population.

To observe coevolution, we introduce mutations in both the prey and the predator populations. The prey evolution is constrained by two forces: the intra-specific competition that favours diversification and the predation pressure that drives prey phenotypes away from the predator preferences. We investigate the effect of these two forces on the community when the relative mutation speeds \(p\) and \(P\) vary. On Fig. 4, we represent the traits \(q_a\) (

figure p

) and \(\rho \) (\(\times \)) present in the community through time. On Fig. 4a \(p=P\), we observe that the predator trait jumps close to the value of the defense of the prey population. Afterwards, the prey population becomes polymorphic. This diversity is due to the competition interaction. Finally, as predators do not adapt their preference fast enough, their population dies out. In this case, the competitive force has more impact than the predation pressure and induces a diversification of the prey phenotypes (see Loeuille et al. 2002). On Fig. 4b, we raise the mutation probability of predators: \(P=5p\) and choose smaller mutations steps. We observe two phases: in the first one (for \(t\in [0:4000]\)) the distance between the prey qualitative defense and preference of predators decreases. After this time, both traits seem to evolve simultaneously. This phenomenon recalls the Red Queen or Arm races observed by biologists (see Marrow et al. 1992; Abrams 2000; Dercole et al. 2006; Becerra et al. 2009), which corresponds to a parallel variation of the traits of partner species in time.

5.1.2 Evolution of the quantitative defense

We now model the variations in the quantity \(q_n\) of quantitative defense. Unlike the qualitative defenses considered above, quantitative defenses impact the prey birth rate and not their competitive ability. In these simulations the mutations do not affect the prey trait \(q_a\) and the mutation probability of predators is null again. The initial community is composed of \(K\) prey individuals holding trait \((0,0.6)\) and of \(K\) predators holding trait \((0.2,0.6)\). Figure 5a represents the traits \(q_n\) borne by prey through time. Figure 5b gives the dynamics of the rescaled sizes of the prey sub-populations associated with the initial trait \((0,0.6)\) in red, \((0.189,0.6)\) in green, \((0.311,0.6)\) in blue, \((0.703,0.6)\) in pink and \((0.260,0.6)\) in light blue. These traits are represented using the same colors on Fig. 5a. The remaining traits, in black on Fig. 5a, correspond to mutations which did not invade the community. The dynamics of the rescaled number of predators is given on Fig. 5c. The vertical line corresponds to the predator extinction.

Note that the quantity of defense produced by prey increases in the presence of predators and that the number of predators decreases when prey increase their defenses. When prey holding trait \((0.311, 0.6)\) and \((0.703,0.6)\) coexist, the number of predators decreases quickly. We observe long time oscillations that correspond to the behavior of the dynamical systems associated to these three populations. As the competition is constant in the prey population, these simulations do not enter the mathematical framework we described (Assumption C1). These oscillations illustrate that evolution can induce instability in the interaction networks (e.g. Loeuille 2010). After the extinction of predators, prey producing many defenses are penalized because their reproduction is weaker. The direction of natural selection changes with the extinction of the predators. We observe here what is called apparent competition: the coexistence of two prey traits with predators relies on the fact that the predation pressure is stronger on the most competitive prey population (see Armstrong and McGehee 1980).

This change in the direction of evolution illustrates a new difficulty induced by coevolution: the same mutation will not have the same impact on the community depending on the presence or the absence of predators. It is thus necessary to consider the coevolution of both populations.

Fig. 5
figure 5

a Gives the values of \(q_n\) borne by prey through time. On b we draw the rescaled sizes of the prey population holding trait \((0,0.6)\) in red, \((0.189,0.6)\) in green, \((0.311,0.6)\) in blue, \((0.703,0.6)\) in pink and \((0.260,0.6)\) in light blue. The dynamics of the rescaled number of predators is given on c. Other parameters are given by \(K=1000\), \(u_K= 10^{-4}\), \(p=1\), \(P=0\), \(\pi (q_n,l)\sim \fancyscript{N}(q_n,0.1)\) \(b_0=2\), \(d_0=0\), \(c_0=1.5\), \(D=0.5\), \(r=0.8\), \(\alpha _n=0.1\), \(\beta _n=2\) (color figure online)

5.2 Limit in the rare mutation time scale and jump process

We consider the limit of the community process in a large population scaling with rare mutations. The number of traits present in the community varies when mutations appear in the community. We represent the community by a couple of empirical measures \((\nu ^K(t), \eta ^K(t))\):

$$\begin{aligned} \nu ^K(t)=\frac{1}{K}\sum _{i=1}^{N^K(t)}\delta _{x_i}, \quad \eta ^K(t)=\frac{1}{K}\sum _{l=1}^{H^K(t)}\delta _{y_l}, \end{aligned}$$

where \(\delta _x\) is the Dirac measure at point \(x\). This process takes values in the set \(\fancyscript{M}_F(\fancyscript{X})\times \fancyscript{M}_F(\fancyscript{Y})\) of couples of finite measures on \(\fancyscript{X}\) and \( \fancyscript{Y}\) respectively.

We recall that the mutation frequencies in both populations are scaled by a parameter \(u_K\) such that \(Ku_K\rightarrow 0\). This assumption is consistent with the adaptive dynamics framework in which mutations occur when the resident population is at equilibrium (Metz et al. 1992; Dieckmann and Law 1996). Further assumptions will be given in Theorem 5.1 on the exact scaling of the mutation frequency.

The fact that the mutation frequency decreases with the population size is not unexpected, considering population genetics arguments. Indeed, the genetic variation among a population increases with respect to the number of individuals. However, assuming that mutant effects are distributed around 0 with a given variance, large numbers of mutations in large populations eventually produce very similar mutants. Due to this redundancy, the amount of variation produced by the mutation process saturates in large populations (see Frankham 1996; Soulé 1976; Leimu et al. 2006). This saturation can be interpreted as a decrease of the outcome of new mutants.

The next proposition states that mutations cannot occur in a bounded time interval.

Lemma 2

Let us assume Assumptions A, B and that the mutation densities satisfy:

$$\begin{aligned} \begin{aligned}&\forall x\in \fancyscript{X}, \forall u\in \mathbb {R}^p,\quad \pi (x,u)\le \bar{m}(u),\quad \int _{\mathbb {R}^p}\bar{m}(u)du<\infty ,\\&\forall y\in \fancyscript{Y}, \forall v\in \mathbb {R}^P,\quad \varPi (y,v)\le \bar{M}(v),\quad \int _{\mathbb {R}^P}\bar{M}(v)dv<\infty , \end{aligned} \end{aligned}$$
(24)

then for every \(\delta >0\), there exists \(\varepsilon >0\) such that for all \(t>0\),

$$\begin{aligned} \limsup _{K\rightarrow \infty }\mathbb {P}\left( \text {a mutation occurs in }\left[ \frac{t}{Ku_K},\frac{t+\varepsilon }{Ku_K}\right] \right) \le \delta . \end{aligned}$$

The proof of this Lemma can be easily adapted from the proof of Corollary 2.2 in Champagnat et al. (2014). It is based on a coupling of the community process before the first mutation time with a multi-type birth and death process whose birth and death rates depend on \(u_K\). This process is independent of the mutation events occurring in \((\nu ^K,\eta ^K)\). As these mutations occur at a rate proportional to \(Ku_K\), the probability to observe a mutation in a time interval of length \(\varepsilon {/}Ku_K\) is negligible.

We state the main result of this section. It describes the convergence of the community process in the mutational scale toward a pure jump process. This process extends the polymorphic evolutionary sequence introduced by Champagnat and Méléard (2011) to a prey-predator network.

As we have seen in the simulations, the limiting process takes values in the set of the stable equilibria \((\mathbf {n}^*(\mathbf {x},\mathbf {y}), \mathbf {h}^*(\mathbf {x},\mathbf {y}))\) of the deterministic system \(LVP(\mathbf {x},\mathbf {y})\) (introduced in (3) for \(\mathbf {x}\in \fancyscript{X}^d\) and \(\mathbf {y}\in \fancyscript{Y}^m\)) as long as predators survive. Remark that after the extinction of predators, the behavior of the prey population is well known (see Champagnat 2006; Champagnat and Méléard 2011) and the limiting process takes values in the set of equilibria \(\bar{\mathbf {n}}(\mathbf {x})\) of the \(LVC(\mathbf {x})\) system defined in (4).

The process describing the successive states of the community is a Markovian jump process \(\varLambda =(\varLambda ^1,\varLambda ^2)\) taking values in \(\fancyscript{E}\):

$$\begin{aligned} \fancyscript{E}= & {} \left\{ \left( \sum _{i=1}^dn_i\delta _{x_i},\sum _{l=1}^mh_l\delta _{y_l}\right) ; \mathbf {x}\in \fancyscript{X}^d,\mathbf {y}\in \fancyscript{Y}^m,\right. \\&\left. \qquad (\mathbf {n},\mathbf {h})\in \left\{ (\mathbf {n}^*(\mathbf {x},\mathbf {y}), \mathbf {h}^*(\mathbf {x},\mathbf {y})), ({\bar{\mathbf {n}}}(\mathbf {x}),0)\right\} \right\} . \end{aligned}$$

The dynamics of \(\varLambda \) depends on the arrivals of mutations in the prey and the predator populations. A successful mutant invasion modifies both the prey and the predator populations (see Figs. 2, 3, 5). From any state \((\sum _{i=1}^dn^*_i(\mathbf {x},\mathbf {y})\delta _{x_i},\sum _{l=1}^mh^*_k(\mathbf {x},\mathbf {y})\delta _{y_l})\) where predators are alive

  • for every \(j\in \{1,\ldots ,d\}\) the process jumps to the equilibrium associated with the modified vector of traits \(((\mathbf {x},x_j+u),\mathbf {y})\) at infinitesimal rate:

    $$\begin{aligned} p(x_j)n^*_j(\mathbf {x},\mathbf {y})b(x_j)\frac{[s(x_j+u;(\mathbf {n}^*(\mathbf {x},\mathbf {y}), \mathbf {h}^*(\mathbf {x},\mathbf {y}))]_+}{b(x_j+u)}\pi (x_j,u)du. \end{aligned}$$

    This corresponds to the invasion of a mutant prey population with trait \(x_j+u\) in the community.

  • for every \(k\in \{1,\ldots ,m\}\) the process jumps to the equilibrium associated with the modified vector of traits \((\mathbf {x},(\mathbf {y},y_k+v))\) at infinitesimal rate:

    $$\begin{aligned}&P(y_k)h^*_k(\mathbf {x},\mathbf {y})\left( \sum _{i=1}^drB(x_i,y_k)n^*_i(\mathbf {x},\mathbf {y})\right) \\&\quad \times \frac{[F(y_k+v;(\mathbf {n}^*(\mathbf {x},\mathbf {y}), \mathbf {h}^*(\mathbf {x},\mathbf {y}))]_+}{\sum _{i=1}^drB(x_i,y_k+v)n^*_i(\mathbf {x},\mathbf {y})}\varPi (y_k,v)dv. \end{aligned}$$

    This corresponds to the invasion of a predator population holding the mutant trait \(y_k+v\).

We recall that the fitness functions \(s\) and \(F\) are defined in (18) and (19) respectively.

Remark 2

As in Figs. 2 and 5, the community jump process \(\varLambda \) can reach a state where the predator population dies out. Since the invasion of mutant predators requires the positivity of their invasion fitness (see Theorem 3.3), the predator extinction can only result from the invasion of a mutant prey which diminishes the growth rate of the resident predator. The behavior of the community after the predator extinction is described by the PES introduced in Theorem 2.7 Champagnat and Méléard (2011). We recall that the infinitesimal jump rate from a state \((\sum _{i=1}^d\bar{n}_i(\mathbf {x},\mathbf {y})\delta _{x_i},0)\) to \((\sum _{i=1}^d\bar{n}_i((\mathbf {x},x_j+u))\delta _{x_i}+\bar{n}_{d+1}((\mathbf {x},x_j+u))\delta _{x_j+u}),0), \) is given by

$$\begin{aligned} p(x_j)b(x_j)\bar{n}_j(\mathbf {x})\frac{[s(x_j+u;(\bar{\mathbf {n}}(\mathbf {x}),0)]_+}{b(x_j+u)}\pi (x_j,u)du. \end{aligned}$$

We now formulate the limiting theorem.

Theorem 5.1

Fix \(\mathbf {x}\in \fancyscript{X}^d\) and \(\mathbf {y}\in \fancyscript{Y}^m\). Let us assume Assumptions A, B, C, D, (24) and that the initial condition \((\sum _{i=1}^{d}n^K_i\delta _{x_i},\sum _{l=1}^m h^K_l\delta _{y_l})\) converges in probability toward \( (\sum _{i=1}^{d}n^*_i\delta _{x_i},\sum _{l=1}^m h^*_l\delta _{y_l})\). If furthermore

$$\begin{aligned} \log (K)\ll \frac{1}{Ku_K} \ll \exp (VK), \quad \forall \, V>0, \end{aligned}$$
(25)

then the process \(\bigl (\nu ^K(\frac{t}{Ku_K}),\eta ^K(\frac{t}{Ku_K})\bigr )_{t\ge 0}\) converges toward the pure jump process \(\varLambda =((\varLambda ^1_t,\varLambda ^2_t);t\ge 0)\) defined above and whose initial condition is given by \((\sum _{i=1}^dn^*_i(\mathbf {x},\mathbf {y})\delta _{x_i},\sum _{l=1}^mh^*_l(\mathbf {x},\mathbf {y})\delta _{y_l}).\)

This convergence takes place in the sense of convergence of the finite dimensional distributions for the topology on \(\fancyscript{M}_F(\fancyscript{X}\times \fancyscript{Y})\) induced by the total variation norm.

Assumption (25), introduced by Champagnat (2006), reflects the separation between the demographic and the mutational time scales (see Figs. 2b, c, 3b, c). The demographic time scale is of order \(\log K\). It corresponds to the evolution of the stochastic process close to its deterministic approximation. The process \(\mathbf {Z}^K\) enters a neighbourhood of the attractive deterministic equilibrium and the deleterious traits die out (Propositions 4.1 and 4.3). The mean time between two mutations is of order \(1/Ku_K\), therefore the resident population is close to the equilibrium of the associated \(LVP\) system when a mutant appears in the community (Theorem 4.2).

The proof derives from the proof of Theorem 1 in Champagnat (2006) and from the results obtained in Sect. 4. The main idea is to study the invasion of a mutant trait in the community. Starting from an initial condition \((\sum _{i=1}^dn^*_i(\mathbf {x},\mathbf {y})\delta _{x_i},\sum _{l=1}^mh^*_l(\mathbf {x},\mathbf {y})\delta _{y_l})\) at the deterministic equilibrium, the next mutation occurs after an exponential time of parameter

$$\begin{aligned} E(\mathbf {x},\mathbf {y})=\sum _{i=1}^d p(x_i)b(x_i)n^*_i(\mathbf {x},\mathbf {y})+\sum _{l=1}^mP(y_l)h^*_l(\mathbf {x},\mathbf {y})\left( r\sum _{i=1}^d B(x_i,y_l)n^*_i(\mathbf {x},\mathbf {y})\right) . \end{aligned}$$

The mutant individual then comes from the prey population with trait \(x_j\) (\(1\le j \le d\)) with probability

$$\begin{aligned} \frac{p(x_j)b(x_j)n^*_j(\mathbf {x},\mathbf {y})}{E(\mathbf {x},\mathbf {y})}, \end{aligned}$$

or from the population of predators holding trait \(y_k\) (\(1\le k\le m\)) with probability

$$\begin{aligned} \frac{P(y_k)h^*_k(\mathbf {x},\mathbf {y})\sum _{i=1}^drB(x_i,y_k)n^*_i(\mathbf {x},\mathbf {y})}{E(\mathbf {x},\mathbf {y})}. \end{aligned}$$

In the sequel we consider a mutant trait \(y_k+v\) where \(v\) is distributed according to \(\varPi (y_k,v)dv\). While the number of individuals holding the mutant trait is small, we compare thanks to Theorem 4.2 the size of the mutant population with a continuous time birth and death process with birth rate \(r\sum _{i=1}^d B(x_i,y_k+v)n^*_i(\mathbf {x},\mathbf {y})\) and death rate \(D(y_k+v)\). Its growth rate is then given by the invasion fitness \(F(y_k+v; (\mathbf {n}^*(\mathbf {x},\mathbf {y}),\mathbf {h}^*(\mathbf {x},\mathbf {y})))\) of the mutant trait in the resident population. If the fitness is negative we prove that the mutant population goes extinct similarly as in Lemma 4.3. Otherwise, the probability that the mutant population reaches a positive density \(\varepsilon \) is close to the survival probability of the supercritical branching process which is given by

$$\begin{aligned} \frac{F(y_k+v; (\mathbf {n}^*(\mathbf {x},\mathbf {y}),\mathbf {h}^*(\mathbf {x},\mathbf {y})))}{r\sum _{i=1}^d B(x_i,y_k+v)n^*_i(\mathbf {x},\mathbf {y})}, \end{aligned}$$

(see Athreya and Ney 2004, p. 102). Moreover this phase lasts a time of order \(\log K\) (see the proof of Lemma 3 in Champagnat 2006). Then using the large population approximation on a finite time interval (Proposition 2.2), we establish that the process \(\varLambda \) jumps to the equilibrium of the system \(LVP(\mathbf {x},(\mathbf {y},y_k+v))\).

5.3 Small mutations: a canonical equations system for coevolution

In this subsection we consider a different scaling for the jump process \(\varLambda \) where the mutation steps of both populations are of order \(\varepsilon \) (see Champagnat and Méléard 2011; Champagnat et al. 2014). We study the limit of the sequence \(\varLambda _{\varepsilon }\) in a long time scale \(\frac{t}{\varepsilon ^2}\) to observe global evolutionary dynamics. We establish that the limiting behavior of the prey and predator traits satisfies a coupled system of differential equations. These equations were heuristically introduced by Marrow et al. (1996). They extend the canonical equation of adaptive dynamics to the coevolution of a prey and predator interaction.

In the sequel we assume that every couple of a prey and a predator trait can coexist although two prey traits cannot coexist.

Assumption E

  1. (a)

    For every \((x,y)\in \fancyscript{X}\times \fancyscript{Y}\), predators survive in the equilibrium of \(LVP(x,y)\):

    $$\begin{aligned} \frac{b(x)-d(x)}{c(x,x)}>\frac{D(y)}{rB(x,y)}. \end{aligned}$$
    (26)
  2. (b)

    Invasion implies fixation: For every \((x,x^{\prime })\in \fancyscript{X}^2\) and \(y\in \fancyscript{Y}\), we have

    $$\begin{aligned} \begin{aligned}&s(x^{\prime };(n^*(x,y),h^*(x,y)))<0,\\ \text {or }&s(x^{\prime };(n^*(x,y),h^*(x,y)))>0\quad \mathrm{and }\quad s(x;(n^*(x^{\prime },y),h^*(x^{\prime },y)))<0. \end{aligned} \end{aligned}$$
  3. (c)

    The mutation densities \(\pi \) and \(\varPi \) are Lipschitz continuous on \(\fancyscript{X}\times \mathbb {R}^p\) and \(\fancyscript{Y}\times \mathbb {R}^P\).

  4. (d)

    The functions \(g\) and \(G\) defined for \(x,x^{\prime }\in \fancyscript{X}\)   and   \(y,y^{\prime }\in \fancyscript{Y}\) by

    $$\begin{aligned} \begin{aligned} g(x^{\prime };(x,y))&\,=p(x)n^*(x,y)b(x)\frac{s(x^{\prime };(x,y))}{b(x^{\prime })},\\ G(y^{\prime };(x,y))&\,=P(y)h^*(x,y)B(x,y)\frac{F(y^{\prime };(x,y))}{B(x,y^{\prime })}, \end{aligned} \end{aligned}$$
    (27)

    are continuous and \(\fancyscript{C}^1\) with respect to their first variable.

Remark 3

Condition (26) compares the equilibrium sizes of the prey populations evolving in the presence or the absence of predators. The predator survival requires that the prey population size decreases in the presence of predators.

For every couple of traits \((x,y)\), the equilibrium \((n^*(x,y),h^*(x,y))\) of the system \(LVP(x,y)\) given by Theorem 3.3 equals

$$\begin{aligned} n^*(x,y)=\frac{D(y)}{rB(x,y)} \quad {\text { and }}\quad h^*(x,y)=\frac{1}{B(x,y)}\left( b(x)-d(x)-c(x,x)\frac{D(y)}{rB(x,y)}\right) . \end{aligned}$$
(28)

To ease notation, we denote in this section

$$\begin{aligned} \begin{aligned} s(x^{\prime };(x,y))&=s(x^{\prime };(n^*(x,y),h^*(x,y))),\\ F(y^{\prime };(x,y))&=F(y^{\prime };(n^*(x,y),h^*(x,y))). \end{aligned} \end{aligned}$$

Assumption E.b) and Theorem 3.3 entail that two prey types cannot coexist in the equilibrium of the deterministic system \(LVP\). Together with the competitive exclusion principle introduced in Sect. 3, this ensures that two predator populations cannot coexist either. Therefore each mutant invasion (prey or predator) leads to the replacement of the resident trait. The community is then always composed of a monomorphic prey population and a monomorphic predator population:

$$\begin{aligned} \varLambda ^1_{\varepsilon }(t)=n^*(X_{\varepsilon }(t),Y_{\varepsilon }(t))\delta _{X_{\varepsilon }(t)},\quad \varLambda ^2_{\varepsilon }(t)=h^*(X_{\varepsilon }(t),Y_{\varepsilon }(t))\delta _{Y_{\varepsilon }(t)}. \end{aligned}$$
(29)

The trait process \((X_{\varepsilon }(t),Y_{\varepsilon }(t))\) is a Markovian jump process taking values in \(\fancyscript{X}\times \fancyscript{Y}\) whose infinitesimal generator is given for any measurable bounded function \(\phi \) by

$$\begin{aligned} \begin{aligned} \fancyscript{L}_{\varepsilon }\varPhi (x,y)&\,= \int _{\fancyscript{X}} \Bigl (\varPhi (x+\varepsilon u,y)-\varPhi (x,y)\Bigr ) [g(x+\varepsilon u;(x,y))]_+\pi (x,u)du\\&\quad \,+\int _{\fancyscript{Y}} \Bigl (\varPhi (x,y+\varepsilon v)-\varPhi (x,y)\Bigr )[G(y+\varepsilon v;(x,y))]_+\varPi (y,v)dv. \end{aligned} \end{aligned}$$

The following Theorem states the limiting behavior of the process \((X(t{/}\varepsilon ^2),Y(t{/}\varepsilon ^2))\) as \(\varepsilon \) goes to \(0\). The proof relies on a classical compactness-uniqueness argument that can be immediately extended from Champagnat and Méléard (2011) (Appendix C).

Theorem 5.2

Let us assume Assumptions A, B, C, D and E and that the sequence of initial conditions \((X_{\varepsilon }(0), Y_{\varepsilon }(0))\) is bounded in \(\mathbb {L}^2\) and converges in law toward a deterministic vector \((x_0,y_0)\), then for every \(T>0\) the process \((X(t{/}\varepsilon ^2),Y(t{/}\varepsilon ^2))\) converges in law in \(\mathbb {D}([0,T],\fancyscript{X}\times \fancyscript{Y})\) toward a couple of deterministic functions \((x(t),y(t))_{t\in [0,T]}\) unique solution of the system of differential equations

$$\begin{aligned} \left\{ \begin{aligned}&\frac{d}{dt}x(t)=\int _{R^p} u[u \cdot \nabla _1 g(x(t);(x(t),y(t)))]_+ \pi (x(t),u) du, \\&\frac{d}{dt}y(t)=\int _{R^P} v[v \cdot \nabla _1 G(y(t);(x(t),y(t)))]_+ \varPi (y(t),v) dv, \end{aligned} \right. \end{aligned}$$
(30)

with initial condition \((x_0,y_0)\).

This system is strongly coupled through the functions \(g\) and \(G\).

In the specific case where the mutation measures \(\pi \) and \(\varPi \) are symmetrical, with covariance matrices \(\gamma \) and \(\varGamma \), the system (30) becomes

$$\begin{aligned} \left\{ \begin{aligned}&\frac{d}{dt}x(t)= \frac{1}{2}p(x)\gamma (x)n^*(x,y) \nabla _1 s(x;(x,y)) , \\&\frac{d}{dt}y(t)=\frac{1}{2} P(y)\varGamma (y) h^*(x,y)\nabla _1 F(y;(x,y)). \end{aligned} \right. \end{aligned}$$

Remark 4

In the large population limit with rare and small mutations, diversification events of the population can be observed. These evolutionary branching are well understood in the case of the evolution of a single population (see Champagnat and Méléard 2011; Champagnat et al. 2014). They rely on the behavior of the jump process \(\varLambda \) when coexistence of two traits occurs. The prey-predator coevolution make the evolutionary branching properties of the trait processes complex to study. In particular, if two prey traits coexist, the next mutation can lead to the coexistence of two predator traits as well.

5.3.1 Application

We apply those results to the example introduced in Sect. 2.2 where prey individuals hold a trait \((q_n,q_a)\in \mathbb {R}\times \mathbb {R}_+\) and predators a trait \((\rho ,\sigma )\in \mathbb {R}\times \mathbb {R}_+\). We recall that the rate functions are given in (2) and that the mutation measures are gaussian with respective variance \(\gamma \) and \(\varGamma \).

Derivating the fitness functions with respect to the mutant trait, we obtain

$$\begin{aligned} \begin{aligned} \nabla _1s((q_n,q_a);(q_n,q_a,\rho ,\sigma ))=&\left( \begin{array}{c} \left( -\alpha _n b_0\exp (-\alpha _n q_n)+\beta _n\frac{h^*(q_n,q_a,\rho ,\sigma )}{B(q_n,q_a,\rho ,\sigma )}\right) \mathbf {1}_{q_n>0}\\ \frac{q_a-\rho }{\sigma ^2}\frac{h^*(q_n,q_a,\rho ,\sigma )}{B(q_n,q_a,\rho ,\sigma )} \end{array} \right) . \end{aligned} \end{aligned}$$
(31)

and

$$\begin{aligned} \nabla _1F((\rho ,\sigma );(q_n,q_a,\rho ,\sigma ))= \left( \begin{array}{c} \frac{q_a-\rho }{\sigma ^2}D\\ \left( \frac{(q_a-\rho )^2}{\sigma ^3}-\frac{1}{\sigma }\right) D\mathbf {1}_{\sigma >0} \end{array} \right) \end{aligned}$$
(32)

In particular \(\nabla _1F((\rho ,\sigma );(q_n,q_a,\rho ,\sigma ))=0\) if and only if \(\rho =q_a\) and \(\sigma = |q_a-\rho |\).

We first study the coevolution of the traits \(q_a\) and \(\rho \), the values of \(\sigma \) and \(q_n\) being fixed, as in Fig. 4. The system of differential equations governing the dynamics of \(q_a\) and \(\rho \) is then

$$\begin{aligned} \begin{aligned} \frac{d}{dt}q_a(t)&=\frac{\gamma \pi (q_a)}{r} \phi (q_a,\rho )\\ \frac{d}{dt}\rho (t)&=\varGamma \varPi (\rho )\phi (q_a,\rho ), \end{aligned}\end{aligned}$$
(33)

where

$$\begin{aligned} \phi (q_a,\rho )=\frac{D(q_a-\rho )}{\sigma ^2}h^*(q_a,\rho ), \end{aligned}$$

and the equilibrium \(h^*(q_a,\rho )\) is given in (28). The function \(\phi \) vanishes if \(q_a=\rho \) or if the predator population dies out (\(h^*(q_a,\rho )=0\)).

We deduce from the specific form of the system that for all \(t\ge 0\), \(h^*(q_a(t),q_n(t))>0\). Moreover there exist three cases depending on the respective values of the mutation probabilities and variances and on the parameter \(r\):

  • If \(r\varGamma \varPi (\rho ) >\gamma \pi (q_a)\), the difference \(|q_a(t)-\rho (t)|\) decreases with time. This phenomena was observed on Fig. 4b on the first part of the graph.

  • If \(r\varGamma \varPi (\rho )=\gamma \pi (q_a)\), both derivatives are equal for all times. The evolution then follows an arm race dynamics : both traits evolve continuously and \(|q_a(t)-\rho (t)|\) remains constant (see Marrow et al. 1992; Abrams 2000; Dercole et al. 2006).

  • If \(r\varGamma \varPi (\rho )<\gamma \pi (q_a)\), prey escape the predator influence as the distance between \(q_a\) and \(\rho \) increases. When \(t\rightarrow \infty \), the solution converges toward a vector \((q_a^*,\rho ^*)\) that doesn’t satisfy (26). However the extinction of the predator population is not possible in finite time unlike in the process \(\varLambda \) (see Fig. 4a)

Then we consider, as in Fig. 5, the prey strategies for the quantitative defense \(q_n\), when the other traits are not affected by mutations:

$$\begin{aligned} \frac{d}{dt}q_n(t)=\pi (q_n)\gamma \frac{D}{rB(q_n)}\bigl ( (\alpha _n-\beta _n)b_0\exp (-\alpha _n q_n)-\beta _n c_0\frac{D}{rB(q_n)}\bigr )\mathbf {1}_{q_n>0} \end{aligned}$$

In the case where \(\alpha _n\ge \beta _n\), meaning that the allocative trade-off between producing a large quantity of defense and having a good reproduction is important, the quantitative defense \(q_n(t)\) decreases to \(0\). If \(\alpha _n<\beta _n\), the derivative of \(q_n\) vanishes at the point

$$\begin{aligned} q_n^*= \frac{-1}{\alpha _n+\beta _n} \ln \left( \frac{\beta _n c_0 D}{rB(0)(\beta _n-\alpha _n)b_0}\right) . \end{aligned}$$

Then either \(q_n^*\) is negative and again \(q_n(t)\rightarrow 0\) or \(q_n^*\ge 0\) and \(q_n(t)\) converges to \(q_n^*\) when \(t\rightarrow \infty \). With the parameters of Fig. 5, \(q_n^*\approx 0.58\). We observed first an increase of \(q_n\) and then the extinction of predators. Thus, an important question is whether or not the predator population dies out as \(t\rightarrow \infty \). An easy calculation gives that

$$\begin{aligned} h^*(q_n^*)>0 \iff \frac{\beta _n}{\beta _n-\alpha _n}>1, \end{aligned}$$

which is always true if \(0<\alpha _n<\beta _n\). We deduce that the evolution of the quantitative defense does not drive the predators to extinction. This prediction contradicts the extinction observed in Fig. 5. However, in this simulation Assumption E.b is not satisfied and the predator extinction is due to the coexistence of two prey types.

6 Discussion

We introduced three different objects to describe the prey-predator community: a deterministic system \(LVP\) in (3), a stochastic jump process \(\varLambda \) in Sect. 5 and a couple of two canonical equations in (30). These processes correspond to three different limits of the individual based process introduced in Sect. 2.

The jump process \(\varLambda \) describes the dynamics of the community when mutations are rare. It describes the successive equilibria of the community. In this sense, it justifies a simulation method developped in Ecology to study the phenotypic evolution of communities (see Loeuille and Loreau 2005; Loeuille and Leibold 2008; Brännström et al. 2011). In these articles, the community evolves as the solution of a system of differential equations. Each equation of the system describes the dynamics of a sub-population. When a mutation occurs (at a very low rate), it increases the number of sub-populations and thus a new equation is added to the system. Their method gives the successive equilibria of the community similarly to the jump process \(\varLambda \), however, it does not take into account the demographic stochasticity as every mutant with a positive fitness invades the community.

Our model highlights the implications of coevolutionary dynamics for the ecological dynamics of the community and its maintenance in time (see Sect. 5.1). Particularly, we show that such consequences depend on the trait under scrutinity and on the costs that are associated to these traits. For instance, the two categories of defenses have different implications in this regard. If the evolution of qualitative defenses is fast enough, it can lead to the disappearance of the predators as in Figs. 2 and 4, a phenomenon called “evolutionary murders” (as the evolution of a species in the community eventually kills another species). We note that such evolutionary murders do not happen when one considers the evolution of quantitative defenses. Likewise, the evolution of predators does not lead to the extinction of prey. Therefore, our study highlights how evolutionary murder phenomenons, already known in ecology (Brännström et al. 2012; Georgelin et al. 2015; Dercole et al. 2006) depend on evolving species and on types of traits that evolve. Even in the absence of species extinction, we note that the coevolution also modifies the strength of the interactions between species and can thus lead to the reinforcement of an interaction (see Fig. 3). Or as observed in Fig. 2, evolution can induce the disappearance of an interaction (through diminishing the competition between two plants). Interactions then progressively weaken and become “ghosts from the past”, as commonly observed in phylogenetic or evolutionary studies (e.g. Tobias et al. 2013; Bennett et al. 2013).

Such variations in interaction strength can have important consequences for the overall stability of the system. Indeed, in food webs, stability analyses suggest that distributions of interaction strengths including weak interactions have a stabilizing effect on the dynamics of the community (McCann et al. 1998) with important implications for the conservation of species and for the delivery of ecosystem services. The question of the links between evolution and stability of the network is therefore crucial. As shown in Fig. 5, evolution can induce instability in the network so that small perturbations of a population may lead to the extinction of one or several populations (cf Loeuille 2010).

The jump process contains the various behaviors present in ecological communities however we only have little predictive information on the composition of the community at all times. Therefore it can be interesting to consider the canonical system (30). This process represents the dynamics of the traits present in the community under strong assumptions on the small size of the mutation steps and on the non-coexistence of different traits of prey and predators. The strong influence of prey on predators and vice versa can be well understood when we consider the equilibria of this system. We only consider one-dimensional traits (\(p=P=1\)). If we consider the specific case of an equilibrium \((x^*,y^*)\) such that

$$\begin{aligned} \partial _1s(x^*;(x^*,y^*))=0\quad \text { and } \quad \partial _1F(y^*;(x^*,y^*))=0 \end{aligned}$$
(34)

This equilibrium corresponds to a two-dimensional version of the Evolutionary strategies introduced in Metz et al. (1996) for the one-dimensional canonical equation. A natural question about this equilibrium is a condition for its stability. The Jacobian matrix at a point \((x,y)\) is given by:

$$\begin{aligned} \left( \begin{array}{ll} n^*(x,y)(\partial _{11}s(x;(x,y))+\partial _{12}s(x;(x,y))) &{}\quad n^*(x,y)(\partial _{13}s(x;(x,y))\\ h^*(x,y)\partial _{12}F(y;(x,y)) &{}\quad h^*(x,y)(\partial _{11}F(y;(x,y))) +\partial _{13}F(y;(x,y)) )\\ \end{array}\right) \end{aligned}$$
(35)

Note that the conditions

$$\begin{aligned} \partial _{11}s(x^*;(x^*,y^*))+\partial _{12}s(x^*;(x^*,y^*)<0 , \end{aligned}$$

and

$$\begin{aligned} \partial _{11}F(y^*;(x^*,y^*))) +\partial _{13}F(y^*;(x^*,y^*)) <0, \end{aligned}$$

are not sufficient nor necessary to ensure the stability of the equilibrium \((x^*,y^*)\). These two conditions correspond to the local stability of the equilibrium \(x^*\) when we consider the evolution of the prey trait in the presence of a fixed predator trait \(y^*\) and conversely for the evolution of the predator trait in the presence of prey individuals holding the fixed trait \(x^*\) (see Champagnat and Méléard 2011).

The branching properties of the community are complex to study. Indeed, they rely on a precise study of the jump process \(\varLambda \) after the first coexistence of two traits. As we have seen in the simulations, the coexistence in the prey population can lead to extinction of predators (see Figs. 2, 5), or to the coexistence of different trait of predators (see Fig. 4b).

Throughout this work we considered the same time scales for both prey and predators. Note that while this hypothesis of similar evolutionary time scales allows a first grasp on the effects of coevolution on the ecological dynamics of such interactions, strong asymmetries actually occur in nature. Taking again the example of plant-herbivore interactions, large asymmetries of demographic and evolutionary time scales can arise when the two partners have large differences in terms of body size and generation time (eg, tree-insect interactions such as Robinson et al. 2012 or, at the other extreme, grass-large herbivore interactions Bakker et al. 2006). We will consider such asymmetries of time scales in a future work.