1 Introduction

Evolutionary games have been useful in modelling reproductive successes of different types of species (individuals) based on their traits and interactions with other types in the population. The fitness of different species types depends on relative proportions (frequencies) of types, on the specific type of interactions, and on the environment. Traditionally these models were deterministic (see Hofbauer and Sigmund 1998) and formulated in terms of a system of ordinary differential equations (replicator equation) that describes the evolution of species frequencies. The underlying assumption in deterministic models is that populations are infinite and that the role of fluctuations in determining long term dynamics is negligible.

Over the last decade stochastic versions of evolutionary games were used (Nowak et al. 2004; Taylor et al. 2005; Imhof and Nowak 2006; Traulsen et al. 2005) to model the behaviour in finite populations and to reveal the effects caused by fluctuations in the long term dynamics of populations. In the limit as the population size grows some aspects of stochastic models are well approximated by the dynamics of its mean, but the role of fluctuations is crucial for describing events that happen on a very long time scale, such as fixation or extinction. For example, in Moran type models final outcomes are not always determined by the same conditions as in deterministic dynamics. There are examples in which selection completely favours one or the other species for finite values of the population size where the deterministic dynamics has both species as evolutionarily stable. (For recent reviews of both deterministic and stochastic evolutionary game modelling see Sandholm 2007; Traulsen and Hauert 2009).

In stochastic evolutionary models one is particularly interested in chances of reaching different equilibrium or absorption points (extinction and fixation probabilities), as well as the amount of time needed to reach them (first extinction and fixation times). When the number of types is greater than two both deterministic and stochastic behaviour becomes much more complex and difficult to quantify. For individual-based (Markov chain) models one needs to solve a system of difference equations which, for large population sizes, is numerically intensive. One can use stochastic differential equations (diffusions) to approximate Markov chain dynamics in large finite populations (e.g. Traulsen 2012) to simplify some of these computations. For example, calculating the probability of fixation via the optimal stopping theorem would then require solving PDEs with appropriately prescribed boundary values. The validity of a diffusion approximation depends on the characteristics of the individual-based models as well as on the quantities one wishes to compute. For example, calculating the time to fixation involves events in which the population values are very near the boundary of the state space for the system and the approximation will not be accurate for types whose size is very small. A diffusion approximation is only valid after a careful rigorous justification.

Specifics of the individual-based model affect the type of diffusion that is appropriate. We focus on a model in which the approximating diffusion is degenerate (its fluctuations are negligible in certain directions) and hence the diffusion space is of lower dimension than the original system. A rigorous justification and the derivation of the coefficients for such a diffusion are more complex than in the standard case, but long term behaviour of such degenerate diffusions is much simpler to quantify. Such an approximation is a useful tool for model reduction and eliminates the need for time consuming simulations.

There is also an important body of literature examining how random fluctuations due to demographic and environmental stochasticity can contribute to persistence and coexistence in populations during a long transient period of time that precedes the ultimate extinction. (For an excellent survey of recent mathematical results see Schreiber 2017). One can characterize transient meta-stable behaviour of a Markov chain model by quasi-stationary distributions (Cattiaux et al. 2009). When the model is such that stochastic effects can be represented as random perturbation of a dynamical system with a small noise coefficient, then, under additional assumptions on the system, these quasi-stationary distributions concentrate on positive attractors of deterministic dynamics and the probability of extinction decays exponentially with population size (Faure and Schreiber 2014).

The model we consider does not fit into this framework, though it has the same behaviour in its initial time period. However, on a longer time scale our model can with a non-negligible probability (not decaying exponentially with population size) leave the positive attractors of the deterministic dynamics, and can no longer be described as a random perturbation of a dynamical system. We complement the aforementioned body of work (Schreiber 2017; Faure and Schreiber 2014) by presenting a model whose behaviour in the transient period can be approximated by a diffusion of lower dimension (living on a stable manifold of positive attractors for the deterministic dynamics), and by describing a technique that approximates the length of the transient period and the probabilities of extinction. Our model explores the case in which stochasticity is responsible for fixation after a long periods of coexistence, and where the proportions of types during coexistence are constrained to a lower dimensional subspace. On this subspace its long term behaviour is similar to Moran models with selection whose long term limits can be approximated by a diffusion (Durrett 2008, Ch. 7.4, 7.12), although a direct comparison of replacement rates and resulting probabilities of fixation is, due to our dimension reduction to a submanifold, analytically intractable.

Here we present a simple evolutionary game model in which the environment affects inter-species dynamics. The environment is introduced as altering the reproductive fitness of each species type (so the relative fitness of species are different in different environments). We further let the environment itself be stochastic. Our model considers three species competing in an environment which has two possible states. Two of the species are specialists for the two environments (advantageous in one environment but disadvantaged in the other), and the third is a generalist (indifferent to the environment). We consider a stationary independent environment (i.i.d. at all reproduction-competition times), and we express long term properties of the model in terms of the environment statistics. The effect of the random environment results in long-term stability of a fixed linear combination of the two specialist species, while fluctuating in their proportion relative to the generalist species. This form of stability in a coexistence region allows us to use a degenerate (reduced dimension) diffusion approximating the stochastic dynamics of the system. This allows us to calculate the first extinction probabilities and upper bound the expected time to this event, and to quantify the effect of the environment on the long term dynamics.

2 Stochastic evolutionary game model

2.1 Individual based model in a random environment

Our model is a version of the frequency dependent Moran model with three different species and a constant large population size N. Each individual lives for an Exponentially distributed (with rate 1) amount of time at the end of which it is replaced by an individual whose type is chosen at random with chance proportional to its fitness in the current environment. The state of the environment is random from one of two possible states.

To illustrate the simple model idea let us denote the three species as C, H and M, and the environment states as c and h. The two species C and H are specially adapted to the environments c and h respectively. The third species M is a generalist which is equally adapted to both of the environments but with less of an advantage than the specialist in either environment. More precisely, C has fitness 1 in the environment c it is specialized for and 0 in the other environment h, H has fitness 1 in the environment h and 0 in c, while M has fitness 1/2 in both c and h. The environment has probability q of being in state c and \(1-q\) of being in state h.

The random environment is assumed to fluctuate over time, independently over time steps and according to the same distribution of state c versus state h. In other words, at each time when a species is chosen to die and be replaced by another, the environment affecting the fitness of different species is chosen independently as \((\mathbf{c}, \mathbf{h})\) with distribution \((q, 1-q)\).

The stochastic dynamics of this evolution game model can be represented by a continuous-time Markov chain \((X_t) _{t\ge 0}=(C_t,H_t,M_t) _{t\ge 0}\) on the state space \(\{0,1,\ldots ,N\}^3\) subject to the constraint of a constant population size \(C_t\,+\,H_t\,+\,M_t=N\), \(\forall t\ge 0\). The Markov chain has three absorbing states (N, 0, 0), (0, N, 0), (0, 0, N), corresponding to the fixation of any of the species. Outside of the absorbing states the transition kernel is specified as follows. Let

$$\begin{aligned} \mu _c(X)=C, \;\mu _H(X)=H, \;\mu _M(X)=M \end{aligned}$$

denote the density dependent probabilities at which individuals of different species are chosen to die, i.e. be replaced, and let

$$\begin{aligned} \lambda _C(X)&=q\frac{2C}{(2C+M)},\quad \lambda _H(X)=(1-q)\frac{2H}{(2H+M)},\\ \lambda _M(X)&=q\frac{M}{(2C+M)}+(1-q)\frac{M}{(2H+M)} \end{aligned}$$

denote the environment dependent rates at which they are chosen to be born, i.e be the replacement (the fitness takes into account the state of the environment and the environment specific fitnesst in each state); the possible jumps of X and their rates are given by

\(\varDelta (C,H,M)\)

Jump rate

(1,0,−1)

\(\lambda _C(X)\mu _M(X)\)

(1,−1,0)

\(\lambda _C(X)\mu _H(X)\)

(0,1,−1)

\(\lambda _H(X)\mu _M(X)\)

(−1,1,0)

\(\lambda _H(X)\mu _C(X)\)

(0,−1,1)

\(\lambda _M(X)\mu _H(X)\)

(−1,0,1)

\(\lambda _M(X)\mu _C(X)\)

with no change in the process \(\varDelta (C,H,M)=(0,0,0)\) occurring with rate \(\lambda _C(X)\mu _C(X)+\lambda _H(X)\mu _H(X)+\lambda _M(X)\mu _M(X)\).

Due to the constraint on constant population size the process X is only two dimensional. If \(q=\frac{1}{2}\) there is an obvious symmetry between the two specialists C and H. For any \(q\in (0,1)\) it is useful to let \(D=C-H\) and work with the process \(Y=(D,M)\) on \(\{-N,\ldots , N\}\times \{0,1,\ldots ,N\}\) subject to the constraints \(M+D\le N, M-D\le N\). The process Y is also Markov with absorbing states \((N,0),(-N,0),(0,N)\) and transitions rates expressed explicitly in terms of D and M by

\(\varDelta (D,M)\)

Jump rate

(1,−1)

\(\frac{qM(N+D-M)}{(N+D)}\)

(2,0)

\(\frac{q(N+D-M)(N-D-M)}{2(N+D)}\)

(−1,−1)

\(\frac{(1-q)M(N-D-M)}{(N-D)}\)

(−2,0)

\(\frac{(1-q)(N+D-M)(N-D-M)}{2(N-D)}\)

(1,1)

\(\frac{M(N-D-M)(N+D-2qD)}{2(N+D)(N-D)}\)

(−1,1)

\(\frac{M(N+D-M)(N+D-2qD)}{2(N+D)(N-D)}\)

and no change \(\varDelta (D,M)=(0,0)\) occurring at rate

$$\begin{aligned} \frac{q[(N+D-M)^2+2M^2]}{2(N+D)}+\frac{(1-q)[(N-D-M)^2+2M^2]}{2(N-D)}. \end{aligned}$$

The infinitesimal change in the mean \(b(Y_t)=\lim _{h\rightarrow 0}\frac{1}{h} E[Y_{t+h}-Y_t|Y_t]\) has vector components

$$\begin{aligned} \begin{aligned} b_D(Y_t)=&\,\frac{-(N^2-NM-D^2)(N+D-2qN)}{(N+D)(N-D)}\\ b_M(Y_t)=&\,\frac{DM(N+D-2qN)}{(N+D)(N-D)}\\ \end{aligned} \end{aligned}$$
(1)

and the infinitesimal change in the variance and covariance of the process can be calculated as \(a(Y_t)=\lim _{h\rightarrow 0}\frac{1}{h} E[(Y_{t+h}-Y_t)^T(Y_{t+h}-Y_t)|Y_t]\) whose matrix components are

$$\begin{aligned} \begin{aligned} a_{DD}(Y_t)&=\frac{q(MD+2N^2-2D^2-2MN)}{(N+D)}\\&\quad +\frac{(1-q)(-MD+N^2-2D^2-2MN)}{(N-D)}\\ a_{MM}(Y_t)&=\frac{qM(2N+D-2M)}{(N+D)}+\frac{(1-q)M(2N-D-2M)}{(N-D)}\\ a_{MD}(Y_t)&=\frac{-qM(-N-2D+M)}{(N+D)}+\frac{(1-q)M(2N-D-2M)}{(N-D)}\\ \end{aligned} \end{aligned}$$
(2)

In case \(q=\frac{1}{2}\) there is symmetry in the model between species C and H and their long term coexistence in equal amounts is then expected. Note that \(b_D(Y_t)=0\) when \(D=C-H=0\) and that under the same conditions \(b_M(Y_t)=0\). This implies that in the symmetric environment case the whole line \(\{D=0\}\) is in the null space of the mean dynamics, so there is no expected push towards a change. In case \(q>\frac{1}{2}\) there is an expected level of dominance of C over H. Specifically, when \(D=C-H=N(2q-1)\) there is no expected change for \(b_D(Y_t)=0\) and \(b_M(Y_t)=0\). The long term coexistence at this level of difference in their amounts is then expected, as the whole line \(\{D=N(2q-1)\}\) is an invariant for the mean. In case \(q<\frac{1}{2}\) there is an expected level of dominance of H over C and the same argument with the roles of C and H reversed show that \(\{D=N(1-2q)\}\) is an invariant line for the mean dynamics.

Importantly, note that on the above invariant lines the variance is not zero, hence the stochastic evolutionary process still changes over time. Before we proceed with a detailed analysis, we first make a few comparisons.

Consider the version of the model when the environment is not random. If the environment were always in state h, i.e. \(q=0\), species C would be non-viable and replacement by H would be favoured over replacement by M (by a factor of 2), hence the dynamics would result in fixation at \(H=N, C=M=0\). If the environment were always in state c, i.e. \(q=1\), then H would be non-viable, replacement by C would be favoured to that by M, so the dynamics would result in fixation at \(C=N, H=M=0\).

One could also consider a corresponding deterministic evolutionary game: in the environment c it would have pay-offs equal to (2, 0, 1) for (C, H, M) respectively, and in the environment h it would have pay-offs (0, 2, 1) for (C, H, M) respectively. The replicator equations in a randomized environment with chances \((q,1-q)\) for (c, h) predict stable coexistence only when \(q=\frac{1}{2}\). In case \(q>\frac{1}{2}\) the dynamics ultimately always leads to fixation in C and extinction of H and M, and in case \(q<\frac{1}{2}\) it leads to fixation in H and extinction of C and M.

Thus, both randomness of the environment and random demographic fluctuations are required for the dynamics to feature long term coexistence of competing species. In order to quantify its stochastic properties, i.e. evaluate the length of its coexistence phase and determine the relative chances of ultimate fixation of competing species, we will derive a diffusion approximation of the individual based model. The approximation accurately reflects the long term coexistence of the original model, and allows us to efficiently approximate and upper bound the forementioned stochastic quantities.

2.2 Short time competitive dynamics

We assume the population size N is large and consider the dynamics of a rescaled process of \((d,m)=(\frac{D}{N},\frac{M}{N})\) proportions of species in the population:

$$\begin{aligned} y_t=(d_t,m_t):=\frac{Y_{t}}{N}=\Big (\frac{D_{t}}{N},\frac{M_{t}}{N}\Big ) \end{aligned}$$

The rescaled process is contained in the triangle

$$\begin{aligned} S=\{-1\le d\le 1, 0\le m\le 1, m+d\le 1, m-d \le 1\}. \end{aligned}$$

Let the vector \(b=(b_d, b_m)\) and matrix \(a=((a_{dd}, a_{dm}),(a_{md},a_{mm}))^t\) with \(a_{md}\equiv a_{dm}\) be the functions of y which reflect the infinitesimal change in the mean and (co)variance of the rescaled process:

$$\begin{aligned} b_d(y_t)= & {} \lim _{h\rightarrow 0}\frac{1}{h} E[d_{t+h}-d_t|y_t], \quad b_m(y_t)=\lim _{h\rightarrow 0}\frac{1}{h} E[m_{t+h}-m_t|y_t] \\ \frac{1}{N}a_{dd}(y_t)= & {} \lim _{h\rightarrow 0}\frac{1}{h} E[(d_{t+h}-d_t)^2|y_t],\quad \frac{1}{N}a_{mm}(y_t)=\lim _{h\rightarrow 0}\frac{1}{h} E[(m_{t+h}-m_t)^2|y_t] \\ \frac{1}{N}a_{dm}(y_t)= & {} \lim _{h\rightarrow 0}\frac{1}{h} E[(d_{t+h}-d_t)(m_{t+h}-m_t)|y_t] \end{aligned}$$

then, from (1) and (2) we have

$$\begin{aligned} \begin{aligned} b_d(y_t)=&\,\frac{-(1-m-d^2)(1+d-2q)}{(1+d)(1-d)}\\ b_m(y_t)=&\,\frac{dm(1+d-2q)}{(1+d)(1-d)}\\ a_{dd}(y_t)=&\,\frac{q(md+2-2d^2-2m)}{(1+d)}+\frac{(1-q)(-md+2-2d^2-2m)}{(1-d)}\\ a_{mm}(y_t)=&\,\frac{qm(2+d-2m)}{(1+d)}+\frac{(1-q)m(2-d-2m)}{(1-d)}\\ a_{dm}(y_t)=&\,\frac{qm(-1-2d+m)}{(1+d)}+\frac{(1-q)m(1-2d-m)}{(1-d)} \end{aligned} \end{aligned}$$
(3)

The flow of means determined by gradient \((b_d,b_m)\) is shown in Fig. 1 and its magnitude of the mean change is shown in Fig. 2.

Fig. 1
figure 1

Gradient of mean change when \(q=\frac{1}{2}\) (flow of \((b_d,b_m)=E(\varDelta d,\varDelta m)\) with d on the x-axis and m on the y-axis). The flow is directed towards the line \(\{d=0\}\) where its gradient vanishes. Also, here m is increasing (except at \(m=0\))

Full size image
Fig. 2
figure 2

Magnitude of the gradient of mean change when \(q=\frac{1}{2}\) (density plot of \(\Vert (b_d,b_m)\Vert \) with d on the x-axis and m on the y-axis). The gradient has the greatest pull at the absorbing points and decreases as it approaches \(\{d=0\}\)

Full size image

The scaling of the mean and covariance in the dynamics of the model suggests that on most of the space the mean dominates the fluctuations. This dominance \(\Vert b(y)\Vert \sim \frac{1}{N}\Vert a(y)\Vert \) holds only when

$$\begin{aligned} |d-(2q-1)|\gg o(1/N)\;\hbox { and }\;|1-d^2|\gg o(1/N), \end{aligned}$$
(4)

while when \(|d-(2q-1)|\sim O(1/N)\) the mean term compares in magnitude with the noise. The absorbing boundary dominates when \(\Vert (d,m)-(\pm 1, 0)\Vert \sim O(1/N)\). When the mean is dominant the rescaled process \((y_t)_{t\ge 0}\) can be approximated by the dynamics of the deterministic solution of the ODE

$$\begin{aligned} {\bar{y}}_t={\bar{y}}_0+\int _0^tb({\bar{y}}_s)ds \end{aligned}$$
(5)

The following result insures that away from the three absorbing points the deterministic approximation holds for \(y_t\) on any finite time interval.

Proposition 1

If \(y_t=\frac{Y_{t}}{N}\) is the rescaled process started from \(y_0\) and \({\bar{y}}\) is the deterministic solution of (5) started from \({\bar{y}}_0=(d_0,m_0)\), with \(y_0\Rightarrow \bar{y}_0\), then for any \(0<\epsilon <1\), \(N\ge 1\), \(T>0\) there are constants \(C(T)<\infty \) and \(\sup _NL(N,T)<\infty \), such that for \(\gamma ^N_\epsilon =\inf \{t:|1-d_t^2|\le \epsilon \}\) and \(1-d_0^2>\epsilon \) we have

$$\begin{aligned} E(\sup _{t\le T\wedge \gamma ^N_\epsilon } \Vert y_t-{\bar{y}}_t\Vert ^2)\le C(T)\Vert y_0-{\bar{y}}_0\Vert ^2+\frac{L(T)}{{N}}. \end{aligned}$$
(6)

The proof follows from the approximation of density dependent Markov processes by solutions to an ODE obtained by Kurtz (1978). For the rescaled process \(y=\frac{Y}{N}\) small jumps of \(O(\frac{1}{N}\)) that are frequent of O(N) lead to a deterministic process with a Lipschitz continuous (when \(|1-d^2|>\epsilon \)) vector field. The proof of Proposition 1 is given in the “Appendix” (for more details on theory of approximating density dependent processes see Kurtz 1987).

The evolution of the deterministic dynamics given by (5) on the state space \(S=\{-1\le d\le 1, 0\le m\le 1, m+d\le 1, m-d \le 1\}\) is as follows. The gradient vector \(b=(b_d,b_m)\) has a component \(b_d\) vanishing iff either \(d=2q-1\) or \((d, m)\in \{(1, 0),(-1,0)\}\) and the other component \(b_m\) vanishing iff either \(d\in \{0,2q-1\}\) or \(m=0\). The line \(\varGamma =\{(2q-1,m)\}\) is stable, and since the function \(\varphi (y)=(d-2q+1)^2\) is a Lyapunov function for this dynamical system

$$\begin{aligned} \nabla \varphi \cdot b=-2(1-m-d^2)(1+d-2q)^2/(1-d)(1+d)\le 0, \end{aligned}$$
(7)

it is not too hard to show that \(\varGamma \) is also globally attracting on S excluding the corners of this triangle \((d, m)\in \{(1, 0),(-1,0),(0,1)\}\). These corner points are also fixed points for the dynamical system (5) (and absorption points for the original stochastic process). The magnitude of b is highest in the neighbourhood of the two corners \((d, m)\in \{(1, 0),(-1,0)\}\) and lowest near the line \(d=2q-1\), see Figs. 1 and 2.

The trajectory from any initial point in S except the forementioned corners evolves by converging to the stable line \(\varGamma \). Let \(\varPhi (d,m)\) denote the map that takes a point \(y=(d,m)\) to \(\varGamma \), that is, the intersection of the trajectory of \({\bar{y}}\) started from \({\bar{y}}_0=(d,m)\) with the line \(\varGamma \). Let \({\varvec{m}}^*(d,m)\) be the m-coordinate of the map, so \(\varPhi (d,m)=(2q-1,{\varvec{m}}^*)\). From (5) and (3) we have an autonomous system

$$\begin{aligned} \frac{\partial d}{\partial m}=\frac{-(1-m-d^2)}{dm},\quad \frac{\partial m}{\partial d}=\frac{dm}{-(1-m-d^2)} \end{aligned}$$

solved by

$$\begin{aligned} d(m)&=\pm \sqrt{1-2m+Cm^2},\quad m(d)=\frac{1-d^2}{1+\sqrt{1-C(1-d^2)}} \end{aligned}$$

where the constant C is determined by initial conditions \(y_0=(d,m)\) to be

$$\begin{aligned} C=\frac{-1+2m+d^2}{m^2}. \end{aligned}$$

Since \(\varGamma =\{d=2q-1\}\) we get that the m-coordinate of the point of intersection of the trajectory starting from \(\bar{y}_0=(d,m)\) with \(\varGamma \) is

$$\begin{aligned} {\varvec{m}}^* =\frac{4q(1-q)}{1+\sqrt{1-\frac{-1+2m+d^2}{m^2}4q(1-q)}} \end{aligned}$$
(8)

Proposition 1 implies that the first phase of the model dynamics is governed by a strong competitive drive between the two specialists C and H towards their coexistence proportions \(c-h=2q-1\) determined by the distribution of the two environmental states. The proportion of the generalist is \({\varvec{m}}^*\) when the coexistence domain is reached and depends on the initial proportions of the three species. The rest of the time stochastic dynamics will continue in a close neighbourhood of the domain of coexistence \(\varGamma \), where, as we next prove, fluctuations in the proportion of the generalist versus the two specialists will persist for a very long time.

2.3 Long time coexistence and persistent fluctuations

In the initial time period the species count is well approximated by the associated deterministic curve. Asymptotic stability of \(\varGamma \) for this dynamics would suggest that in the long run the species count would converge to \(\varPhi (d,m)\) on \(\varGamma \). However, (4) shows that close to \(\varGamma \) this approximation is no longer accurate, hence we need to examine the original process on a longer time scale. Since in a neighbourhood of the stable line \(\varGamma \) fluctuations are non-negligable, a diffusion approximation is necessary.

The rescaled process \(y_t^{N}=y_{Nt}=Y_{Nt}/N\) under a factor of N speed up of the time scale behaves like the two-dimensional diffusion

$$\begin{aligned} {\tilde{y}}_t={\tilde{y}}_0+N \int _0^t b({\tilde{y}}_s) ds + \int _0^t\sigma ({\tilde{y}}_s) dW_s \end{aligned}$$
(9)

where \(W=(w^d,w^m)\) is a standard Brownian motion in \({\mathbb {R}}^2\), b(y) is as in (3) and \(\sigma (y)\) is the matrix such that \(\sigma ^t(y)\sigma (y)=a(y)\) from (3). This diffusion has a very strong mean and fluctuations only of size O(1). When N is large and the process is at a point outside of \(\varGamma \), the very large mean term overpowers the noise and quickly carries the process onto the stable line \(\varGamma \). Once the process is on \(\varGamma \) the mean term is zero so the noise can move the process away, but as soon as its distance from \(\varGamma \) is more than O(1/N) the mean carries it back, this time to another point on \(\varGamma \). The noise acts as a perpetual source of perturbations for the stabilizing deterministic dynamics. In the limit of a large population size this becomes equivalent to a diffusive process on \(\varGamma \).

In order to calculate the characteristics of the limiting process on \(\varGamma \) we need to use the map \(\varPhi \) which projects each point to its trajectory destination on \(\varGamma \) after it is perturbed by the diffusion (9). Recall that \(\varPhi (d,m)=(2q-1,{\varvec{m}}^*)\) where \({\varvec{m}}^*\) is a function of (d, m) through the constant \(C=C(d,m)\) (see (8) above)

$$\begin{aligned} {\varvec{m}}^*=\frac{4q(1-q)}{1+\sqrt{1-C4q(1-q)}}. \end{aligned}$$

Considering \(\varPhi (d,m)=(2q-1,\frac{4q(1-q)}{1+\sqrt{1-C(d,m)4q(1-q)}})\) as a function applied to the two-dimensional diffusion (9) we can use Ito’s lemma in order to determine the stochastic behaviour of \(\varPhi (d_t,m_t)\). For this we will need the following derivative calculations:

$$\begin{aligned} \frac{\partial {\varvec{m}}^*}{\partial d}= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial C}{\partial d}\nonumber \\ \frac{\partial {\varvec{m}}^*}{\partial m}= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial C}{\partial m}\nonumber \\ \frac{\partial ^2 {\varvec{m}}^*}{\partial ^2 d}= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial ^2 C}{\partial ^2 d}+\frac{({\varvec{m}}^*)^2(12q(1-q)-2{\varvec{m}}^*))}{2(4q(1-q)-{\varvec{m}}^*)^2}\frac{\partial {\varvec{m}}^*}{\partial d}\frac{\partial C}{\partial d}\nonumber \\ \quad \quad \;\;= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial ^2 C}{\partial ^2 d}+\frac{({\varvec{m}}^*)^5(12q(1-q)-2{\varvec{m}}^*))}{4(4q(1-q)-m^*)^3}\Big (\frac{\partial C}{\partial d}\Big )^2\nonumber \\ \frac{\partial ^2 {\varvec{m}}^*}{\partial ^2 m}= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial ^2 C}{\partial ^2 m}+\frac{({\varvec{m}}^*)^2(12q(1-q)-2{\varvec{m}}^*))}{2(4q(1-q)-{\varvec{m}}^*)^2}\frac{\partial {\varvec{m}}^*}{\partial m}\frac{\partial C}{\partial m}\nonumber \\ \quad \quad \;\;= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial ^2 C}{\partial ^2 d}+\frac{({\varvec{m}}^*)^5(12q(1-q)-2{\varvec{m}}^*))}{4(4q(1-q)-{\varvec{m}}^*)^3}\Big (\frac{\partial C}{\partial m}\Big )^2\nonumber \\ \frac{\partial ^2 {\varvec{m}}^*}{\partial d\partial m}= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial ^2 C}{\partial d\partial m}+\frac{({\varvec{m}}^*)^2(12q(1-q)-2{\varvec{m}}^*))}{2(4q(1-q)-{\varvec{m}}^*)^2}\frac{\partial {\varvec{m}}^*}{\partial m}\frac{\partial C}{\partial d}\nonumber \\ \quad \quad \;\;= & {} \frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\frac{\partial ^2 C}{\partial d\partial m}+\frac{({\varvec{m}}^*)^5(12q(1-q)-2{\varvec{m}}^*))}{4(4q(1-q)-{\varvec{m}}^*)^3}\Big (\frac{\partial C}{\partial d}\frac{\partial C}{\partial m}\Big )\nonumber \\ \end{aligned}$$
(10)

and since \(C(d,m)=(-1+2m+d^2)/m^2\) we also have that

$$\begin{aligned} \begin{aligned}&\frac{\partial C}{\partial d}=\frac{2d}{m^2},\\&\frac{\partial C}{\partial m}=\frac{2(1-d^2-m)}{m^3}\\&\frac{\partial ^2 C}{\partial ^2 d}=\frac{2}{m^2},\\&\frac{\partial ^2 C}{\partial ^2 m}=-\frac{2(3-3d^2-2m)}{m^4},\\&\frac{\partial ^2 C}{\partial d\partial m}=-\frac{4d}{m^3} \end{aligned} \end{aligned}$$
(11)

Ito’s formula applied to \(\varPhi (d,m)=(2q-1,{\varvec{m}}^*(d,m))\) implies that the second coordinate of \(\varPhi \) is a one-dimensional diffusion process. Its mean coefficient is given by

$$\begin{aligned} \beta _{{\varvec{m}}^*}(d,m)&=N\Big (\frac{\partial {\varvec{m}}^*}{\partial d}b_d+\frac{\partial {\varvec{m}}^*}{\partial m}b_m\Big )+\frac{1}{2}\Big (\frac{\partial ^2 {\varvec{m}}^*}{\partial d^2}a_{dd}+\frac{\partial ^2 {\varvec{m}}^*}{\partial m^2}a_{mm}+2\frac{\partial ^2 {\varvec{m}}^*}{\partial d\partial m}a_{dm}\Big )\nonumber \\&=\frac{1}{2}\frac{\partial ^2 {\varvec{m}}^*}{\partial d^2}a_{dd}+\frac{1}{2}\frac{\partial ^2 {\varvec{m}}^*}{\partial m^2}a_{mm}+\frac{\partial ^2 {\varvec{m}}^*}{\partial d\partial m}a_{dm} \end{aligned}$$
(12)

since by (10), (11) and (3) we have that

$$\begin{aligned}&\frac{\partial {\varvec{m}}^*}{\partial d}b_d+\frac{\partial {\varvec{m}}^*}{\partial m}b_m\\&\quad =\frac{({\varvec{m}}^*)^3}{2(4q(1-q)-{\varvec{m}}^*)}\left( \frac{2d}{m^2}\frac{-(1-m-d^2)(1+d-2q)}{(1+d)(1-d)}\right. \nonumber \\&\qquad \left. +\frac{2(1-m-d^2)}{m^2}\frac{dm(1+d-2q)}{(1+d)(1-d)}\right) \\&\quad =0, \end{aligned}$$

Its diffusion coefficient is calculated by combining the contributions to fluctuations from the two independent noise components in (9) given by

$$\begin{aligned} \alpha _{{\varvec{m}}^*}(d,m)&=\sqrt{\Big (\frac{\partial {\varvec{m}}^*}{\partial d}\sigma _{dd}+\frac{\partial {\varvec{m}}^*}{\partial m}\sigma _{md}\Big )^2+\Big (\frac{\partial {\varvec{m}}^*}{\partial d}\sigma _{dm}+\frac{\partial {\varvec{m}}^*}{\partial m}\sigma _{mm}\Big )^2}\nonumber \\&=\sqrt{\Big (\frac{\partial {\varvec{m}}^*}{\partial d}\Big )^2a_{dd}+\Big (\frac{\partial {\varvec{m}}^*}{\partial m}\Big )^2a_{dd}+2\Big (\frac{\partial {\varvec{m}}^*}{\partial d}\Big )\Big (\frac{\partial {\varvec{m}}^*}{\partial m}\Big )a_{dm}} \end{aligned}$$
(13)

and consequently

$$\begin{aligned} {\varvec{m}}^*_t={\varvec{m}}^*_0+\int _0^t \beta _{{\varvec{m}}^*}(2q-1,{\varvec{m}}^*)ds + \int _0^t\alpha _{{\varvec{m}}^*}(2q-1,{\varvec{m}}^*)dw_s, \;{\varvec{m}}^*\in [0,1] \end{aligned}$$
(14)

where w is a standard Brownian motion in \({\mathbb {R}}^1\).

The large component in the mean disappears confirming our earlier claim that the limit is a regular diffusion whose state space is the stable line \(\varGamma \). This is due to the fact that the contribution to the mean in the perpendicular direction to \(\varGamma \) of \(\nabla \varPhi \cdot b=0\) vanishes, as there is no change in the projection map \(\varPhi \) along the flow of the mean. The only component in the mean that remains is from the change in \(\varPhi \) in the direction tangential to \(\varGamma \). A rigorous statement for this one-dimensional approximation of the long term behaviour of our original stochastic process is given by the following result.

Proposition 2

If \(y^{N}_t=\frac{Y_{Nt}}{N}\) is the rescaled process started at a point \(y^{N}_0\) in a neighbourhood \(\Vert y^{N}_0-\varGamma \Vert \le N^{-\delta }, \delta \in (0,\frac{1}{2})\) of the stable line, and for \(\epsilon >0\)\(\tau ^N_\epsilon =\inf \{t:m_t<\epsilon \vee m_t>1-|2q-1|-\epsilon \}\), then as \(N\rightarrow \infty \) the rescaled process stays in the neighbourhood with \(P(\sup _{t\le \tau ^N_\epsilon }\Vert y^{N}_t-\varGamma \Vert \le 2N^{-\delta }) \rightarrow 1\) and we have convergence of the stopped process to the diffusion on \(\varGamma \) from (14)

$$\begin{aligned} y^{N}_{\cdot \wedge \tau ^N_\epsilon }\Longrightarrow {\varvec{m}}^*_{\cdot \wedge \tau _\epsilon } \end{aligned}$$

stopped at \(\tau _\epsilon =\inf \{t:{\varvec{m}}^*_t<\epsilon \vee {\varvec{m}}^*_t>1-|2q-1|-\epsilon \}\). For \(\tau =\inf \{t:\varvec{m^*}_t\in \{0,1-|2q-1|\}\}\) we have \(\tau \ge \liminf \limits _N\tau ^N\), for \(\tau ^N=\inf \{t:m_t\in \{0 ,1-|2q-1|\}\}\).

The proof of this result is a consequence of theorem for convergence to degenerate diffusions derived by Katzenberger (1991). A similar technique was used in Durrett and Popovic (2009) to establish approximation by a degenerate diffusion for a model of genetic subfunctionalization, and an excellent practical exposition is given in Parsons and Rogers (2017). It relies on the strong stability of the line \(\varGamma \), differentiability of the projection map \(\varPhi \) onto \(\varGamma \), as well as on the good behaviour of the jumps of the rescaled Markov process. One first uses Lyapunov function \(\varphi \) to show that the distance of the process \(y^N\) to \(\varGamma \) converges to 0, followed by a convergence result for integrals with respect to semi-martingales in order to identify that the limiting process on \(\varGamma \) is indeed given by the Ito’s lemma calculation.

Proposition 2 implies that the second, much longer, phase of the model dynamics consists of a trade-off between a fluctuating proportion of the generalist M and a proportion of a combination of specialists C and H. During this time the combination of the latter two is such that their difference is kept approximately at its ‘coexistence balance’ (as dictated by the distribution of the random environment). Given that this period lasts for a time of order N one could try to find its approximate distribution on the line \(\varGamma \), using the law of diffusion \({\varvec{m}}^*_t\) conditioned on \(\tau >t\). In this manner one can estimate the occupation time distribution during the long transient period of coexistence. For deriving quasi-stationary distributions see Martinez (1995, Theorem B) and Meleard et al. (2009, Theorem 5.2). Ultimately this phase ends with either the fixation of species M or their extinction. In the latter case only C and H remain and continue a stochastic competition until ultimate fixation.

If at the stopping time \(\tau ^N\) the original process \(y^N\) is at \(y^N_{\tau ^N}=(2q-1,0)\), the process loses all individuals of M type first. The only possible transitions forward in time are \(\varDelta y^N\in \{(\pm 2/N,0)\}\) which keep y on the line \(\{m=0\}\) while changing the ‘coexistence balance’ of c, h away from \(\{d=2q-1\}\). The probability of fixation and the expected time to fixation from \(\tau ^N\) onward is well approximated by a subsequent 1-dimensional diffusion \({\varvec{d}}^*\) on the line \(\{m=0\}\) started at \(y_0=(d_0,0)\). This diffusion is a mean reverting Gaussian process with strong drift and small noise: \({\varvec{d}}^*_t={\varvec{d}}^*_0+\int _0^tN\beta _{{\varvec{d}}^*}(d^*_s)ds+\int _0^t\sqrt{\alpha _{{\varvec{d}}^*}(d^*_s)}dw_s\), for \({\varvec{d}}^*\in [-1,1]\) with \(\beta _{{\varvec{d}}^*}(d)=-(d-(2q-1))\) and \(\alpha _{{\varvec{d}}^*}(d)=2-2d(2q-1)\). Since \([-1,1]\) is bounded, the noise will ultimately lead to fixation in C or H with probabilities determined by the value of q.

If at time \(\tau ^N\) the original process is at \(y^N_{\tau ^N}=(2q-1,1-|2q-1|)\), when \(q=\frac{1}{2}\) we already have fixation of species M. Otherwise, the only possible transitions forward in time have: \(\varDelta d+\varDelta m=0\) when \(q>\frac{1}{2}\), and \(-\varDelta d+\varDelta m=0\) when \(q<\frac{1}{2}\), insuring the process stays on the line \(\{d+m=1\}\) when \(q>\frac{1}{2}\), or \(\{-d+m=1\}\) when \(q<\frac{1}{2}\). In this case from \(\tau ^N\) onward \(y^N\) corresponds to a process on the line \(\{(\pm ) d+m=1\}\) started at \(y_0=(2q-1,1-|2q-1|)\) with very strong drift and small noise. Since the drift on this line is always in the direction of \(y=(0,1)\) with probability converging to 1 it will almost instantaneously lead to ultimate fixation in M.

2.4 Probabilities of fixation and its expected time

To illustrate the usefulness of our results we consider the probabilities of fixation and time to fixation and their relationship to the limiting results. In the original (rescaled) process \(y^{N}=Y_{Nt}/N\,\) let \(\tau _C,\tau _H,\tau _M\) denote the times to extinction of each of the species

$$\begin{aligned} \tau _C=\inf \{t: c_t=0\},\, \tau _H=\inf \{t: h_t=0\},\,\tau _M=\inf \{t: m_t=0\} \end{aligned}$$

where we assume \(\inf \{\emptyset \}=\infty \), let \(\tau ^e\) denote the time of the first extinction and \(\tau ^f\) the time of fixation, i.e. second extinction. Note, \(\tau ^e<\infty \) and \(\tau ^f<\infty \), and it is possible for \(\tau ^e=\tau ^f\). Denote the probabilities of first extinction by

$$\begin{aligned} p^e_C=P(\tau ^e=\tau _C),\, p^e_H=P(\tau ^e=\tau _H),\, p^e_M=P(\tau ^e=\tau _M) \end{aligned}$$

and the probabilities of fixation by

$$\begin{aligned} p^f_C=P(y^{N}_{\tau ^f}=C),\, p^f_H=P(y^{N}_{\tau ^f}=H),\, p^f_M=P(y^{N}_{\tau ^f}=M). \end{aligned}$$

Proposition 1 implies that as long as the starting value is away from the corners \((d, m)\in \{(1, 0),(-1,0), (2q-1,1)\}\) the time to reach the ‘coexistence balance’ line \((2q-1,m)\) is negligible on the O(N) time scale.

Proposition 2 further implies that \(\tau ^e\) is upper bounded by the stopping time \(\tau =\inf \{t:{\varvec{m}}^*_t\in \{0,1-|2q-1|\}\}\) on the diffusion \({\varvec{m}}^*\):

\(\cdot \):

if \({\varvec{m}}^*_\tau =1-|2q-1|\) then \(\tau _C=\tau _H=\tau ^e=\tau ^f\) is upper bounded by \(\tau \);  both \(p^e_{C}+p^e_{H}\) and \(p^f_M\) are well approximated by \(P({\varvec{m}}^*_\tau =1)\);

\(\cdot \):

if \({\varvec{m}}^*_\tau =0\) then \(\tau ^e=\tau _M\) is upper bounded by \(\tau \);  both \(p^e_M\) and \(p^f_C+p^f_H\) are well approximated by \(P({\varvec{m}}^*_\tau =0)\).

So, letting \(p_M=P({\varvec{m}}^*_\tau =0)\) we have:

$$\begin{aligned} \begin{aligned} E(\tau ^e)&\lesssim E(\tau ),\; E(\tau ^f)\lesssim E(\tau | \hbox {first extinction is not M})\\ p^e_{C+H}&=1-p^e_M\approx p_M,\; p^f_M\approx 1-p_M \end{aligned} \end{aligned}$$
(15)

Since \({\varvec{m}}^*\) is a one-dimensional diffusion the above quantities have analytical formulae that can certainly be explicitly computed. For a one-dimensional diffusion of the form

$$\begin{aligned} dx_t=\beta (x_t)dt+\sqrt{\alpha (x_t)}dw_t \end{aligned}$$

we can use the natural scale function (unique up to linear map)

$$\begin{aligned} \phi (x)=\int ^x \exp \left\{ \int ^y \frac{-2\beta (z)}{\alpha (z)}dz\right\} dy \end{aligned}$$

from which started from a point \(x\in [x_0,x_1]\) the hitting time probabilities \(\tau _{x_0}\) and \(\tau _{x_1}\) of any two points \(x_0<x_1\) is computed as

$$\begin{aligned} P_x(\tau _{x_0}<\tau _{x_1})=\frac{\phi (x_1)-\phi (x)}{\phi (x_1)-\phi (x_0)}. \end{aligned}$$

We can also use the natural scale function to calculate the expected hitting time \(E_x[\tau _{x_0}\wedge \tau _{x_1}]\) started from a point \(x\in [x_0,x_1]\) via the Green’s function on the interval \([x_0,x_1]\)

$$\begin{aligned} G(x,y)&=2\frac{(\phi (x_1)-\phi (x))(\phi (y)-\phi (x_0))}{\phi (x_1)-\phi (x_0)}\frac{1}{\phi '(x)\alpha (x)},y\le x;\\&=2\frac{(\phi (x)-\phi (x_0))(\phi (x_1)-\phi (y))}{\phi (x_1)-\phi (x_0)}\frac{1}{\phi '(x)\alpha (x)},y> x; \end{aligned}$$

from which the expected hitting time is computed as

$$\begin{aligned} E_x[\tau _{x_0}\wedge \tau _{x_1}]=\int _{x_0}^{x_1} G(x,y)dy. \end{aligned}$$

For ease of display we calculate explicitly the formulae above in the symmetric environment case \(q=\frac{1}{2}\). First, for the coefficients of the two-dimensional process (3) we get

$$\begin{aligned} a_{dd}=\frac{2-2m-2d^2-md^2}{1-d^2}, \; a_{mm}=\frac{m(2-2m-d^2)}{1-d^2},\; a_{md}=\frac{-dm(1+m)}{1-d^2} \end{aligned}$$

Then for \({\varvec{m}}^*\) the partial moments (10) the mean and variance formulae (12) and (13), after some algebraic simplifications, give the following expressions for coefficients:

$$\begin{aligned} \beta _{{\varvec{m}}^*}(d,m)&=\frac{-m(-2d^4-2(m-1)(-2+3\sqrt{-d^2+(m-1)^2}+3m))}{2(-1+d^2)\sqrt{-d^2+(m-1)^2}(\sqrt{-d^2+(m-1)^2})^3}\\&\quad +\frac{-m(d^2(6-4m+3m\sqrt{-d^2+(m-1)^2}+3m^2))}{2(-1+d^2)\sqrt{-d^2+(m-1)^2}(\sqrt{-d^2+(m-1)^2})^3},\\ \alpha _{{\varvec{m}}^*}(d,m)&=\frac{m(2-d^2-2m)}{(\sqrt{-d^2+(-1+m)^2}+m)^4}, \end{aligned}$$

and

$$\begin{aligned} \beta _{{\varvec{m}}^*}(0,m)=\frac{m}{(1-m)^3},\;\; \alpha _{{\varvec{m}}^*}(0,m)=2m(1-m). \end{aligned}$$

Note from above that the diffusion inherently stops when the \(m=0\) boundary point is reached and the drift is not defined when \(m=1\) is reached. The first case corresponds to the event of loss of the generalist species M, while the second case corresponds to fixation in M.

The mapping of the starting value \((d_0,m_0)\) onto the line \(\varGamma =\{d=0\}\) is given by

$$\begin{aligned} {\varvec{m}}^*(d_0,m_0)=\frac{1}{1+\sqrt{1-\frac{d_0^2+2m_0-1}{m_0^2}}} \end{aligned}$$

Using \(x={\varvec{m}}^*(d_0,m_0)\) the probability \(p_M\) is calculated as

$$\begin{aligned} p_M=P_x({\varvec{m}}^*_\tau =0)=\frac{\phi (1)-\phi (x)}{\phi (1)-\phi (0)} \end{aligned}$$

from the natural scale function \(\phi (x)=\int ^x \exp \{\int ^y \frac{-2\beta (z)}{\alpha (z)}dz\}dy\) by numerically integrating the inner integral over the area projected by the ODE onto the line \(\varGamma =\{d=0\}\) between \(m=y\) and \(m=0\). This corresponds to the area defined by

$$\begin{aligned} -\sqrt{1-2m+\frac{-1+2y}{y^2}m^2}\le d\le \sqrt{1-2m+\frac{-1+2y}{y^2}m^2}, 0\le m\le y \end{aligned}$$

Note that when \(y=1\) this corresponds to the integral over the whole triangle S. The expected time to first extinction is then calculated as

$$\begin{aligned} E_x(\tau )=\int _0^1 G(x,y)dy \end{aligned}$$

by further numerical integration to obtain all values the scale function \(\phi (y)\) over \(y\in [0,1]\) to be used in the expression

$$\begin{aligned} G(x,y)&=2\frac{(\phi (1)-\phi (x))(\phi (y)-\phi (0))}{\phi (1)-\phi (0)}\frac{1}{\phi '(x)\alpha (x)},y\le x;\\&=2\frac{(\phi (x)-\phi (0))(\phi (1)-\phi (y))}{\phi (1)-\phi (0)}\frac{1}{\phi '(x)\alpha (x)},y> x; \end{aligned}$$

Explicit values using the one dimensional diffusion for the extinction probability \(p_M\) and the expected extinction time \(E_{y_0}(\tau )\) are shown alongside simulation results in Figs. 5 and 6 below. The explicit values are based on numerical integration for calculating \(\phi \) at 1000 points. The graph of \(p_M\) and of \(E_{y_0}(\tau )\) as functions of the starting point for the process \(y_0=(d_0=0,m_0)\) is made by interpolation between computed values for 1000 different starting points.

2.5 Simulation results

We compared the results based on the diffusion approximations with simulations of the original stochastic model Y in the symmetric environment \(q=\frac{1}{2}\). To assess the sensitivity to the population size we simulated the original process with different orders of magnitude for the population size: \(N=100,\,1000,\,10{,}000\) and estimated the probability distribution of the time to extinction \(\tau ^e\) based on outcomes of 1000 simulation runs for each choice of N. The histograms in Fig. 3 show remarkable similarity, and a heavy tail in their distribution: the best fit to the decay of probabilities for large t was found to scale as \(\sim t^{-\alpha }\) with \(\alpha \in (1.45,1.65)\). This seems reasonable in light of known results for the first passage times of one dimensional diffusions with constant coefficients whose probability density function has a power-law decay scaling as \(\sim t^{-\frac{3}{2}}\).

Fig. 3
figure 3

Distribution of the time to extinction \(\tau ^e\) for Y starting from \(y_0=(0,\frac{1}{3})\) for three different values of total population size \(N=10^2, 10^3, 10^4\) (\(\tau ^e\) is scaled by N in each case)

Full size image

We evaluated the behaviour of the process in its initial phase, by estimating the probability that the process reaches its first extinction before reaching the coexistence balance line \(\varGamma =\{d=0\}\) and by estimating the time until it reaches \(\varGamma \). For \(\tau _\varGamma =\inf \{t:d_t=0\}\) estimates for \(P(\tau _\varGamma <\tau ^e)\) were calculated for eight different starting points in the right half of the triangle S: \((d_0,m_0)\)=(0.334,0.334), (0.48,0.48), (0.495,0.495), (0.5,0.5), (0.5,0.020), (0.5,0.01), (0.97,0.01), (0.985,0.005). Using 1000 runs the estimates showed \(P(\tau _\varGamma<\tau ^e)< 0.01\) for each starting point except for: \(y_0=(0.495,0.495)\) when this probability is \(\approx 0.05\) and \(y_0=(0.985,0.005)\) when it is \(\approx 0.025\). As long as the staring point is at distance \(>0.05\) from one of the absorbing corner points of S the pull towards the coexistence line prevails. The empirical probability distribution of the time \(\tau _\varGamma \) using \((d_0,m_0)\)=(0.334,0.334) is shown in Fig. 4. Note that the upper bound for the unscaled \(\tau _\varGamma \) is significantly smaller than \(N=1000\) indicating that the initial phase is negligible relative to the rest of the stochastic dynamics of the process.

Fig. 4
figure 4

Distribution of the time \(\tau _\varGamma \) for Y to reach \(\varGamma \) starting from \(y_0=(\frac{1}{3},\frac{1}{3})\) (\(\tau _\varGamma \) is unscaled)

Full size image

The dependence of the probability \(p^e_M\) and the mean time to the first extinction \(E_{y_0}(\tau ^e)\) on the value of the starting point for the process \(y_0=(d_0,m_0)\) is shown in Figs. 5 and 6. The simulations were done using population size \(N=1000\) and ten equally spaced values for \(y_0=(0,0), (0,0.1), (0,0.2), \ldots , (0.09)\), with estimates based in 1000 runs. Probability of first extinction at M decreases with increase in the initial proportion of M, in a nonlinear way. The time to extinction at first increases then decreases with increase in the initial proportion of M, in a concave way with a maximum at approximately \(m_0=0.334\).

Fig. 5
figure 5

Graph of \(p_M\) as a function of the starting point \(x=(0,m_0)\) of the process (with \(m_0\) on the x-axis). Purple line is based on values calculated from explicit formula (for 1000 different values of \(m_0\)); blue line connects values (for 10 different values of \(m_0\)) estimated from simulations (using 1000 runs of the original unscaled process Y) (colour figure online)

Full size image
Fig. 6
figure 6

Graph of \(E_x(\tau )\) as a function of the starting point \(x=(0,m_0)\) of the process (\(m_0\) on the x-axis, \(NE_x(\tau )\) on the y-axis for \(N=10^3\)). Purple line is based on explicit formula (for 1000 different values of \(m_0\)); blue line connects values (for 10 different values of \(m_0\)) from simulations (1000 runs of unscaled Y) (colour figure online)

Full size image

The graphs are reminiscent of results for Moran model with selection. In a neutral Moran model the probability \(1-p_M\) of fixation in M would display a linear dependence in \(m_0\), while our model displays higher values. The concave dependence of \(1-p_M\) on \(m_0\) deduced from Fig. 5 indicates an advantage of the generalist species M relative to the two specialists species C and H. The advantage is apparent at times when the specialists C and H are not balanced: on \(\varGamma \) the infinitesimal drift \(b_d=b_m=0\) indicate no average advantage for any species, but off of \(\varGamma \) an average advantage for M appears in the region where \(b_m=dm(1+d-2q)/(1-d^2)>0\) and is higher for higher values of imbalance \(|d|=|c-h|\). In a symmetric \(q=\frac{1}{2}\) environment the generalist always has an advantage \(\nabla m=b_m>0, \forall d\in (-1,1)\). An asymmetric \(q\ne \frac{1}{2}\) environment which favours one of the specialists, gives \(b_m<0\) between \(\{d=0\}\) and \(\{d=2q-1\}\). This plausibly decreases the concave dependence of \(p_M\) in \(m_0\) and increases the expected time to fixation. The advantage of M results from the way dependence of frequencies and fitness in the replacement rates of the model was defined.

Both Figs. 5 and 6 show gaps between values obtained from the diffusion approximation (smooth lines) and those obtained from simulations (interpolated lines) of the full model. The discrepancy is likely caused by numerical problems in evaluating the natural scale and Green’s function of the approximating diffusion whose coefficients \(\beta _{{\varvec{m}}^*}\) and \(\alpha _{{\varvec{m}}^*}\) are ill-defined at \(m=1\). It is also concievable that the first hitting time of this diffusion is in fact only an upper bound for the corresponding hitting times of the original process (see Remark at the end of the proof of Proposition 2).

2.6 Model in a fluctuating environment

In our model the random environment was assumed to be stationary in time, so that the probability distribution of state c versus h stays constant as the population evolves. In other words the model is embedded in an environment whose values at each replacement event are i.i.d. with the given distribution \((q,1-q)\). In addition to a time homogeneous random environment this also approximates well the case of a fast fluctuating random environment that equilibriates on the time scale N of competitive dynamics. Our diffusion approximation results then remain valid with \((q,1-q)\) representing the equilibrium distribution of the c, h environment.

In case of slowly fluctuating environment we can approximate the model dynamics by a process of switching diffusions. In case the change in the environment distribution is slower than the time scale N of coexistence diffusion in between its changes the competitive dynamics occurs and the species start to diffuse on the line of coexistence balance. When the next change occurs it changes the dynamics leading to new phase of competitive dynamics which results in a different line of coexistence balance. Analysis of dynamics induced by environment fluctuating on a time scale that is between (and including) O(1) and O(N) requires a new mathematical approach.

3 Discussion

We present a stochastic evolutionary game model in two dimensions and derive rigorous approximations of its long term behaviour in terms of a 1 dimensional diffusion process. Our model incorporates effects of a random environment which determines the fitness of different species and hence influences the competitive dynamics. Without randomness of the environment stochastic competitive dynamics would lead only towards fixation of the species that is the fittest in that environment. Deterministic competitive dynamics in a random environment would also result only in fixation of the species which is more favoured by the environment (only in the symmetric case would it end in coexistence). In the stochastic random environment model the fixation outcome is not predetermined, and moreover, the dynamics goes through a long transient state of coexistence.

We make predictions for the outcomes of the model, determining probabilities of extinction of species and an upper bound for the expected time to the first extinction event. In order to do so we derive rigorous stochastic approximations characterizing the long term behaviour of the model. Initially its dynamics closely follows approximately deterministic competition between the species (its mean dynamics). This phase is relatively short and results in a region of ‘coexistence balance’ between all the species, in the neighbourhood of which it remains for the rest of the time. In the next, longer, phase the dynamics fluctuates within the region of coexistence, and can be approximated by a diffusion process (whose mean and variance coefficients are explicitly derived). Depending on which boundary of the coexistence region this diffusion reaches first, there may be a subsequent phase of coexistence of the two remaining species. In this case the fixation outcome is determined when this latter randomly perturbed dynamical system first reaches one of its boundary points.

Our results provide a rigorous dimension reduction for the model, which captures the full stochastic nature of its behaviour. It allows one to make calculations and predictions for the model which would analytically be intractable in the original model, with errors that go to zero with increasing population size. The approximating process is a one dimensional diffusions on regions of coexistence for which simple integration yields analytical expressions for the probabilities and expected hitting times. We illustrate the validity and proximity of our approximation to the original stochastic model using simulation results.

With a simple stochastic model we expose a new approach to long term transient coexistence, which does not fit within the mathematical framework that has so far been considered for analysis of stochastic evolutionary games. We emphasize the usefulness of approximating by degenerate diffusions, providing a rigorous model reduction that accurately captures the long term stochastic behaviour.