1 Introduction

This paper deals with the interaction of two populations, a prey and a predator species, modelled by means of two-dimensional differential equations. We are specially interested in the case when the prey population faces difficulties to grow from low densities and avoid extinction. This phenomenon, traditionally known as an Allee effect [6, 31, 36], is characterised by a tendency of the population growth rate to decrease under some minimum critical level [8]. In occasions, the Allee effect is such that the population growth rate becomes negative and induces an extinction threshold—commonly known as the Allee threshold—that the population has to overcome in order to survive and avoid extinction [8, 14, 20, 41].

Among the causes that may generate an Allee effect on a population are the difficulty of finding mates or social interaction [9, 19, 31]. In this paper, we are interested in the consequences of an Allee effect on the prey generated by predation [15, 18, 23, 27, 32]. Indeed, when at low densities, the prey population tends to have difficulties to better defend or hide from the predator and, in general, to show an antipredator behaviour [22, 40, 48, 49]. In this scenario, the identification of basins of attraction of stable equilibrium points and periodic orbits, as well as their basin boundaries, emerge as important challenges in order to predict the survival or extinction of populations.

The aim of this paper is to propose a general class of predator–prey models in which the prey population is subjected to an Allee effect, and to study the conditions for specific dynamical behaviours. The main novelty of this family of models is that, rather than stating all the equation terms explicitly, we model the Allee effect by means of a general term—called an Allee function throughout—that multiplies the classic logistic growth rate in the prey equation; we have called this form a multiplicative Allee effect. Only a few suitably chosen basic generic properties are imposed on the Allee function, in such a way that it agrees with, and generalises, a number of concrete realistic models studied recently [4, 24, 26]. In the same spirit, the predator consumption rate is conveniently modelled by means of a general (non-explicit) predation function that has the same features of the well-known class of Holling type functional responses II–IV [34, 35, 38, 40, 43, 44]; see also [3, 4] and the references therein for further discussion. Moreover, we assume that the typical environmental carrying capacity of the predator is proportional to prey abundance as in the May–Holling–Tanner model [7] and other models recently analysed [3, 4, 25, 35, 50]. In this way, this family of models is of Leslie type [25, 30, 45], as opposed to the Gause type of interaction in which the prey–predator mass conversion due to predation follows a kind of conservation law [24, 26, 29, 46].

While models with known, explicit terms may be the most common object of study in population dynamics, the results obtained from general classes of models, in turn, emerge as a powerful tool to understand—in a broader setting—the rich dynamics shown in population interactions; see [16, 29] for instance. Hence, our goal is to shed light on the specific conditions that our proposed, prototype model must meet in order to present certain dynamical behaviours which are common to several—sometimes unrelated—concrete models. While a Gause-type class of models with the so-called strong Allee effect was proposed recently in the same spirit [46], the work in this paper is the first example, as far as this author knows, in which a general family of Leslie–Gower-type predation models with Allee effect (and with any degree of severity, strong or otherwise) is investigated.

Our family of models is studied with analytical tools from dynamical systems theory. More concretely, by means of topological equivalences [7, 28, 33], suitable changes of parameters, and a smooth extension to the entire first quadrant, we are able to construct suitable vector fields in which the analysis is carried out. With this approach, we construct an invariant compact region in phase plane, which enables us to show that the model is well posed in the sense that any realistic solution remains non-negative and is always bounded. The local stability of equilibria is explored as well. The origin turns out to be a non-hyperbolic equilibrium [21] and, hence, desingularization methods are needed to study the local dynamics near this point. In this way, we found the conditions on the general model such that the origin has a non-trivial basin of attraction and we calculate the relevant separatrices explicitly.

Positive equilibria are studied as well using an approach from bifurcation theory. More concretely, we find specific conditions such that our model may undergo saddle-node and Hopf bifurcations [28]. We also find explicit conditions such that a positive equilibrium undergoes a codimension-two Bogdanov–Takens bifurcation [13, 33]. These results enable us to unravel the role of the Bogdanov–Takens point as an organisation centre for the global dynamics of our model. In particular, we are able to describe the bifurcations of limit cycles and homoclinic orbits that the model may present under certain regimes. These global phenomena are complemented, in each case, with bifurcation diagrams and topological sketches of selected scenarios of different phase portraits for representative parameter values near the Bogdanov–Takens point. Our findings are illustrated with a series of concrete functional forms for the Allee function and functional response.

This paper is structured as follows: the model and general settings are presented in Sect. 2. In Sect. 3 we study the boundedness of the model family. Section 4 treats the local stability of equilibria in the absence of predator, including the analysis at the origin. Section 5 presents the study of positive equilibria and conditions for bifurcations. Section 7 illustrates our findings in concrete model vector fields. Finally, in Sect. 8 we summarise and discuss the main results in this paper, as well as explore possible future avenues of research. An “Appendix” is also included featuring a normal form theorem for the Bogdanov–Takens point studied in Sect. 5.

2 Problem setup

2.1 The general model

We consider the following family of predation models given by differential equations of the general form:

$$\begin{aligned} \mathbf {X}:\left\{ \begin{array}{rcl} \dot{x} &{} = &{} r\,x\left( 1 - \dfrac{x}{k}\right) A(x) - \phi (x)\,y; \\ &{}&{}\\ \dot{y} &{} = &{} s\,y\left( 1 - \dfrac{y}{n x}\right) ;\\ \end{array} \right. \end{aligned}$$
(1)

where \((x(t),y(t)) \in \mathcal {D}:=\{(x,y) \in \mathbb {R}^2|\, x>0,y \ge 0 \}\) represents the population densities of prey and predator, respectively, in time \(t>0\). Parameter \(r>0\) (resp. \(s>0\)) is the intrinsic growth rate or biotic potential of the population \(x\) (resp. \(y\)), and \(k>0\) is the environment carrying capacity of \(x\) [37]. Moreover, the conventional environmental carrying capacity \(K_{y}\) of the predator is proportional to prey availability \(x \) as a resource for the predator [37], that is, \(K_{y}=nx\), where \(n>0\).

The term \(A(x)\) models an Allee effect that affects the prey population. Since \(A(x)\) is effectively multiplying the well-known logistic term in the prey equation, we call it a multiplicative Allee effect as opposed to the additive case found in [3–5, 41]. Our aim is to provide the Allee function \(A(x)\) with the minimum amount of functional properties such that it emerges as a generalisation of a number of existing models with multiplicative Allee effect; see [11, 24, 45] for instance. More concretely, we assume that \(A(x)\) is sufficiently smooth for every \(x\in \mathbb {R}\) and satisfies the following conditions:

  • (A.1) There exists a value \(m\in \mathbb {R}\) with \(|m|<k\), such that \(A(m)=0\).

  • (A.2) \(A'(x)>0\), for all \(x>0\).

  • (A.3) \(\lim _{x\rightarrow 0^+}A(x)=M<\infty \).

Note that \(A(x)\) is an increasing function, but it may not be bounded. In particular, conditions (A.1) and (A.2) ensure that there is a certain value \(m\) such that \(A(x)>0\) for every \(x>m\). Hence, if \(0<x<m\), then \(A(x)<0\) and the prey growth rate is negative for sufficiently small values of \(x\); this phenomenon is called a strong Allee effect [4]. On the other hand, if \(m\le 0\), one speaks of a weak Allee effect; see [5, 9, 41] for details.

On the other hand, the term \(\phi (x)\) models the per capita prey consumption by the predator or functional response. In the same spirit as with the Allee function, we do not assume an explicit form for \(\phi (x)\). Nevertheless, \(\phi (x)\) is taken to be smooth enough and bounded for every \(x\in \mathbb {R}\). More specifically, it is assumed that \(\phi (x)\) satisfies the following conditions:

  • (B.1) \(\phi (0)=0\).

  • (B.2) \(\phi '(x)>0,\) for all \(x\ge 0\).

  • (B.3) \(\lim _{x\rightarrow \infty }\phi (x)=N<\infty \).

For example, any such function \(\phi (x)\) that satisfies conditions (B.1)–(B.3) may have the explicit form of a Holling type II–IV functional response [35, 40, 44, 48, 49].

2.2 An extended system

It is convenient to treat the system (1) as a vector field in which each equation is a coordinate function. The vector field (1) is not defined along the axis \(\{x=0\}\). To deal with this, we provide an extension to the \(y\)-axis by means of the transformation

$$\begin{aligned} (x,y,t)\mapsto \left( x,ny,\frac{xt}{ns}\right) \end{aligned}$$
(2)

and the change of parameters

$$\begin{aligned} (r,k,s,n)\mapsto \left( r,k,\frac{1}{\beta },\frac{r\beta }{\alpha k}\right) . \end{aligned}$$
(3)

In this way, the new vector field, denoted as \(\mathbf {Y}\), is given as

$$\begin{aligned} \mathbf {Y}:\left\{ \begin{array}{rcl} \dot{x} &{} = &{} \alpha (k-x)A(x)\,x^2-\beta \,\phi (x)\,x\,y; \\ \dot{y} &{} = &{} y\,(x-y);\\ \end{array} \right. \end{aligned}$$
(4)

where \(\alpha ,\beta >0\).

The change of coordinates (2) and reparameterization (3) define a smooth orientation-preserving bijection between orbits of \(\mathbf {Y}\) restricted to \(\mathcal {D}\) and orbits of \(\mathbf {X}\) [7, 21, 28, 33]. Hence, the vector field \(\mathbf {Y}\) is a \(\mathcal {C}^{\infty }\)-qualitatively equivalent extension of \(\mathbf {X}\) to the entire first quadrant \(\overline{\mathcal {D}}:=\{(x,y) \in \mathbb {R}^2|\, x\ge 0,y \ge 0 \}\). In other words, the orbits of \(\mathbf {Y}\)—restricted to \(\mathcal {D}\)—have the same topological structure as those of \(\mathbf {X}\) and, additionally, \(\mathbf {Y}\) is well defined in the axis \(\{x=0\}\). As a consequence, it is more convenient to study system (4) throughout; see also [3, 4, 24–26].

2.3 Additional definitions and notation

The vector field \(\mathbf {Y}\) defines a flow \(\Phi ^t\) that determines the dynamics in \(\overline{\mathcal {D}}\). Any point \(\mathbf {x}^{*}=(x^{*},y^{*})\) such that \(\mathbf {Y}(\mathbf {x}^{*})=(0,0)\) is an equilibrium point of (4). If the eigenvalues \(\lambda _1,\lambda _2\) of the Jacobian matrix \(D\mathbf {Y}(\mathbf {x}^{*})\) have non-zero real part, one says that the equilibrium is hyperbolic. In such case, the Hartman–Grobman theorem [7, 28, 33] ensures that the real parts of these eigenvalues are the contraction/expansion rates of the solutions in a vicinity of \(\mathbf {x}^{*}\). Indeed, if both eigenvalues have negative (resp. positive) real part, the equilibrium is an attractor (resp. repeller); in particular, in either scenario, if \(\lambda _1\) and \(\lambda _2\) are complex conjugate, one says that \(\mathbf {x}^{*}\) is a stable or unstable focus, respectively, since the nearby orbits spiral into or out of a neighbourhood of the equilibrium.

If \(\mathbf {x}^{*}\) is a stable equilibrium, then there exists a neighbourhood \(U\) of \(\mathbf {x}^{*}\) that satisfies

$$\begin{aligned} \Phi ^t(U)\subset U \; (\forall t\ge 0) \,\mathrm{and}\, \bigcap _{t>0}\Phi ^t(U)=\mathbf {x}^{*}, \end{aligned}$$
(5)

that is, the set \(U\) is invariant and orbits within it converge to \(\mathbf {x}^{*}\) in forward time. The basin of attraction \(\mathcal {B}(\mathbf {x}^{*})\) of \(\mathbf {x}^{*}\) is then the set of all points in phase space that converge to \(\mathbf {x}^{*}\) in forward time; it can be defined as

$$\begin{aligned} \mathcal {B}(\mathbf {x}^{*})=\bigcup _{t\le 0}\Phi ^t(U), \end{aligned}$$
(6)

where \(U\subset \mathbb {R}^2\) is any open neighbourhood of \(\mathbf {x}^{*}\) satisfying (5).

If the eigenvalues \(\lambda _1\) and \(\lambda _2\) have opposite signs, \(\mathbf {x}^{*}\) is a hyperbolic saddle point. In this case, the Stable Manifold Theorem [28, 33] ensures that \(\mathbf {x}^{*}\) has a one-dimensional stable (resp. unstable) manifold that is tangent at \(\mathbf {x}^{*}\) to the eigenspace associated to \(\lambda _1<0\) (resp. \(\lambda _2>0\)). These invariant sets are (immersed) manifolds as smooth as \(\mathbf {Y}\) and are defined as

$$\begin{aligned} \begin{array}{rcl} W^{s}(\mathbf {x}^{*}) &{}=&{} \left\{ \mathbf {x} \in \mathbb {R}^2 \;|\; \Phi ^t(\mathbf {x}) \rightarrow \mathbf {x}^{*} \text{ as } t \rightarrow \infty \right\} ,\\ &{} &{}\\ W^{u}(\mathbf {x}^{*})&{} =&{} \left\{ \mathbf {x} \in \mathbb {R}^2 \;|\; \Phi ^t(\mathbf {x}) \rightarrow \mathbf {x}^{*} \text{ as } t \rightarrow -\infty \right\} , \end{array} \end{aligned}$$

respectively.

If (at least) one of the eigenvalues of \(\mathbf {x}^{*}\) has a zero real part, the equilibrium is non-hyperbolic and standard linear analysis does not apply. Typically, in such a case, the equilibrium undergoes a local bifurcation whose unfolding depends on the non-zero eigenvalue, if any, and higher order terms of the system (4) [28, 33]. Other techniques to study non-hyperbolic equilibria are the centre manifold theorem and desingularisation methods that enable one to find the local structure of orbits in a vicinity of \(\mathbf {x}^{*}\) [21].

The system (4) may also have a limit cycle \(\gamma \), that is, an isolated closed orbit. Such a periodic solution \(\gamma \) is stable (resp. unstable) if nearby orbits approach \(\gamma \) in forward (resp. backward) time. The basin of attraction \(\mathcal {B}(\gamma )\) of a stable limit cycle \(\gamma \) can be defined in an analogous way by replacing \(\gamma \) in (6) accordingly.

3 The family of models is well posed

Since \(\mathbf {Y}(x,0)=\left( \alpha \left( k-x-A(x)\right) x^2,0\right) ^T\) and \(\mathbf {Y}(0,y)=(0,-y^2)^T\), the vector field \(\mathbf {Y}\) is invariant along the coordinate axes. It follows that every orbit of \(\mathbf {Y}\) in the interior of the first quadrant—and by extension, every corresponding orbit of \(\mathbf {X}\)—remains inside that region.

Theorem 1

Under the assumptions (A.1)–(A.3) and (B.1)–(B.3), the solutions of the general system (4) are bounded.

Proof of Theorem 1

Our aim is to prove that no trajectory of \(\mathbf {Y}\) converges to infinity. The first task is to prove that for every \(\omega >k\), the set

$$\begin{aligned} \mathcal {D}_{\omega }:=\{(x,y)\in \mathbb {R}^2:\,0\le x<\omega ,0\le y<\omega \}\subset \overline{\mathcal {D}} \end{aligned}$$

is invariant by examining the dynamics in its boundary

$$\begin{aligned} \partial \mathcal {D}_{\omega }{=}V_{\omega }\cup H_{\omega }\cup \{(x,y):\,x{=}0\}\cup \{(x,y):\,y=0\}, \end{aligned}$$

where \(V_{\omega }:=\{(x,y):\,x=\omega ,y>0\}\) and \(H_{\omega }:=\{(x,y):\,y=\omega ,x>0\}\); see Fig. 1.

Fig. 1
figure 1

The invariant set \(\mathcal {D}_{\omega }\) defined for a suitable \(\omega >k\)

Full size image

Let us define the function \(f_{\omega }:\mathbb {R}^2\longrightarrow \mathbb {R},\) given by \(f_{\omega }(x,y)=x-\omega .\) For each \(\omega >0\), consider the vertical line

$$\begin{aligned} V_{\omega }=f_{\omega }^{-1}(0)\cap \overline{\mathcal {D}} \end{aligned}$$

in the \((x,y)\)-plane. Geometrically, \(f_{\omega }^{-1}(-\infty ,0)\cap \overline{\mathcal {D}}\) is a vertical stripe that extends in the \(y\)-direction and its boundary is \(\partial \left( f_{\omega }^{-1}(-\infty ,0)\cap \overline{\mathcal {D}}\right) =V_{\omega }\cup \{y=0\}\cup \{x=0\}.\) Similarly, let \(g_{\omega }:\mathbb {R}^2\longrightarrow \mathbb {R}\) be given by \(g_{\omega }(x,y)=y-\omega .\) For each \(\omega >0\), consider the horizontal line

$$\begin{aligned} H_{\omega }=g_{\omega }^{-1}(0)\cap \overline{\mathcal {D}} \end{aligned}$$

which forms the upper boundary of a horizontal stripe in the \((x,y)\)-plane. In this way,

$$\begin{aligned} \mathcal {D}_{\omega }=f_{\omega }^{-1}(-\infty ,0)\cap g_{\omega }^{-1}(-\infty ,0)\cap \overline{\mathcal {D}}; \end{aligned}$$

see Fig. 1.

With this construction in mind, we first prove that every solution of \(\mathbf {Y}\) on the line \(V_{\omega }\) enters the region \(f_{\omega }^{-1}(-\infty ,0)\) transversally; see Fig. 1. Since the vector field \(\mathbf {Y}\) is tangent to every solution, this is equivalent to prove that the dot product \(\mathbf {Y}\cdot \nabla f_{\omega }\) is negative in every point on the line \(V_{\omega }\) for a suitable choice of \(\omega >0\). We have that for every \((x,y)=(\omega ,y)\in V_{\omega }\):

$$\begin{aligned} \varphi _{\omega }(y)\!:=\!\mathbf {Y}\cdot \nabla f_{\omega }|_{V_{\omega }}\!=\!\alpha \,\omega ^2\left( k\!-\!\omega \right) A(\omega )\!-\!\beta \,\phi (\omega )\omega y, \end{aligned}$$

Geometrically, from conditions (B.1)–(B.3), the graph of \(\varphi _{\omega }(y)\) as a function of \(y\) is a straight line with negative slope \(-\beta \,\phi (\omega )\omega <0\) and intercept

$$\begin{aligned} \alpha \,\omega ^2\left( k-\omega \right) A(\omega ). \end{aligned}$$

From conditions (A.1) and (A.2), for every \(\omega >k\), the following inequalities hold:

$$\begin{aligned} k-\omega <0<A(\omega ). \end{aligned}$$
(7)

Hence, for every \(\omega >k\), \(\varphi _{\omega }(y)<0\) as desired.

On the other hand, following the same approach for the structure of \(\mathbf {Y}\) along the line \(H_{\omega }\):

$$\begin{aligned} \mathbf {Y}\cdot \nabla g_{\omega }|_{H_{\omega }}=\omega (x-\omega )<0, \end{aligned}$$

for every point \((x,y)=(x,\omega )\) with \(x<\omega \). Therefore, if \(0<x<\omega \), for every point \((x,y)\in H_{\omega }\), its associated trajectory crosses this line transversally towards the region \(g_{\omega }^{-1}(-\infty ,0)\).

Furthermore, \(\mathbf {Y}(0,y)=(0,-y^2)^T\) along the invariant axis \(x=0\) and, hence, trajectories converge to the origin \((x,y)=(0,0)\). On the other hand, along \(y=0\), \(\mathbf {Y}(x,0)=\left( \alpha \, x^2(k-x)A(x),0\right) ^T\). From (7), \(\alpha \, x^2(k-x)A(x)<0\) for every \(x\ge \omega >k\). As a consequence, no trajectory along the coordinate axes converges to infinity.

To sum up, for any arbitrary \(\omega >k\), every solution inside \(\mathcal {D}_{\omega }\) cannot leave this region again in forward time and, in particular, these solutions remain bounded. Therefore, for any given initial condition \((x(0),y(0))\in \overline{\mathcal {D}}\), there is a value \(\omega ^{*}>k\) such that the orbit \((x(t),y(t))\) of Y through the point \((x(0),y(0))\) is entirely contained, for every \(t>0\), in the bounded, invariant set \(\mathcal {D}_{\omega }\) for every \(\omega \ge \omega ^{*}.\) \(\square \)

4 Equilibrium points in the \(x\)-axis

The class of systems (4) always has an equilibrium point at the origin \(\mathbf {0}\in \mathbb {R}^2\) and at \((k,0)\). In the case of strong Allee effect, i.e. if \(m>0\), it can also have another equilibrium at \((m,0)\).

Lemma 1

Consider the family of models (4) and assume that conditions (A.1)–(A.3) and (B.1)–(B.3) hold. In particular, let \(m\in \mathbb {R}\) be as in (A.1). Then, the following holds:

  1. (a)

    The equilibrium \((k,0)\) is a hyperbolic saddle point.

  2. (b)

    If \(m>0\), the equilibrium \((m,0)\) is a hyperbolic repeller.

Proof of Lemma 1

The eigenvalues of the Jacobian matrix \(D\mathbf {Y}(k,0)\) are \(\lambda _1^k=-\alpha k^2A(k)<0\) and \(\lambda _2^k=k>0\). Similarly, if \(m>0\), the eigenvalues of \(D\mathbf {Y}(m,0)\) are \(\lambda _1^m=\alpha m^2(k-m)A'(m)>0\) and \(\lambda _2^m=m>0\). The results follow directly from the Hartman–Grobman theorem [7, 28, 33]. \(\square \)

By calculating the eigenvectors of \(D\mathbf {Y}(k,0)\), it is easy to find that the upper branch of the stable manifold \(W^s(k,0)\) of \((k,0)\) is contained in the \(x\)-axis.

Figure 2 shows sketches of the different possible qualitative dynamics in a neighbourhood of the origin. In the case of strong Allee effect, the origin is always a local attractor; see the basin of attraction \(\mathcal {B}(\mathbf {0})\) in panel (a). On the other hand, for a weak Allee effect, with \(m\le 0\), extinction is avoided in panels (b) and (c) where every solution in the interior of \(\mathcal {D}\) remains in this region. The fourth possibility—with \(m=0\) and additional conditions on parameters—is shown in panel (d) where a separatrix curve \(\varGamma \) appears and forms the boundary of a parabolic and an elliptic sector [21] in the basin of attraction \(\mathcal {B}(\mathbf {0})\) of the origin. In particular, orbits below \(\varGamma \) make an excursion in an elliptic fashion before converging to the origin.

Fig. 2
figure 2

The four possible qualitative behaviours of the dynamics near the origin in (4). In panel a, the origin is an attractor for all nearby orbits; in panels b and c, no orbit in the interior of \(\mathcal {D}\) converges to \(\mathbf {0}\). In panel d, a separatrix \(\varGamma \) appears and forms the boundary of the parabolic and elliptic components of the basin of attraction \(\mathcal {B}(\mathbf {0})\) of the origin

Full size image

In order to find and prove the different configurations in Fig. 2 in dependence of parameters, we define the following quantities:

$$\begin{aligned}&\mu : = 1-\alpha kA(0), \end{aligned}$$
(8)
$$\begin{aligned}&C_1(m): = \beta \phi '(0)-\alpha kA'(0), \end{aligned}$$
(9)

and

$$\begin{aligned} C_2(m):=2\alpha A'(0)+\beta \phi ''(0)-\alpha kA''(0). \end{aligned}$$
(10)

Theorem 2

Consider the family of models (4) and assume that conditions (A.1)–(A.3) and (B.1)–(B.3) hold. In particular, let \(m\in \mathbb {R}\) be as in (A.1). Then, the local dynamics near \(\mathbf {0}\) is as in Fig. 2 in a sufficiently small neighbourhood of \(\mathbf {0}\). More concretely,

  1. (a)

    If \(m>0\), the origin \(\mathbf {0}\) is a local attractor as in Fig. 2a. In addition, the same configuration occurs if \(m=0\), \(A'(0)=0\) and \(A''(0)<0\).

  2. (b)

    If \(m<0\) and \(\mu >0\), the origin \(\mathbf {0}\) has a parabolic repelling sector and a hyperbolic sector as in Fig. 2b. Additionally, the same configuration happens if \(m=0\) and \(C_1(0)<0\), or if \(m=0\), \(C_1(0)=0\) and \(C_2(0)<0\).

  3. (c)

    If \(m<0\) and \(\mu \le 0\), the origin \(\mathbf {0}\) has an hyperbolic sector as in Fig. 2c.

  4. (d)

    If \(m=0\) and \(C_1(0)>0\), the origin \(\mathbf {0}\) has a parabolic attracting sector and an elliptic sector as in Fig. 2d. Additionally, the same configuration happens if \(m=0\), \(C_1(0)=0\) and \(C_2(0)>0\), or if \(m=0\), \(A'(0)=0\) and \(A''(0)>0\).

Proof of Theorem 2

From Theorem 1, any orbit along the \(y\)-axis converges towards the origin. Moreover, from Lemma 1, if \(m>0\), any orbit along the \(x\)-axis converges to the origin provided its initial condition \((x(0),0)\) satisfies \(x(0)<m\). In turn, if \(m<0\), any orbit along the \(x\)-axis diverges from the origin provided its initial condition \((x(0),0)\) satisfies \(x(0)<k\). (The case \(m=0\) is analysed later in this proof). It remains to find the other separatrices near \(\mathbf {0}\), if any, and the dynamics restricted to them.

Since \(D\mathbf {Y}(\mathbf {0})\) is the null matrix, the origin \(\mathbf {0}\) is a non-hyperbolic equilibrium point of (4). Therefore, we use the blow-up technique [3, 21, 28] in order to get information about the dynamics near the origin. We consider the horizontal blow-up given by the transformation

$$\begin{aligned} (x,y)=\left( u,u\, v\right) \!. \end{aligned}$$
(11)

Note that (11) effectively ‘blows up’ the origin \((x,y)=(0,0)\) in the entire line \(u=0\) which consists of a continuum of equilibria. Hence, in addition, we use the time rescaling

$$\begin{aligned} t\mapsto t/u \end{aligned}$$
(12)

and obtain the new vector field, denoted as \(\widetilde{\mathbf {Y}}\), given by

$$\begin{aligned} \widetilde{\mathbf {Y}}:\left\{ \begin{array}{rcl} \dot{u} &{} = &{} u[\alpha (k-u)A(u)-\beta v \phi (u)]; \\ \dot{v} &{} = &{} v[1-v+\alpha (u-k)A(u)+\beta v \phi (u)],\\ \end{array} \right. \end{aligned}$$
(13)

which has only two equilibria of the form \((0,v)\). For a start, the origin \((u,v)=(0,0)\) is an equilibrium point of (13) with Jacobian matrix

$$\begin{aligned} D\widetilde{\mathbf {Y}}(0,0)=\left( \begin{array}{cc} 1-\mu &{} 0 \\ 0 &{} \mu \\ \end{array} \right) , \end{aligned}$$

where \(\mu \) is as in (8).

Moreover, there is a second equilibrium point on the \(v\)-axis at \((u,v)=(0,\mu )\) with associated Jacobian matrix

$$\begin{aligned} D\widetilde{\mathbf {Y}}(0,\mu )=\left( \begin{array}{cc} 1-\mu &{} 0 \\ V &{} -\mu \\ \end{array} \right) , \end{aligned}$$

where \( V:=-\mu \big (\alpha (kA'(0)-A(0))-\beta \mu \phi '(0)\big ).\)

Case 1: \(m>0\). If \(m>0\), then \(A(0)<0\) and one always has \(\mu >0\) and \(1-\mu <0\). Therefore, \((0,0)\) is a saddle and \((0,\mu )\) is an attractor of (13). Furthermore, the associated local invariant curves are the axes \(u=0\) and \(v=0\).

Hence, by taking the inverse of (11) and (12), the line \(u=0\)—including the point \((0,\mu )\)—is collapsed to the origin of system (4), the line \(v=0\) is mapped to \(y=0\) and the basin of attraction of \((0,\mu )\) is mapped to the region (locally) bounded by \(x=0\) and \(y=0\). Since (12) preserves time-orientation for \(u>0\), every orbit in this open region converges to the origin of (4) in forward time. Hence, \(\mathbf {0}\) is a local attractor as stated in part (a).

Case 2: \(m<0\). Analysed in a similar fashion to the previous case. We state the essential facts and leave the details to the interested reader. Condition \(m<0\) implies that \(A(0)>0\) and one has the following three cases:

Case 2.1: \(\mu >0\). The equilibrium \((0,0)\) is now a repeller. On the other hand, the equilibrium \((0,\mu )\)—which is now a saddle—is repelling along the curve \(\widetilde{\varGamma }:=\{(u,v)|\,v=\mu +Vu+O(u^2)\}\). (Note that the curve \(\widetilde{\varGamma }\) is a local approximation of the unstable manifold \(W^u(0,\mu )\) of \((0,\mu )\)). Hence, by ‘blowing-down’ to the system (4), \(\widetilde{\varGamma }\) is mapped to \(\varGamma :=\{(x,y)|\,y=\mu x+Vx^2+O(x^3)\,\}\), which is a local separatrix and the dynamics near \(\mathbf {0}\) is as in part (b).

Case 2.2: \(\mu <0\). Analogous to Case 2.1. The origin \((0,0)\) is a repeller and \((0,\mu )\) is a saddle in (13). However, by blowing-down to system (4), the slope of \(\varGamma \) at \(\mathbf {0}\) is \(\mu <0\) and, hence, this curve is contained in the exterior of \(\mathcal {D}\). Therefore, the local dynamics near \(\mathbf {0}\) is as in part (c).

Case 2.3: \(\mu =0\). The equilibrium \((0,\mu )\) collides with the origin of (13) and undergoes a transcritical bifurcation, i.e. their topological stability is interchanged [28, 33]; see Cases 2.1 and 2.2. By the continuous dependence of (13) on the model parameters, the local dynamics near \(\mathbf {0}\) is as in part (c).

Case 3: \(m=0\). This condition implies \(A(0)=0\) and \(\mu =1\). Hence, \(V\) is reduced to \(C_1(0)\) in (9). In what follows we simplify the notation and denote \(C_1(0)=C_1\) and \(C_2(0)=C_2\).

Both equilibria \((0,0)\) and \((0,1)\) in (13) have a zero eigenvalue and, hence, both are non-hyperbolic. In order to find the local dynamics near these equilibria, we calculate their centre manifolds \(W^c(0,0)\) and \(W^c(0,1)\), which are one-dimensional invariant manifolds associated with the zero eigenvalue of \((0,0)\) and \((0,1)\), respectively [28, 33]. A local approximation of \(W^c(0,1)\) in the \((u,v)\)-plane near \((0,1)\) is given as the graph of the function

$$\begin{aligned} v&= h(u):=1+C_1u+\frac{1}{4}\big (C_2+2\beta \phi '(0)C_1\big )u^2\\&+\,O(u^3), \end{aligned}$$

where \(C_1\) is as in (9) and \(C_2\) is as in (10). Analogously, in the \((u,v)\)-plane near \((0,0)\), the centre manifold \(W^c(0,0)\) is given locally as the graph of a function of the form \(v=O(u^n),\) for any \(n\) sufficiently large. Hence, we approximate \(W^c(0,0)\) locally as the axis \(\{v=0\}\).

From centre manifolds theory [28, 33], the dynamics near each equilibrium \((0,0)\) and \((0,1)\) is dominated by the vector field restricted to its centre manifold. For a start, the horizontal component of the vector field \(\widetilde{\mathbf {Y}}\) restricted to \(W^c(0,1)\) near \((0,1)\) is given as

$$\begin{aligned} \dot{u}|_{W^c(0,1)}=-C_1u^2-\frac{1}{4}\big (C_2+2\beta \phi '(0)C_1\big )u^3+O(u^4). \end{aligned}$$

Analogously, the horizontal component of the vector field \(\widetilde{\mathbf {Y}}\) restricted to \(W^c(0,0)\) near \((0,0)\) is given as

$$\begin{aligned} \dot{u}|_{W^c(0,0)}&= \alpha kA'(0)u^2-\frac{1}{2}\alpha \big (2A'(0)-A''(0)k\big )u^3\\&+\,O(u^4). \end{aligned}$$

The desired statements now follow directly from ‘blowing-down’ the dynamics in \(\widetilde{\mathbf {Y}}\) back to \(\mathbf {Y}\). In particular, in a neighbourhood of \(\mathbf {0}\) in (4), the inverse image of the curve \(\{(u,v)|\,v=h(u)\}\) by (11)–(12) is the invariant curve

$$\begin{aligned} \varGamma&=\{(x,y)|\,y=x+C_1x^2\\&\qquad +\frac{1}{4}\big (C_2+2\beta \phi '(0)C_1\big )x^3+O(x^4)\}. \end{aligned}$$

On the other hand, by blowing-down \(W^c(0,0)\), one obtains the \(x\)-axis in (4).

Finally, note that a vertical blow-up does not provide additional information.\(\square \)

In the previous proof, if the coefficients \(C_1\) and \(C_2\) vanish simultaneously, one needs to compute higher order terms in the expansion of the vector field \(\widetilde{\mathbf {Y}}\) restricted to \(W^c(0,1)\) in order to determine the dynamics in the case \(m=0\). A similar analysis has to be done for \(W^c(0,0)\) as well if \(A'(0)=A''(0)=0\). Given a Taylor expansion of \(\widetilde{\mathbf {Y}}\) around either equilibrium, a quick look at the coefficients up to order 4 suggests the following conjecture for the dynamics restricted to \(W^c(0,1)\) and \(W^c(0,0)\):

Conjecture

Consider the family of models (4) and assume that conditions (A.1)–(A.3) and (B.1)–(B.3) hold. In particular, let \(m=0\). If \(C_1=C_2=\cdots =C_{n-1}=0\) and \(C_n\ne 0\), then

$$\begin{aligned} \dot{u}|_{W^c(0,1)}=-M_CC_nu^{n+1}+O(u^{n+2}), \end{aligned}$$

with \(C_n=n\alpha A^{(n-1)}(0)+\beta \phi ^{(n)}(0)-\alpha kA^{(n)}(0)\) and where \(M_C>0\) is a constant.

Similarly, if \(A'(0)=A''(0)=\cdots =A^{(n-1)}(0)=0\) and \(A^{(n)}(0)\ne 0\), then

$$\begin{aligned} \dot{u}|_{W^c(0,0)}=M_AA^{(n)}(0)u^{n+1}+O(u^{n+2}), \end{aligned}$$

where \(M_A>0\) is a constant.

Nevertheless, regardless of how many vanishing coefficients one has in the corresponding Taylor expansion, condition \(A'(0)=0\) implies \(C_1>0\) and, hence, \(\dot{u}|_{W^c(0,1)}<0\) near \((0,1)\). On the other hand, \(C_1=0\) implies \(A'(0)>0\); as a consequence, \(\dot{u}|_{W^c(0,0)}>0\) near \((0,0)\). Therefore, the possible qualitative dynamics near the origin \(\mathbf {0}\) in (4) is always limited to any of the four options stated in Theorem 2.

5 Local stability of positive equilibria

Equilibrium points of (4) in the interior of \(\mathcal {D}\) are located on the diagonal \(y=x\). Let \(\mathbf {p}=(p,p)\) be any such equilibrium, with \(p>0\). It is straightforward to see that a necessary condition for \(\mathbf {p}\) to exist is:

  • (C.1) \((k-p)A(p)>0\).

Consider the change of parameters

$$\begin{aligned} (r,k,\beta )\mapsto \left( r,k,\frac{(k-p)A(p)}{\phi (p)}\right) \end{aligned}$$
(14)

and the transformation

$$\begin{aligned} (x,y,t)\mapsto \left( x+p,y+p,t\,\phi (p)\right) . \end{aligned}$$
(15)

Since conditions (B.1) and (B.2) ensure that \(\phi (p)>0\), (14) and (15) define a \(\mathcal {C}^{\infty }\)-equivalency between (4) and a new vector field, denoted as \(\mathbf {Z}\), given as the system

$$\begin{aligned} \mathbf {Z}:\left\{ \begin{array}{rcl} \dot{x} &{} = &{} \alpha (x{+}p)\left[ (x{+}p)(k{-}p{-}x)A(x{+}p)\phi (p)\right. \\ &{}&{}\left. -(k-p)(y+p)A(p)\phi (x+p)\right] ; \\ \dot{y} &{} = &{} (x-y)(y+p)\phi (p).\\ \end{array} \right. \end{aligned}$$
(16)

The advantage of (16) is that the equilibrium point of interest is now at the origin.

Let us define the following quantities:

$$\begin{aligned} S(k)&{:=}&(k-p)\left[ \phi '(p)A(p)-\phi (p)A'(p)\right] \nonumber \\&+A(p)\phi (p), \end{aligned}$$
(17)
$$\begin{aligned} H(\alpha ,k)&{:=}&\alpha \left[ (k-p)A(p)\phi (p)-\, p\,S(k)\right] -\phi (p).\nonumber \\ \end{aligned}$$
(18)

Lemma 2

Consider the family of models (16) and assume that conditions (A.1)–(A.3), (B.1)–(B.3) and (C.1) are met. Then, the following holds:

  1. (a)

    If \(S>0\) and \(H<0\), the equilibrium \((0,0)\) is a hyperbolic attractor.

  2. (b)

    If \(S>0\) and \(H>0\), the equilibrium \((0,0)\) is a hyperbolic repeller.

  3. (c)

    If \(S<0\), the equilibrium \((0,0)\) is a hyperbolic saddle.

  4. (d)

    If \(S>0\) and \(H=0\), the equilibrium \((0,0)\) undergoes a Hopf bifurcation.

Proof of Lemma 2

The eigenvalues of the Jacobian matrix \(D\mathbf {Z}(0,0)\) are

$$\begin{aligned} \lambda _{1,2}^p=\,\frac{p}{2}\left( H\pm \sqrt{H^2-4\,\alpha \,p\,\phi (p)S}\right) . \end{aligned}$$

Since \(\alpha >0\), \(p>0\) and \(\phi (p)>0\), statements (a), (b) and (c) follow directly from the Hartman–Grobman theorem.

In the particular case that \(H^2\le 4\,\alpha \,p\,\phi (p)S\) and \(S>0\), the eigenvalues of \((0,0)\) are complex conjugate with real part \(Re(\lambda _{1,2}^p)=\dfrac{pH}{2}\) and \((0,0)\) is a focus. Hence, if \(H=0\), the eigenvalues \(\lambda _{1,2}^p\in \mathbb {C}\) cross the imaginary axis and the equilibrium \((0,0)\) undergoes a Hopf bifurcation. \(\square \)

Next, we give conditions such that our family of models undergoes a Bogdanov–Takens bifurcation under suitable parameter variation.

Let us define the following quantities:

$$\begin{aligned} G_1&=A^2(p)+p[\phi '(p)A(p)-\phi (p)A'(p)],\\ G_2&=\phi (p)\left( A(p)A''(p)-2(A'(p))^2\right) \\&\quad +A(p)\left( 2A'(p)\phi '(p)-A(p)\phi ''(p)\right) ,\\ J_{20}&=\!\phi (p)\left( 4(A'(p))^2-2A(p)A''(p)\!+\!pA'(p)A''(p)\right) \\&\quad +A(p)\left( 2A(p)\phi ''(p)\!-\!\phi '(p)(4A'(p)\!+\!pA''(p))\right) \!,\\ J_{11}&=2A^2\phi '(p)+p\left( 3p\phi (p)[2A^2(p)\phi ''(p)-J_{20}]\right. \\&\quad \left. +A^2(p)\phi ''(p)(1-6p\phi (p))\right) . \end{aligned}$$

Theorem 3

Suppose that the family of vector fields (16) satisfies assumptions (A.1)–(A.3), (B.1)–(B.3) and (C.1), and let \(S\) and \(H\) be as in (17) and (18), respectively. Additionally, suppose that the following conditions hold:

  • (BT.1) \(G_1\ne 0\).

  • (BT.2) \(G_2\ne 0\).

  • (BT.3) \(J_{20}\ne 0\).

  • (BT.4) \(A^2(p)\left( 2\phi '(p)+p\phi ''(p)\right) -3pJ_{20}\ne 0.\)

Then, if \(S=0\) and \(H=0\), the origin \((0,0)\) in (16) undergoes a codimension-two Bogdanov–Takens bifurcation.

Proof of Theorem 3

The proof is based on the construction in the “Appendix”, namely, it suffices to verify that the theorem therein is satisfied: We prove the existence of a germ of a Bogdanov–Takens bifurcation and show that the conditions (BT.1)–(BT.4) ensure that the system is locally topologically equivalent to a normal form of the Bogdanov–Takens bifurcation. We refer to the “Appendix” and the references therein for the derivation of the genericity conditions that need to be verified during this proof.

For the sake of clarity, it is convenient to state the dependence of the vector field \(\mathbf {Z}\) on parameters \(\alpha \) and \(k\) explicitly. Hence, throughout this proof we write

$$\begin{aligned} \mathbf {Z}=\mathbf {Z}(x,y;\alpha ,k) \end{aligned}$$

and denote the Jacobian matrix of \(\mathbf {Z}\) with respect to the variables \((x,y)\) as

$$\begin{aligned} \frac{\partial \mathbf {Z}}{\partial (x,y)}(x,y;\alpha ,k). \end{aligned}$$

Step 1. We verify that the singularity has a double zero eigenvalue with geometric multiplicity one.

From condition (C.1), \(S=0\) implies that

$$\begin{aligned} \varphi :=\phi '(p)A(p)-\phi (p)A'(p)\ne 0. \end{aligned}$$
(19)

Hence, equations \(S(k)=0\) and \(H(\alpha ,k)=0\) determine the bifurcation point \((\alpha ,k)=(\alpha ^{*},k^{*})\) implicitly in the form:

$$\begin{aligned} \alpha ^{*}&= -\,\frac{1}{A(p)}\,\frac{\varphi }{A(p)\phi (p)}; \end{aligned}$$
(20)
$$\begin{aligned} k^{*}&= p-\frac{A(p)\phi (p)}{\varphi }. \end{aligned}$$
(21)

From the proof of Lemma 2, at \((\alpha ,k)=(\alpha ^{*},k^{*})\), the equilibrium \((x,y)=(0,0)\) has a double zero eigenvalue. Moreover, the linear part of \(\mathbf {Z}\) with respect to \((x,y)\) at \((0,0)\) is

$$\begin{aligned} \frac{\partial \mathbf {Z}}{\partial (x,y)}(0,0;\alpha ^{*},k^{*})=\left( \begin{array}{cc} p\phi (p) &{} -p\phi (p) \\ p\phi (p) &{} -p\phi (p) \\ \end{array} \right) , \end{aligned}$$
(22)

and its generalised eigenvectors are

$$\begin{aligned} \mathbf {v}_1&= (1,1)^T\; \mathrm{and} \end{aligned}$$
(23)
$$\begin{aligned} \mathbf {v}_2&= \left( 1,\frac{p\phi (p)-1}{p\phi (p)}\right) ^T. \end{aligned}$$
(24)

In particular, note that the Jacobian matrix \(\frac{\partial \mathbf {Z}}{\partial (x,y)}\) at \((0,0;\alpha ^{*},k^{*})\) is not the null matrix.

Step 2. The next goal is to prove that the following genericity condition of a Bogdanov–Takens bifurcation is met, namely, that the map

$$\begin{aligned} \Psi :\mathbb {R}^4\rightarrow \mathbb {R}^4,\,(x,y,\alpha ,k)\mapsto (\mathbf {Z},T,\Delta ) \end{aligned}$$

is regular at \((x,y,\alpha ,k)=(0,0,\alpha ^{*},k^{*})\), where \(T\) and \(\Delta \) are the trace and determinant of the matrix

$$\begin{aligned} \frac{\partial \mathbf {Z}}{\partial (x,y)}(x,y;\alpha ,k), \end{aligned}$$

respectively. In particular, note that \(T(0,0;\alpha ,k)=pH(\alpha ,k)\) and \(\Delta (0,0;\alpha ,k)=\alpha p^3\phi (p)S(k)\).

After some calculations, \(\det D\Psi (0,0,\alpha ^{*},k^{*})\) can be written as the following product:

$$\begin{aligned}&\det D\Psi (0,0,\alpha ^{*},k^{*})=-p^3\phi ^2(p)(k^{*}-p)\\&\quad \times \left( \frac{\partial \Delta }{\partial x}(0,0,\alpha ^{*},k^{*})+p^2\phi (p)\phi '(p)\right) G_1, \end{aligned}$$

where

$$\begin{aligned}&\frac{\partial \Delta }{\partial x}(0,0,\alpha ^{*},k^{*})=\frac{-p^2\phi (p)}{A^2(p)}\bigg (2\varphi \,p\,A'(p)\bigg .\\&\quad \bigg .+A(p)\big (3\varphi +\phi (p)(3A'(p)+pA''(p))\big )\\\nonumber&\quad -A^2(p)(2\phi '(p)+p\phi ''(p))\bigg ). \end{aligned}$$

Note that condition (C.1) prevents the term \(k^{*}-p\) in \(\det D\Psi (0,0,\alpha ^{*},k^{*})\) to vanish. Furthermore, from condition (BT.1), \(G_1\ne 0\). Moreover, the remaining factor

$$\begin{aligned} \frac{\partial \Delta }{\partial x}(0,0,\alpha ^{*},k^{*})+p^2\phi (p)\phi '(p)=\frac{-p^3\phi (p)}{A^2(p)}\,G_2\ne 0, \end{aligned}$$

after some algebraic manipulation and thanks to condition (BT.2). Therefore,

$$\begin{aligned} \det D\Psi (0,0,\alpha ^{*},k^{*})\ne 0 \end{aligned}$$
(25)

and the map \(\Psi \) is regular at \((x,y,\alpha ,k)=(0,0,\alpha ^{*},k^{*})\).

Step 3. We now prove that \(\mathbf {Z}(x,y;\alpha ,k)\) satisfies the remaining genericity conditions to exhibit the Bogdanov–Takens bifurcation; we refer to the “Appendix” again.

Let \(\mathbf {P}=[\mathbf {v}_1, \mathbf {v}_2]\) be the matrix whose columns are \(\mathbf {v}_1\) and \(\mathbf {v}_2\); see (23) and (24). Next, consider the following change of coordinates:

$$\begin{aligned} \left( \begin{array}{c} u\\ v \\ \end{array} \right) = \mathbf {P}^{-1}\left( \begin{array}{c} x\\ y \\ \end{array} \right) , \end{aligned}$$
(26)

Note that the origin \((x,y)=(0,0)\) is mapped by (26) to \((u,v)=(0,0)\). Then, the vector field given by

$$\begin{aligned} \mathbf {J}=\mathbf {P}^{-1}\circ \mathbf {Z}\circ \mathbf {P}, \end{aligned}$$

is \(\mathcal {C}^{\infty }\)-conjugated to \(\mathbf {Z}\).

Taking a Taylor expansion of \(\mathbf {J}(u,v;\alpha ,k)\) with respect to \((u,v)\) around \((u,v)=(0,0)\) and evaluating at \((\alpha ,k)=(\alpha ^{*},k^{*})\), one obtains

$$\begin{aligned} \left( \begin{array}{c} \dot{u}\\ \dot{v} \\ \end{array} \right)&= \left( \begin{array}{cc} 0 &{} 1 \\ 0 &{} 0 \\ \end{array} \right) \left( \begin{array}{c} u\\ v \\ \end{array} \right) \\&\quad + \frac{1}{A^2(p)} \left( \begin{array}{c} a_{20}u^2\!+\!O(||(u,v)||^3)\\ b_{20}u^2\!+\!b_{11}uv\!+\!O(||(u,v)||^3),\\ \end{array} \right) \end{aligned}$$

where

$$\begin{aligned}&a_{20}=3p\left( p\phi (p)-1\right) J_{20},\nonumber \\&b_{20}=-3p^2\phi (p)J_{20},\nonumber \\&\mathrm{and} \; b_{11}=J_{11}. \end{aligned}$$
(27)

Condition (BT.3) ensures that

$$\begin{aligned} b_{20}\ne 0. \end{aligned}$$
(28)

Furthermore, after some algebraic manipulation, one obtains

$$\begin{aligned} a_{20}+b_{11}=A^2(p)\left( 2\phi '(p)+p\phi ''(p)\right) -3pJ_{20}\ne 0, \end{aligned}$$
(29)

because of condition (BT.4).

Therefore, (22) and inequalities (25), (28) and (29) ensure that the genericity conditions of a Bogdanov–Takens normal form are satisfied; see the “Appendix”. Hence, there exists a smooth, invertible transformation of coordinates, an orientation-preserving time rescaling, and a reparametrization such that, in a sufficiently small neighbourhood of \((x,y,\alpha ,k)=(0,0,\alpha ^{*},k^{*})\), the system (16) is topologically equivalent to one of the following normal forms of a Bogdanov–Takens bifurcation:

$$\begin{aligned} \left\{ \begin{array}{rcl} \dot{\xi }_1 &{} = &{}\xi _2, \\ \dot{\xi }_2 &{} = &{} \beta _1+\beta _2 \xi _2 +\xi _2^2\pm \xi _1\xi _2,\\ \end{array} \right. \end{aligned}$$
(30)

where the sign of the term \(\xi _1\xi _2\) in (30) is determined by the sign of \(b_{20}(a_{20}+b_{11})\). \(\square \)

Under the conditions of Theorem 3, the unfolding of the Bogdanov–Takens point in (16) includes a single limit cycle, which can be either stable or unstable. Conditions for either case are collected in the following corollary.

Corollary 1

Under the hypotheses of Theorem 3, there is an open region in parameter space such that system (16) has a unique small-amplitude limit cycle that surrounds the origin \( (x,y) = (0,0)\). Moreover, the limit cycle is stable if \(\hbox {sign}\left( b_{20}(a_{20}+b_{11})\right) =-1\), and it is unstable if \(\hbox {sign}\left( b_{20}(a_{20}+b_{11})\right) =1\), where \(b_{20}\), \(a_{20}\) and \(b_{11}\) are as in (27).

Proof of Corollary 1

The result follows from the unfolding of the normal form (30) of the Bogdanov–Takens bifurcation. More precisely, the associated Hopf bifurcation near the codimension-two point is either supercritical if \(\hbox {sign}\left( b_{20}(a_{20}+b_{11})\right) =-1\), or subcritical if \(\hbox {sign}\left( b_{20}(a_{20}+b_{11})\right) =1\), which gives rise to a stable or unstable limit cycle, respectively; we refer to [33] for details.

In the proof of Theorem 3 , the implicit function theorem guarantees that there exists an open neighbourhood \(V\) of \((\alpha ,k)=(\alpha ^{*},k^{*})\), in which the corresponding terms \(b_{20}=b_{20}(\alpha ,k)\), \(a_{20}=a_{20}(\alpha ,k)\) and \(b_{11}=b_{11}(\alpha ,k)\) in the Taylor expansion of system \(\mathbf {J}(u,v;\alpha ,k)\) near \((u,v)=(0,0)\) are such that the expression \(b_{20}(a_{20}+b_{11})\) does not vanish and, hence, it is either positive or negative. This ensures that the Hopf bifurcation in \(V\) is of codimension-one and rules out the possibility of additional infinitesimal limit cycles. Finally—from the unfolding of the Bogdanov–Takens point [33], under suitable variation of \((\alpha ,k)\) in \(V,\) this limit cycle disappears at a codimension-one homoclinic bifurcation and no extra bifurcations of limit cycles occur. \(\square \)

Note that this corollary does not dismiss the possibility of additional limit cycles under different parameter regimes. Indeed, bifurcations of higher codimension may occur, such as generalised Hopf bifurcations, giving rise to the coexistence of multiple limit cycles [4]. In addition, whenever one of the conditions (BT.1)–(BT.4) is violated, the system may undergo a higher codimension Bogdanov–Takens bifurcation. In either case, one has to look at higher order terms in suitable normal forms in order to characterise the associated unfolding.

6 Bifurcation analysis

Theorem 3 gives a criterium to detect a Bogdanov–Takens bifurcation in (4). Under the hypotheses of Theorem 3, only two types of unfoldings are possible depending on the stability of the bifurcated limit cycle, which is determined by Corollary 1. Figures 3a and 4a show topological sketches of the two possible bifurcation diagrams in the \((\alpha ,k)\)-plane near the Bogdanov–Takens point—labelled as \(\mathbf {BT}\)—in the case when the limit cycle is stable and unstable, respectively. In both cases, these diagrams remain valid in a sufficiently small neighbourhood of the \(\mathbf {BT}\) point. The accompanying panels show sketches of the qualitative dynamics near the point \(\mathbf {p}\) for representative parameter values around the codimension-two point \(\mathbf {BT}\).

Fig. 3
figure 3

The unfolding of the Bogdanov–Takens bifurcation in panel a and corresponding sketches of the dynamics in panels b–f near the point \(\mathbf {p}\) in the case when the Hopf bifurcation is supercritical

Full size image
Fig. 4
figure 4

The unfolding of the Bogdanov–Takens bifurcation in panel a and corresponding sketches of the dynamics in panels b–f near the point \(\mathbf {p}\) in the case when the Hopf bifurcation is subcritical

Full size image

Let us first describe in more detail the elements in the bifurcation diagrams. From Lemma 2, equation \(S(k)=0\) in (17) defines a locus of saddle-node bifurcation. Geometrically, the curve \(\{(\alpha ,k)\in \mathbb {R}^2_+:\,S(k)=0\}\)—denoted as \(\mathbf {SN}\) in Figs. 3a and 4a—is a horizontal line for \(k=k^{*}\) fixed in the first quadrant of the \((\alpha ,k)\)-plane; see (21). Moreover, the curve \(\mathbf {SN}\) indicates the presence of a secondary equilibrium point \(\mathbf {q}=(q,q)\), with \(q>0\), which is a hyperbolic saddle for parameter values in the half plane \(k<k^{*}\).

On the other hand, the equation \(H(\alpha ,k)=0\) in (18) defines a locus of Hopf bifurcation. Moreover, since \(\phi (p)>0\), condition \(H(\alpha ,k)=0\) implies that

$$\begin{aligned} \frac{\partial H}{\partial \alpha }=(k-p)A(p)\phi (p)-pS>0. \end{aligned}$$
(31)

Hence, the Hopf bifurcation curve in the \((\alpha ,k)\)-plane is defined implicitly as the graph of the function

$$\begin{aligned} \alpha =\alpha _H(k):=\frac{\phi (p)}{(k-p)A(p)\phi (p)-pS(k)}. \end{aligned}$$
(32)

The Hopf bifurcation curve \(\{(\alpha ,k))\in \mathbb {R}^2_+:\,\alpha =\alpha _H(k)\}\)—denoted as \(\mathbf {H}\) in Figs. 3a and 4a—meets the curve \(\mathbf {SN}\) at the Bogdanov–Takens point \(\mathbf {BT}\) located at \((\alpha ^{*},k^{*})\) defined in (20)–(21), where \(\alpha _H(k^{*})=\alpha ^{*}\). Note that, from condition \(k^{*}>0\) one obtains

$$\begin{aligned} 0<1/p<\frac{\varphi }{A(p)\phi (p)}. \end{aligned}$$

In addition, since \(\alpha ^{*}>0\) in (20) and \(\phi (p)>0\), we have \(A(p)<0\) and \(\varphi <0\). In this way, since \(\frac{dS}{dk}=\varphi <0\), the set \(S^{-1}(0,\infty )\) corresponds to the region where \(k<k^{*}\). On the other hand, if \(\epsilon >0\) is sufficiently small, then

$$\begin{aligned}&(k^{*}-\epsilon -p)A(p)\phi (p)-\\&\quad pS(k^{*}-\epsilon )>(k^{*}-p)A(p)\phi (p). \end{aligned}$$

Therefore, evaluating in (32), one obtains \(\alpha _H(k^{*}-\epsilon )<\alpha _H(k^{*})\). Hence, the curve \(\mathbf {H}\) is locally increasing near \((\alpha ^{*},k^{*})\), and is entirely contained in the region \(S^{-1}(0,\infty )\) in the first quadrant of the \((\alpha ,k)\)-plane. Moreover, from (31), the region \(H^{-1}(-\infty ,0)\) is always located on the left side of the curve \(\mathbf {H}\), i.e. for values of \((\alpha ,k)\) with \(\alpha <\alpha _H(k)\).

In addition, the unfolding of a Bogdanov–Takens bifurcation includes a homoclinic bifurcation curve that emerges from the codimension-two singularity as well [13, 28, 33]. For any point \((\alpha ,k)\) along this bifurcation curve, denoted as hom in Figs. 3a and 4a, a homoclinic orbit connects the saddle point \(\mathbf {q}\) in both forward and backward time to form a loop. In this way, the bifurcation curves SN, H and \(hom\) separate four different open regions near the point BT in the \((\alpha ,k)\)-plane, where the dynamics is topologically non-equivalent. The analytical form and exact location of the curve hom, in relation to the other bifurcation curves, depend on the critical parameters in the versal unfolding of the normal form of (16) at the Bogdanov–Takens bifurcation [13, 33]. Nevertheless, one can still sketch its relative position depending on the nature of the Hopf bifurcation and the bifurcated limit cycle; see Corollary 1.

Figure 3a shows the case when the Hopf bifurcation is supercritical and the curve hom is located in the region \(H^{-1}(0,\infty )\), i.e. on the right side of \(\mathbf {H}\). For any point \((\alpha ,k)\) in region 1, the equilibrium \(\mathbf {p}\) is a stable focus; see Fig. 3b. In particular, one of the branches of the one-dimensional unstable manifold \(W^u(\mathbf {q})\) (which is rendered as a red curve) is contained in the basin of attraction \(\mathcal {B}(\mathbf {p})\) of \(\mathbf {p}\)—which is highlighted as a shaded region—and, hence, approaches the attractor \(\mathbf {p}\) in an exponential spiral. The basin boundary is given by the stable manifold \(W^s(\mathbf {q})\) of \(\mathbf {q}\) which is rendered as a blue curve. The focus \(\mathbf {p}\) undergoes a supercritical Hopf bifurcation when the point \((\alpha ,k)\) is on the curve \(\mathbf {H}\). At the moment of the Hopf bifurcation, the focus \(\mathbf {p}\) is not hyperbolic but remains an attractor; see Fig. 3c. If \((\alpha ,k)\) crosses \(\mathbf {H}\) towards region 2a, a small stable limit cycle \(\gamma \) appears surrounding the focus \(\mathbf {p}\), which is now a hyperbolic repeller; see Fig. 3d. The basin \(\mathcal {B}(\gamma )\) can be seen as the continuation of \(\mathcal {B}(\mathbf {p})\) after the supercritical Hopf bifurcation. As the point \((\alpha ,k)\) approaches the curve hom, the limit cycle \(\gamma \) increases both its amplitude and its period until it becomes a homoclinic orbit \(\gamma _h\) at the moment when the point \((\alpha ,k)\) reaches the curve hom in Fig. 3e. At this global bifurcation, the invariant manifolds \(W^u(\mathbf {q})\) and \(W^s(\mathbf {q})\) of \(\mathbf {q}\) intersect along the homoclinic loop \(\gamma _h\). Note that the homoclinic orbit \(\gamma _h\) is internally stable and, hence, it has a basin of attraction \(\mathcal {B}(\gamma _h)\) whose boundary is \(\partial \mathcal {B}(\gamma _h)=\gamma _h\cup \mathbf {p}\). In particular, the focus \(\mathbf {p}\) remains a hyperbolic repeller. As the point \((\alpha ,k)\) enters the region \(\mathbf {3}\) by crossing the curve \(hom\) in Fig. 3f, the homoclinic orbit is broken and the stable manifold \(W^s(\mathbf {q})\), when followed in backward time, approaches the unstable focus \(\mathbf {p}\). Finally, when the point \((\alpha ,k)\) crosses the curve SN from either region 1 or 3, the two equilibria \(\mathbf {p}\) and \(\mathbf {q}\) collide with one another and no longer exist in the region \(k>k^{*}\), labelled as 4.

In turn, Fig. 4 shows the unfolding of the Bogdanov–Takens point when the Hopf bifurcation is subcritical and the homoclinic bifurcation curve hom is located in the region \(H^{-1}(-\infty ,0)\), i.e. on the left side of \(\mathbf {H}\). In this case, if the point \((\alpha ,k)\) approaches the curve \(hom\) from region \(\mathbf {1}\), the invariant manifolds \(W^u(\mathbf {q})\) and \(W^s(\mathbf {q})\) approach one another until they intersect along the homoclinic orbit \(\gamma _h\); see Fig. 4b, c. Note that the focus \(\mathbf {p}\) remains a hyperbolic attractor at the moment of the homoclinic bifurcation. However, its basin of attraction \(\mathcal {B}(\mathbf {p})\) is now bounded by \(\gamma _h\). In region \(\mathbf {2b}\), the homoclinic orbit is broken and a global limit cycle \(\gamma \) bifurcates. The periodic orbit \(\gamma \) is unstable and bounds the basin \(\mathcal {B}(\mathbf {p})\); see Fig. 4d and compare to Fig. 3d. Ultimately, this limit cycle disappears at the subcritical Hopf bifurcation along the curve H and \(\mathbf {p}\) becomes an unstable focus; see Fig. 4e, f.

Figures 3 and 4 are helpful in that they unravel the role of the Bogdanov–Takens point \((\alpha ^{*},k^{*})\) as an organisation centre for the dynamics of the system (16). In this way, we are able to understand the topological mechanisms for (dis)appearance of limit cycles via both local and global phenomena given by Hopf and homoclinic bifurcations, respectively, and the associated configuration of attractors, basins of attraction and separatrices in the phase plane.

7 Examples

Among the explicit forms for the Allee function that satisfy conditions (A.1)–(A.3), one of the most common is

$$\begin{aligned} A(x)=x-m, \end{aligned}$$
(33)

where \(m\) is as in condition (A.1); see also [14, 24, 45]. In particular, by replacing (33) in (1), the factor \(r (x - m)\) represents a decrease of the population growth rate at low densities due to the Allee effect. Another possible concrete form for the Allee function is given by

$$\begin{aligned} A(x)=1-\frac{m+c}{x-c}, \end{aligned}$$
(34)

where \(m\) is as in condition (A.1) and \(c>0\) is an additional parameter. The Allee function (34), proposed in [11], is a generalisation of a model first studied in [12, 17, 18] in the case \(c=0\), where the behaviour of the prey population approaches a logistic-type of growth as \(x\rightarrow \infty \), provided the predator population is sufficiently low.

On the other hand, the function \(\phi (x)\) may have the explicit form of a Holling type functional response [35, 40, 44, 48, 49]:

$$\begin{aligned} \phi (x)=\left\{ \begin{array}{rl} \dfrac{N x}{x+a}&{} \hbox {(Type II)}; \\ &{}\\ \dfrac{N x^{\nu }}{x^{\nu }+a^{\nu }}&{}\hbox {(Type III)};\\ &{}\\ \dfrac{N x}{x^2+a}&{}\hbox {(Type IV)};\\ \end{array} \right. \end{aligned}$$
(35)

where \(N\) is as in condition (B.3), and \(a>0\) and \(\nu >1\) are additional parameters.

For instance, in the case that \(A(x)\) is given as in (33) and assuming \(\phi (x)\) is as in (35)-Type II, the quantities \(\mu \), \(C_1(0)\) and \(C_2(0)\) in Theorem 2 take the particular forms in Table 1. Also shown are the functions \(S(k)\) and \(H(\alpha ,k)\) that determine saddle-node and Hopf bifurcations in Theorem 3, as well as the particular form of the Bogdanov–Takens point \((\alpha ^{*},k^{*})\). The same quantities are shown in Tables 2, 3, 4 and 5 in other particular combinations of Allee functions (33)–(34) and functional responses (35). In particular, in the case when \(A(x)=x-m\) and \(\phi (x)\) is a Holling type III functional response, if \(m=0\) one obtains \(C_1(0)=-\alpha k<0\) and \(A'(0)=1\); see Tables 2 and 3. Hence, in these particular cases, the hypotheses of part (d) in Theorem 2 do not hold for any combination of parameter values.

Table 1 The relevant functions in Theorems 2 and 3, and the Bogdanov–Takens point in the case when the Allee function \(A(x)\) is given as in (33) and \(\phi (x)\) is a Holling-type II functional response as in (35)
Full size table
Table 2 The relevant functions in Theorems 2 and 3, and the Bogdanov–Takens point in the case when the Allee function \(A(x)\) is given as in (33) and \(\phi (x)\) is a Holling-type III functional response as in (35) with \(\nu =2\)
Full size table
Table 3 The relevant functions in Theorems 2 and 3, and the Bogdanov–Takens point in the case when the Allee function \(A(x)\) is given as in (33) and \(\phi (x)\) is a Holling-type III functional response as in (35) with \(\nu >1\), \(\nu \ne 2\)
Full size table
Table 4 The relevant functions in Theorems 2 and 3, and the Bogdanov–Takens point in the case when the Allee function \(A(x)\) is given as in (33) and \(\phi (x)\) is a Holling-type IV functional response as in (35)
Full size table
Table 5 The relevant functions in Theorems 2 and 3, and the Bogdanov–Takens point in the case when the Allee function \(A(x)\) is given as in (34) and \(\phi (x)\) is a Holling-type II functional response as in (35)
Full size table

Nevertheless, Tables 1, 2, 3, 4 and 5 serve the purpose of illustrating the procedure to find the explicit forms of relevant functions for the bifurcations and dynamical behaviour of interest for any model vector field of the general form (1) whenever the particular conditions of the theorems are satisfied. In addition, the results in these tables may also form the basis for further investigation of the corresponding particular models.

8 Discussion

In this work we introduced a class of general two-dimensional predator–prey models which present the main mathematical characteristics of a multiplicative Allee effect on prey population \(x=x(t)\). The proposed model vector fields are described by just a minimum amount of explicit terms and yet are able to describe and illustrate some of the main dynamical features of the species interaction that are common in a number of concrete Leslie–Gower-type examples.

By means of a topologically equivalent extension of the model vector fields to the entire first quadrant and a suitable reparameterization, we prove that our family of models is well posed in the sense that any realistic solution remains both non-negative and bounded. In particular, our approach involved the construction, for any given trajectory, of an invariant set in the first quadrant which turned out to be a trapping set for that particular orbit.

We also examined the existence and local stability of equilibrium points in the absence of predators. One of such equilibria appears when the prey population density \(x\) is equal to its carrying capacity \(k\), and it is always a saddle point. A second equilibrium exists—a repeller—in the case when the Allee effect is of strong type.

On the other hand, the study of the equilibrium at the origin \((x,y)=(0,0)\) in the extended system emerged as a particular challenge, since this point is always non-hyperbolic. By means of desingularization techniques, it was found that the local stability of the origin depends on the sign of parameter \(m\) (which determines either a weak or strong Allee effect), the Allee function \(A(x)\), the functional response \(\phi (x)\) and their respective derivatives at zero. More concretely, we state specific conditions on the model such that the origin has an open two-dimensional basin of attraction and, hence, both species may extinguish simultaneously. Furthermore, we found the explicit boundaries of this basin of attraction near the origin.

Next, we treated the bifurcations of equilibria in the interior of the first quadrant. For any such equilibrium, the proportion of prey and predator densities is equal to parameter \(n\), i.e. the same proportion between the carrying capacity of the predator and the prey abundance. We found the conditions on the model parameters, on the Allee function \(A(x)\) and on the functional response \(\phi (x)\), such that an equilibrium undergoes saddle-node, Hopf and Bogdanov–Takens bifurcations. In particular, specific genericity conditions are stated such that the Bogdanov–Takens bifurcation is of codimension-two. In doing so, we find the boundaries of an open region in parameter space for which the system has a limit cycle, which can be either stable or unstable depending on certain coefficients in the normal form of the Bogdanov–Takens bifurcation. Moreover, we also describe how this limit cycle converges to a homoclinic orbit under parameter variation. This is illustrated by means of bifurcation diagrams which were constructed from careful examination of the bifurcation loci and analytical results from bifurcation theory. In this way, our findings shed light on the conditions our model has to fulfil in order to the codimension-two Bogdanov–Takens point to be an organisation centre in terms of basins of attraction, coexistence of both species and multi-stability of the ecological system. In particular, our analytical results are applied to a number of particular models with explicit forms for the Allee function \(A(x)\) and Holling-types functional response \(\phi (x)\). In every case, the relevant quantities and bifurcation values that determine the dynamics in our results are calculated explicitly.

Altogether, the results in this paper provide a theoretical basis for further investigation in concrete Leslie–Gower predation models with multiplicative Allee effect and prey-dependent functional responses. While more complicated dynamics is known to occur in explicit predation models with Allee effect, our findings emerge as a first step in the detection and extension of the local basin boundaries of the origin as global separatrices in the phase plane; such study should also treat the interaction of these separatrices with the (un)stable manifolds of other equilibria, possibly with the aid of numerical methods for the computation of global invariant manifolds [1]. This investigation would identify the Allee thresholds as global objects in the phase plane that both species have to overcome in order to avoid extinction. Another important question is related to the number of limit cycles that a concrete model may have. For instance, the analytical criteria found in this work can be used as starting data in standard continuation packages in order to trace curves of Hopf bifurcation that contain higher codimension points, from which one can find the existence of additional limit cycles. Similar procedures can be used to explore loci of homoclinic and heteroclinic bifurcations.

Our results also serve the purpose of highlighting the mathematical methods used in this paper as helpful tools for analysing the qualitative dynamics of any two-dimensional model of species interaction, regardless of the specific functional properties of the vector field. In this way, we are also currently studying a similar class of general predation models where the prey equation is modelled with an additive Allee effect [3, 4]. Other future challenges involve, for instance, different ways of modelling the predation consumption rate, for instance, as a ratio-dependent functional response \(\phi =\phi (x,y)\) [2, 39, 47]. Another alternative of modelling the species interaction is to consider Gause-type models for the predator growth rate [24, 26, 29] in which it is assumed that a mass conservation principle acts on the prey consumption by the predator. Furthermore, the Allee effect may also interact with random environmental conditions such as alien species invasions or other catastrophic events; see [5, 20] and the references therein. As a consequence, the amplitude of population fluctuations may increase and even drive a population to extinction.