Mathematical Study About a Predator–Prey Model with Anti-predator Behavior

Abstract

Species use their defense mechanisms to fight, kill or escape from predators. Each species shows its unique defense mechanism to avoid the predation. In this paper, we propose and analyze a three species predator–prey system with group defense mechanism. Conditions for boundedness and positivity of equilibrium points have been discussed. We establish conditions for stability of positive equilibrium points and periodic solutions via Hopf-bifurcation. The predator–prey system is numerically studied with the help of phase portrait, time evolution and bifurcation diagrams. Lyapunov exponent and sensitivity analysis ensure the chaotic behaviour of the model system. Period doubling and period halving bifurcations show dynamical complexities of the food chain system. This study suggests that the repercussion of better group defense by the intermediate predator is able to produce chaos in food chain models.

Introduction

During the development of millions of years, animals have developed many ways to protect themselves against predators. Obviously, being able to escape from a predator is the choice of every prey. Some field studies of ecology demonstrate that predation rate is decreased due to the increased defense ability of the prey to better defend. There are some defense strategies adopted by prey to better defend themselves from predators. These include reproduction of toxic chemicals, group defense, camouflage, and mimicry. Group defense is common defense mechanism in social prey. Tener [1] observed that pairs of musk-oxen can be successfully attacked by wolves but groups are rarely attacked. Another example of defense by reproducing toxic chemical is studied by Holmes and Bethel [2]. There are many other examples of defense mechanism adopted by prey to deter predators [3,4,5,6].

The ecological models have a long history of deepening our understanding of the ecological world. Their simplicity makes some of the consequences of basic biological processes transparent, but at the same time they exhibit complex behaviors that surprise us. Population biology is perhaps the most mathematically developed areas of ecology with a long history of interest by mathematicians in the problems associated with the dynamics of populations. First Gause type predator–prey model was studied by Freedman [7] of the form

$$\begin{aligned} \left\{ \begin{aligned} x\prime&=xg(x,K)-yp(x),\\ y\prime&=y(-d+cp(x)), \end{aligned} \right. \end{aligned}$$

(1)

where g(x, K) is the growth function of prey in the absence of predation, and p(x) is known as a functional response of predators to prey. In population ecology, a functional response is the intake rate of a consumer as a function of food density. Holling [8] introduced three basic functional responses to model the interaction between predator and prey. Functional responses introduced by Holling in I-III do not have any component about defense mechanism and are thus useful for modeling the predator capturing of innocent prey. Andrews [9] suggested a non-monotonic functional response of the form

$$\begin{aligned} p(x)=\frac{mx}{ax^2+bx+c}. \end{aligned}$$

(2)

to model the impact of nutrients inhibitory effect on microorganisms. In general, this functional response resembles inhibitory effect of toxic phytoplankton acting on zooplankton population and known as Holling type IV functional response. In this functional response predator’s capture efficiency decrease at high prey density, which biologically shows the defense mechanism in prey [9,10,11,12]. Liznarova and Pekar [13] studied that Holling type IV functional response has a component related to prey defense. Sokol and Howell [14] proposed a modified Holling type IV functional response of the form

$$\begin{aligned} p(x)=\frac{mx}{ax^2+c}. \end{aligned}$$

(3)

This simplified Monod–Haldane functional response represents a negative predation impact on the predator’s density when prey are abundant that resembles anti-predator behavior of prey against predators.

After pioneer work of Lotka [15] and Volterra [16], huge number of articles have been published introducing different types of interaction like competition, mutual interference and defensive ability etc. [8, 17,18,19,20,21]. Among various predator–prey interaction studies, very less number of articles have been published focusing on defense in prey. Freedman [22] studied conditions for the survival and extinction in a predator–prey system with group defense. He observed that predator population extinct due to group defense of prey. Wolkowicz [23] studied a generalized Gauss type predator–prey system and observed that system undergoes for a sequence of bifurcations that includes a Hopf bifurcation as well as homoclinic bifurcation. Xiao and Zhu [24] studied stability and periodic oscillations via Hopf bifurcation in a non-monotonic predator–prey model exhibiting group defense. Predator–prey models of non-monotonic functional response are abundant in literature and are extensively studied by several scientists [11, 18, 23, 25, 26].

Freedman [22] proposed a three species food chain model with prey group defense and studied Hopf bifurcation in temporal as well as delay model. The phenomenon of the paradox of enrichment and persistence criteria of species are studied in a many predator–prey systems exhibiting group defense [22, 27, 28]. Li [29] studied generalized group defense ability and impulsive control strategy in a predator–prey system. Pei and De’Aguiar [20] found complex dynamics in a three species food chain system with anti-predator behaviour of prey against predator. Raw et al. [30] proposed a three species predator–prey model in which two species with group defense participate in the food chain. Recently, Mishra et al. [31] proposed a diffusive food chain model with anti-predator behavior and studied temporal as well as spatio-temporal complexity of the model.

Recent researches in predator–prey systems demonstrated that chaotic dynamics can arise in more than two species food chain models [17]. Klebanoff and Hastings [32] found chaotic dynamics in three species model. Upadhyay et al. [33] observed chaos in a damaged eco-epidemiological systems and suggested that to avoid chaos in such systems, mortality of infected prey should be brought down. Li et al. [34] proposed and analyzed a bioeconomic predator–prey model with nonlinear harvesting of prey. Several researchers observed complex dynamics in various ecological interaction models [21, 25, 35,36,37,38,39].

This article explores the complex dynamics of a predator–prey system where intermediate predator is equipped with defense ability. This model based study is organized in following manner: In “Formulation of Model System” section, we present the description of proposed model system (4). Boundedness and equilibrium analysis are presented in third section. In “Stability Analysis” section, we have discussed local as well as global stability of the system. Hopf bifurcation analysis is performed in fifth section. Numerical simulation is presented in sixth section. A brief discussion and conclusion are given in seventh section.

Formulation of Model System

This mathematical study is widely applicable is many real world situations. For example, plants, monkeys and jaguars are linked together in a food chain. Large African monkeys called baboons live in groups. Monkeys are primary consumers and they sometimes work together to drive off predators. Similarly, if the wolf invades the herd, musk makes a cycle with the calves in between the oxen and their horns get out. Keeping above real world situations in our mind, we consider that prey, intermediate and top predator participate in a food chain with the population densities P, O and W, respectively at any instant time t, where intermediate predator exhibits group defense. We consider following points to formulate the differential equations, which describes the predator–prey model:

($A_1$):

We assume logistic growth in prey with intrinsic growth rate $r_{1}$ and environmental carrying capacity $K_{1}$ in absence of intermediate predator O(t).

($A_2$):

Intermediate predator grows logistically with growth rate $r_{2}$ and environmental carrying capacity $K_{2}$. It is assumed that intermediate predator is herbivores and graze on grasses, reeds, sedges and other ground plants. Thus, intermediate predator O(t) consumes prey P(t) following Holling type II functional response.

($A_3$):

We assume that intermediate predator makes a group to deter their predator. Thus we incorporate simplified non-monotonic Monod–Haldane functional response for feeding of top predator upon intermediate predator. We also assume that intermediate predator is being harvested due to any reason with harvesting rate qE.

($A_4$):

We assume that top predator is a specialist predator and can not survive without the intermediate predator. Top predator dies out naturally with the constant death rate of d in the absence of food. Top predator increases in manner of Monod–Haldane functional response due to predation.

From above assumptions, we impose following nonlinear differential equations

$$\begin{aligned} \left\{ \begin{aligned} \frac{dP}{dt}&= r_{1}P\Big (1-\frac{P}{K_{1}}\Big )-\frac{w_{1}PO}{a+P},\\ \frac{dO}{dt}&= r_{2}O\Big (1-\frac{O}{K_{2}}\Big )+ \frac{w_{2}PO}{a+P} -\frac{\xi _{1}OW}{bO^2+c}-qEO,\\ \frac{dW}{dt}&= -dW+\frac{\xi _{2}OW}{bO^2+c}, \end{aligned} \right. \end{aligned}$$

(4)

with positive initial condition $P(0)>0,\ O(0)>0$ and $W(0)>0$. In model (4), b is defense efficiency of intermediate predator against top predator. q is the catchability coefficient of intermediate predator and E as harvesting effort. A brief description of variable and parameters of model system (4) is presented in Table 1.

Table 1 Description of parameters

In the next sections, we perform boundedness, existence of all possible positive equilibrium points, stability of equilibrium point and Hopf bifurcation. Numerical simulation is performed to discuss the theoretical results and applicability of our predator–prey model.

Boundedness and Equilibrium Analysis

In this section, we analyze the conditions under which system (4) is well behaved and also derived the existence conditions of biologically relevant equilibrium points. First, we begin with below theorem which states that system (4) is bounded for all time $t\ge 0$.

Theorem 1

All the solution of the predator–prey system (4) initiating in ${\mathbb {R}}_{+}^3$ is bounded for all $t\ge 0$.

Proof

From Eq. (4), we have

$$\begin{aligned} \lim _{t\rightarrow \infty } \text {sup} \ P(t) \le K_{1}. \end{aligned}$$

We define a function $S(t)= \displaystyle \frac{w_{2}}{w_{1}}P+O+\frac{\xi _{1}}{\xi _{2}}W$. It is easy to verify that

$$\begin{aligned} \frac{dS}{dt}+\eta S \le L, \end{aligned}$$

where $\displaystyle L= \frac{w_{2}}{w_{1}}(r_{1}+1)K_{1}+(1+r_{2})K_{2}$, $\displaystyle \eta =min(r_{1},\theta ,d)$ and $\theta =r_2-qE$. From comparison lemma [40], we obtain

$$\begin{aligned} S(t) \le \frac{L}{\eta }-\Big [\frac{L}{\eta }-S(0)\Big ]e^{-\eta (t-{\bar{T}})}. \end{aligned}$$

(5)

For all $0\le {\tilde{T}}\le t$. If ${\tilde{T}}=0$, then

$$\begin{aligned} S(t)&\le \frac{L}{\eta }-\Big [\frac{L}{\eta }-S(0)\Big ]e^{-\eta t}, \\ \Rightarrow S(t)&\le \frac{L}{\eta }, \ \text {as} \ t \rightarrow \infty . \end{aligned}$$

Thus, we have

$$\begin{aligned} \lim _{t\rightarrow \infty } \sup S(t)&= \frac{w_{2}}{w_{1}}P+O+\frac{\xi _{1}}{\xi _{2}} \ \le \frac{L}{\eta }. \end{aligned}$$

Thus the whole population is uniformly bounded for all $t>0$ in ${\mathbb {R}}_{+}^3$. $\square$

Now in order to examine existence of biologically feasible equilibrium points, we have

$$\begin{aligned} \left\{ \begin{aligned} H_1&=r_{1}\Big (1-\frac{P}{K_{1}}\Big )-\frac{w_{1}O}{a+P},\\ H_2&=r_{2}\Big (1-\frac{O}{K_{2}}\Big )+ \frac{w_{2}P}{a+P}-\frac{\xi _{1}W}{bO^2+c}-qE,\\ H_3&=-d+\frac{\xi _{2}O}{bO^2+c}, \end{aligned} \right. \end{aligned}$$

(6)

From above equation, we obtain following biologically feasible equilibrium points:

• (i) The extinction equilibria $E_{0}=(0,0,0)$ always exists.

• (ii) The predator-eradication equilibrium point $E_{1}=(K_{1},0,0)$ always exists.

• (iii) The intermediate predator survives in the absence of prey and top predator. Hence equilibria $\displaystyle E_{2}=(0,O_2,0)$ always exist where $\displaystyle O_2=\frac{K_{2}(r_2-qE)}{r_2}$. In absence of prey and intermediate predator, top predator can not survive. Thus equilibrium (0, 0, d) does not exist.

• (iv) Intermediate predator survives on its prey in the absence of top predator. Hence top predator free equilibrium point $E_{3}=({\bar{P}},{\bar{O}},0)$ exists where (${\bar{P}},{\bar{O}})$ are given as:

  $$\begin{aligned} {\bar{O}}=\frac{r_{1}}{w_{1}}\Big (\frac{K_1-{\bar{P}}}{K_{1}}\Big )(a+{\bar{P}}). \end{aligned}$$

  Substituting ${\bar{O}}$ in second equation of system (4) and performing some straight forward calculations, we get a polynomial $A{\bar{P}}^3+B{\bar{P}}^2+C{\bar{P}}+D=0$, where

  $$\begin{aligned} A&=\frac{r_{1}r_{2}}{w_1K_{1}K_2},\ B=\frac{r_{1}r_{2}}{w_1 K_2}\Big (\frac{2a}{K_1}-1\Big ),\ C=\frac{r_{1}r_{2}a^2}{w_1K_1K_{2}}- \frac{2r_{1}r_{2}}{w_1K_1}+w_{1}+r_2+qE,\\ D&=r_2 a\frac{r_{1}r_{2}a^2}{w_1K_1K_{2}}. \end{aligned}$$

  Here we have $A>0,\ D>0$, and if we choose $B<0,\ C>0$, then from Descartes’ rule of sign ensures least a positive root ${\bar{P}}$ for the above polynomial. Thus positive equilibrium $E_{3}=({\bar{P}},{\bar{O}},0)$ exists if the following conditions are satisfied.:

  $$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle 2a<K_1,\\ \displaystyle a^2>2K_2. \end{array}\right. } \end{aligned}$$

• (v) Intermediate predator free equilibrium does not exist because in the absence of intermediate predator, prey survives but specialist top predator can not survive the food chain.

• (vi) Due to availability of favorite food (i.e. middle predator), top predator also survives in the absence of prey. Hence the equilibria $E_{4}=(0,{\tilde{O}},{\tilde{W}})$ exists, where ${\tilde{O}}$ and ${\tilde{W}}$ are given as follows

  $$\begin{aligned} \left\{ \begin{aligned} {\tilde{O}}&=\frac{l_1\pm \sqrt{l_1^2-4l_2}}{2}, \\ {\tilde{W}}&=\frac{1}{\xi _{1}}\Big [qE-r_2\Big (1-\frac{{\tilde{O}}}{K_{2}}\Big )\Big ] (b{\tilde{O}}^2+c),\\ \end{aligned} \right. \end{aligned}$$

  (7)

  where $\displaystyle l_1=\frac{\xi _2}{bd},\ l_2=\frac{c}{b}$. Thus positive equilibrium $E_{4}=(0,{\tilde{O}},{\tilde{W}})$ exist if $l_1^2\ge 4l_2$ holds.

• (vii) Let the positive equilibrium point $E^\star =(P^\star ,O^\star ,W^\star )$ exists, then it must satisfy following:

  $$\begin{aligned} H_{1}&= r_{1}\Big (1-\frac{P^\star }{K_{1}}\Big )-\frac{w_{1}O^\star }{a+P^\star }=0, \end{aligned}$$

  (8)
  $$\begin{aligned} H_{2}&= r_{2}\Big (1-\frac{O^\star }{K_{2}}\Big )+ \frac{w_{2}P^\star }{a+P^\star } -\frac{\xi _{1}W^\star }{b{O^\star }^2+c}-qE=0, \end{aligned}$$

  (9)
  $$\begin{aligned} H_{3}&= -d+\frac{\xi _{2}O^\star }{b{O^\star }^2+c}=0. \end{aligned}$$

  (10)

  From Eqs. (9) and (10), we have

  $$\begin{aligned} O^\star&=\frac{l_1\pm \sqrt{l_1^2-4l_2}}{2}, \forall \ l_1^2 \ge 4l_2\ \text {where}\ l_1=\frac{\xi _{2}}{bd},\ l_2=\frac{c}{b} \end{aligned}$$

  (11)
  $$\begin{aligned} W^\star&=\frac{1}{\xi _{1}}\Bigg [qE-\frac{r_{2}P^\star }{a+P^\star }-r_{2} \Big (\frac{K_{2}-O^\star }{K_{2}}\Big )\Bigg ](b{O^\star }^2+c). \end{aligned}$$

  (12)

  Substituting value of $O^\star$ and $W^\star$ in Eq. (8), we get a quadratic polynomial $\displaystyle B_{1}{P^\star }^2+B_{2}{P^\star }+B_{3}=0$, where $B_{1}=r_1,\ B_{2}=r_{1}(a-K_1), B_{3}=K_1(w_{1}O^\star -r_{1}a)$. Therefore from Descartes’ rule of sign, a positive root $P^\star$ exists if $\displaystyle K_{1}>a$ and $\displaystyle O^\star \ge \frac{r_{1}a}{w_{1}}$ satisfy. Therefore positive equilibrium $E^\star =(P^\star ,O^\star ,W^\star )$ exists if conditions

  $$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle l_1^2 \ge 4 l_2, \ K_1>a ,\\ \displaystyle O^\star \le K_{1},\ O^\star \ge \frac{r_{1}a}{w_{1}}. \end{array}\right. } \end{aligned}$$

  (13)

  hold.

Stability Analysis

In order to derive local stability conditions around the positive equilibrium points of predator–prey system (4), we need to compute variational matrices at all the positive equilibrium points. The variational matrix around any equilibrium $E=(P,O,W)$ is calculated as

$$\begin{aligned} V(P, O, W)= \begin{pmatrix} \displaystyle P \frac{\partial H_{1}}{\partial P}+H_{1} &{} \quad \displaystyle P \frac{\partial H_{1}}{\partial O} &{}\quad \displaystyle P\frac{\partial H_{1}}{\partial W} \\ \displaystyle O\frac{\partial H_{2}}{\partial P} &{}\quad \displaystyle O\frac{\partial H_{2}}{\partial O}+H_{2} &{}\quad \displaystyle O\frac{\partial H_{2}}{\partial W} \\ \displaystyle W\frac{\partial H_{3}}{\partial P} &{}\quad \displaystyle W\frac{\partial H_{3}}{\partial O} &{}\quad \displaystyle W\frac{\partial H_{3}}{\partial W}+H_{3} \end{pmatrix}, \end{aligned}$$

(14)

where

$$\begin{aligned} \frac{\partial H_{1}}{\partial P}&=-\frac{r_1}{K_1}+\frac{w_{1}O}{(a+P)^2},\quad \frac{\partial H_{1}}{\partial O}=-\frac{w_{1}}{a+P},\quad \frac{\partial H_{1}}{\partial W}=0,\\ \frac{\partial H_{2}}{\partial P}&=\frac{w_{2}a}{(a+P)^2},\quad \frac{\partial H_{2}}{\partial O}=-\frac{r_{2}}{K_{2}} +\frac{2b\xi _{1}OW}{(bO^2+c)^2},\quad \frac{\partial H_{2}}{\partial W} =\frac{-\xi _{1}}{bO^2+c},\\ \frac{\partial H_{3}}{\partial P}&=0,\quad \frac{\partial H_{3}}{\partial O}=\frac{\xi _{2}(c-bO^2)}{(bO^2+c)^2},\ \frac{\partial H_{3}}{\partial W}=0. \end{aligned}$$

Therefore, we have calculated variational matrices at various positive equilibrium points.

• (i) The variational matrix at trivial equilibrium $E_{0}=(0,0,0)$ is

  $$\begin{aligned} V(E_0)= \begin{pmatrix} r_{1} &{} \quad 0 &{}\quad 0\\ 0 &{}\quad (r_{2}-qE) &{} \quad 0\\ 0 &{}\quad 0 &{} \quad -d \end{pmatrix}. \end{aligned}$$

  The eigenvalues of $V(E_0)$ are $r_1,\ r_2-qE$ and $-d$. Thus equilibrium is unstable manifold in PO-direction and stable manifold in W direction.

• (ii) Variational matrix at positive equilibrium $E_1=(K_1,0,0)$ is given by

  $$\begin{aligned} \ V(E_1)= \begin{pmatrix} \displaystyle -r_{1} &{}\quad \displaystyle -\frac{w_{1}K_{1}}{a+K_{1}} &{}\quad \displaystyle 0\\ \displaystyle 0 &{} \quad (r_{2}-qE)+\displaystyle \frac{w_{2}K_{1}}{a+K_{1}} &{} \quad 0\\ 0 &{} \quad 0 &{} \quad -d \end{pmatrix}. \end{aligned}$$

  The eigenvalues of $V(E_1)$ are $-r_1,\ (r_{2}-qE)+\displaystyle \frac{w_{2}K_{1}}{a+K_{1}}$ and $-d$. Equilibrium point $E_1$ is stable manifold in PW- direction and unstable manifold in O direction provided that $(r_{2}-qE)$ is positive. Hence equilibrium $E_1$ is always unstable.

• (iii) The variation matrix at equilibria $E_2=(0,O_2,0)$ is calculated as

  $$\begin{aligned} V(E_2)= \begin{pmatrix} \displaystyle r_{1}-\frac{w_1 O_2}{a} &{} \quad 0 &{} \quad 0\\ \displaystyle \frac{w_{2}a{O_2}}{a^2} &{}\quad \displaystyle -(r_{2}-qE) &{} \quad \displaystyle -\frac{\xi _{1} O_2}{b{O_2}^2+c}\\ 0 &{} \quad 0 &{}\quad \displaystyle -d+\frac{\xi _2 O_2}{b{O_2}^2+c} \end{pmatrix}. \end{aligned}$$

  Eigenvalues of $V(E_2)$ are $\displaystyle r_{1}-\frac{w_1 O_2}{a},\ -(r_{2}-qE)$ and $\displaystyle -d+\frac{\xi _2 O_2}{b{O_2}^2+c}$. Thus for the stability around $E_2$, all eigenvalues must be negative.

• (iv) Variational matrix around top predator free equilibria $E_3=({\bar{P}},{\bar{O}},0)$ is

  $$\begin{aligned} \ V(E_3)&= \begin{pmatrix} \displaystyle {\bar{P}}\Big [\frac{w_{1}{\bar{O}}}{(a+{\bar{P}})^2} -\frac{r_{1}}{K_{1}}\Big ] &{} \quad \displaystyle \displaystyle \frac{w_{1}{\bar{P}}}{(a+{\bar{P}})} &{} \quad 0\\ \displaystyle \frac{w_{2}a{\bar{O}}}{(a+{\bar{P}})^2} &{}\quad -(r_{2}-qE) &{}\quad \displaystyle -\frac{\xi _{1}{\bar{O}}}{b{{\bar{O}}}^2+c}\\ 0 &{} \quad 0 &{}\quad \displaystyle -d+\frac{\xi _2 {\bar{O}}}{b{{\bar{O}}}^2+c} \end{pmatrix}\\&=\begin{pmatrix} v_{11} &{} \quad v_{12} &{} \quad v_{13}\\ v_{21} &{} \quad v_{22} &{} \quad v_{23}\\ v_{31} &{} \quad v_{32} &{} \quad v_{33}. \end{pmatrix} \end{aligned}$$

  The characteristic equation of $V(E_3)$ is $\lambda ^3+\rho _1 \lambda ^2+\rho _2 \lambda +\rho _3=0$, where

  $$\begin{aligned} \rho _1&=-(v_{11}+v_{22}+v_{33}),\\ \rho _2&=v_{11}v_{22}+v_{11}v_{33}+v_{22}v_{22}-v_{12}v_{21},\\ \rho _3&=v_{33}(v_{12}v_{21}-v_{11}v_{22}). \end{aligned}$$

  Positive equilibrium $E_3$ is locally asymptotically stable if $\rho _1,\ \rho _2,\ \rho _3$ and $\rho _1\rho _2-\rho _3$ are positive. Thus one can easily verify that $E_3$ is locally asymptotically stable under the conditions $\displaystyle K_1w_{1}{\bar{O}}<r_{1}(a+{\bar{P}})^2$ and $\displaystyle \xi _2 {\bar{O}}<d(b{{\bar{O}}}^2+c)$ hold.

• (v) The variation matrix around the prey free equilibrium $E_4=(0,{\tilde{O}},{\tilde{W}})$ is

  $$\begin{aligned} \ V(E_4)= \begin{pmatrix} \displaystyle \big (r_{1}-\frac{w_{1}{\tilde{O}}}{a}\big ) &{} \quad 0 &{}\quad 0\\ \displaystyle \frac{w_{2}{\tilde{O}}}{a} &{}\quad \displaystyle {\tilde{O}} \Big [-\frac{r_{2}}{K_{2}}+\frac{2b\xi _{1}OW}{(bO^2+c)^2}\Big ] &{}\quad \displaystyle -\frac{\xi _1{\tilde{O}}}{(b{\tilde{O}}^2+c)^2}\\ 0 &{}\quad \displaystyle \frac{\xi _{2}{\tilde{W}}(c-b{{\tilde{O}}}^2)}{(b{\tilde{O}}^2+c)^2} &{} \quad 0 \end{pmatrix}, \end{aligned}$$

  If $\lambda _1,\ \lambda _2$ and $\lambda _3$ are eigenvalues of variational matrix $V(E_4)$, then we have

  $$\begin{aligned} \lambda _1&=\Big (r_{1}-\frac{w_{1}{\tilde{O}}}{a}\Big ),\\ \lambda _2+\lambda _3&={\tilde{O}}\Big [-\frac{r_{2}}{K_{2}} +\frac{2b\xi _{1}{\tilde{O}}{\tilde{W}}}{(bO^2+c)^2}\Big ],\\ \lambda _2 \lambda _3&= \frac{\xi _1 \xi _{2}(c-b{{\tilde{O}}}^2){\tilde{O}}{\tilde{W}} }{(b{\tilde{O}}^2+c)^3}. \end{aligned}$$

  If we choose $\displaystyle r_{1}<\frac{w_{1}{\tilde{O}}}{a}$, $\displaystyle {\tilde{O}}^2<\frac{c}{b}$ and $\displaystyle \frac{2b\xi _{1}{\tilde{O}}{\tilde{W}}}{(bO^2+c)^2}<\frac{r_{2}}{K_{2}}$, then it can easily verify that $V(E_4)$ has negative eigenvalues. Thus prey-free equilibrium point $E_4$ is locally asymptotically stable.

• (vi) The variational matrix of system (4) around coexistence equilibrium point $E^\star$ is given by $V(P^\star ,O^\star ,W^\star ) = (V_{ij}); i, j = 1, 2, 3$:

  $$\begin{aligned} V(E^\star )&= \begin{pmatrix} \displaystyle P^\star \Big [\frac{w_{1}O^\star }{(a+p^\star )^2} -\frac{r_{1}}{K_{1}}\Big ] &{}\quad \displaystyle -\frac{w_{1}P^\star }{(a+P^\star )} &{} \quad 0\\ \displaystyle \frac{w_{2}aO^\star }{(a+P^\star )^2} &{}\quad \displaystyle O^\star \Big [-\frac{r_{2}}{K_{2}} +\frac{2b\xi _{1}O^\star W^\star }{(b{O^\star }^2+c)^2}\Big ] &{}\quad \displaystyle -\frac{\xi _{1}O^\star }{b{O^\star }^2+c}\\ 0 &{}\quad \displaystyle \frac{\xi _{2}W^\star (c-b{O^\star }^2)}{(b{O^\star }^2+c)^2} &{} \quad 0 \end{pmatrix} \nonumber \\&=\begin{pmatrix} a_{11} &{} \quad a_{12} &{} \quad a_{13}\\ a_{21} &{} \quad a_{22} &{} \quad a_{23} \\ a_{31} &{} \quad a_{32} &{} \quad a_{33} \end{pmatrix}. \end{aligned}$$

  (15)

  The characteristic polynomial of variational matrix (15) can be written as

  $$\begin{aligned} \lambda ^3+B_{1}\lambda ^2+B_{2}\lambda +B_{3}=0, \end{aligned}$$

  (16)

  where

  $$\begin{aligned} B_{1}&=-(a_{11}+a_{22}),\\ B_{2}&=a_{11}a_{22}-a_{12}a_{21}-a_{23}a_{32},\\ B_{3}&=a_{11}a_{23}a_{32}. \end{aligned}$$

  If we choose $a_{11}<0,\ a_{22}<0$ and $a_{32}>0$, then one can easily verify $B_{1}>0,\ B_{2}>0,\ B_{3}>0$ and $\varDelta =B_{1}B_{2}-B_{3}>0$. Thus from Routh stability criterion, $E^\star$ is locally asymptotically stable. Above analysis is summarized by following theorem:

Theorem 2

• (i) Extinction equilibrium $E_0$ is always unstable.

• (ii) Predator-free equilibrium $E_1$ is saddle point.

• (iii) If $r_1a<w_1 O_2$ and $\xi _2 O_2<d({b{O_2}^2+c})$ hold, then $E_2$ is locally asymptotically stable.

• (iv) If $\displaystyle K_1w_{1}{\bar{O}}<r_{1}(a+{\bar{P}})^2$ and $\displaystyle \xi _2 {\bar{O}}<d(b{{\bar{O}}}^2+c)$ hold, then top predator-free equilibrium $E_3$ is locally asymptotically stable.

• (v) If $\displaystyle r_{1}a<w_{1}{\tilde{O}}$, $\displaystyle {\tilde{O}}^2<\frac{c}{b}$ and $\displaystyle \frac{2b\xi _{1}{\tilde{O}}{\tilde{W}}}{(bO^2+c)^2}<\frac{r_{2}}{K_{2}}$ hold, then prey-free equilibria $E_4$ is locally asymptotically stable.

• (vi) If following conditions

  $$\begin{aligned} K_{1}w_{1}O^\star&< r_{1}(a+p^\star )^2,\\ 2K_2b\xi _{1}O^\star W^\star&<r_2w_{2}(b{O^\star }^2+c)^2,\\ {O^\star }^2&<\frac{c}{b}, \end{aligned}$$

  hold, then $E^\star$ is locally asymptotically stable.

Next theorem explores global stability of system (4) around positive equilibrium $E^\star$. Global asymptotic stability of any dynamical system states that any perturbation in the initial condition does not affect stability of the system. Global stability of ecological system shows ability to face big disturbances without affecting the efficiency of a ecological unit.

Theorem 3

If the following conditions

$$\begin{aligned} K_{1}w_{1}O^\star&<r_{1}(a+P^\star )(a+P_c), \end{aligned}$$

(17)

$$\begin{aligned} K_{2}\xi _{1}(O_c+O^\star )W^\star&<r_{2}(b{{O}^\star }^2+c)(b{O_c}^2+c) , \end{aligned}$$

(18)

$$\begin{aligned} \xi _{2}&<d(b{O^\star }^2+c)(b{O_c}^2+c) \end{aligned}$$

(19)

hold, then the coexisting biological feasible equilibrium $E^\star$ of system (4) is globally asymptotically stable.

Proof

We consider a function

$$\begin{aligned} S&=\Big (P-P^\star -P^\star \log \frac{P}{P^\star }\Big )+Z_{1} \Big (O-O^\star -O^\star \log \frac{O}{O^\star }\Big )+Z_2\Big (W-W^\star \Big )^2 \end{aligned}$$

(20)

Time derivative of Eq. (20) and a little algebraic manipulation gives

$$\begin{aligned} \frac{dS}{dt}=&-\frac{1}{2} b_{11}(P-P^\star )^2+b_{12}(P-P^\star )(O-O^\star )-\frac{1}{2} b_{11}(P-P^\star )^2-\frac{1}{2}b_{22}(O-O^\star )^2\\&-\frac{1}{2}b_{22}(O-O^\star )^2-\frac{1}{2}b_{33}(W-W^\star )^2+b_{23}(O-O^\star )(W-W^\star )-\frac{1}{2}b_{33}(W-W^\star )^2. \end{aligned}$$

where

$$\begin{aligned} b_{11}&=\Big [\frac{r_{1}}{K_{1}}-\frac{w_{1}O^\star }{(a+P)(a+P^\star )}\Big ],\ b_{12}=\Big [\frac{Z_{1}aw_{2}}{(a+P)(a+P^\star )}-\frac{w_{1}}{(a+P)}\Big ],\\ b_{22}&=Z_{1}\Big [\frac{r_{2}}{K_{2}}-\frac{b\xi _{1}W^\star (O+O^\star )}{(bO^2+c)(b{O^\star }^2+c)}\Big ],\ b_{23}=\Big [\frac{Z_2\xi _{2}W^\star (c-bOO^\star )}{(bO^2+c)(b{O^\star }^2+c)}-\frac{Z_{1}\xi _{1}}{(bO^2+c)}\Big ],\\ b_{33}&=Z_2\Big [d-\frac{\xi _{2}(bO{O^\star }^2+c)}{(b{O^\star }^2+c)(bO^2+c))}\Big ]. \end{aligned}$$

For the global stability, we need to show $\frac{dS}{dt}<0$. Thus $\frac{dS}{dt}<0$, if following inequalities hold:

$$\begin{aligned} b_{11}&>0,\ b_{22}>0,\ b_{33}>0 \end{aligned}$$

(21)

$$\begin{aligned} b_{12}^2&<b_{11}b_{22}, \end{aligned}$$

(22)

$$\begin{aligned} b_{23}^2&<b_{22}b_{33}, \end{aligned}$$

(23)

$$\begin{aligned} b_{13}^2&<b_{11}b_{33} . \end{aligned}$$

(24)

It can be easily verify that under the conditions (21–24), $\frac{dS}{dt}<0$ is negative definite. Since $b_{13}=0$, condition (24) is automatically satisfied. Now it can be observed that under condition (17),the condition $b_{11}>0$ holds. Under conditions (18) and (19), $b_{22}>0$ and $b_{33}>0$, hold. If we choose $\displaystyle Z_{1}=\frac{(a+P^\star )w_1}{aw_{2}}$ and $\displaystyle Z_2= \frac{Z_1\xi _1(b{O^\star }^2+c)}{\xi _2 MW^\star }$, where $M=(c-bO_cO^\star )>0$, then it can be seen that inequalities (22) and (23) automatically hold. This shows that S is a positive definite function (i.e. Lyapunov function) to $E^\star$, which completes the proof. $\square$

In next section, we study Hopf bifurcation for parameter b.

Hopf-Bifurcation Analysis

It is well known that defensive efficiency has large effect on the predation rate of predators. Thus we consider defense efficiency of intermediate predator as bifurcation parameter. Hopf bifurcation is a critical point where a system switches its stability and a periodic solution arises. In next theorem, we derive conditions for the occurrence of Hopf bifurcation for the parameter b.

Theorem 4

The necessary and sufficient conditions for the Hopf bifurcation at $b=b^\star$ are stated as follows

• (i)
  $$\begin{aligned} B_{i}(b^\star )>0,\ i=1,2,3,\\ \end{aligned}$$
• (ii)
  $$\begin{aligned} B_{1}(b^\star )B_{2}(b^\star )=\ B_{3}(b^\star ),\\ \end{aligned}$$
• (iii)
  $$\begin{aligned} \left( \frac{d \varDelta }{db}\right) |_{b=b^\star } \ne 0, \end{aligned}$$

where $\varDelta =B_1B_2-B_3$ and is defined in Eq. (16).

Proof

For critical value $b=b^\star$, the characteristic equation (16) can be written as

$$\begin{aligned} (\lambda ^2+B_2)(\lambda +B_1)=0, \end{aligned}$$

which has three roots $\lambda _1=i\sqrt{B_2},\ \lambda _2=-i\sqrt{B_2}$ and $\lambda _3=-B_1$. Let for all b the roots are in general of the form

$$\begin{aligned} \lambda _1(b)&=\phi _1(b)+i\phi _2(b),\\ \lambda _2(b)&=\phi _1(b)-i\phi _2(b),\\ \lambda _3(b)&=-B_1(b). \end{aligned}$$

Now, we must verify the transversality condition

$$\begin{aligned} \left( \frac{d \varDelta }{db}\right) |_{b=b^\star } \ne 0,\ j=1,2. \end{aligned}$$

Substituting $\lambda _{j}(m)=\phi _1(b)+i\phi _1(b)$ into the characteristic equation. Some straight forward calculation gives

$$\begin{aligned} M(b)+\phi _{1}^{'}(b)-N(b)\phi _{2}^{'}(b)+U(b)=0,\\ N(b)+\phi _{1}^{'}(b)+M(b)\phi _{2}^{'}(b)+V(b)=0, \end{aligned}$$

where

$$\begin{aligned} M(b)&=3\phi _1^2(b)+2B_1(b)\phi _1(b)+B_2(b)-2\phi _2^2(b),\\ N(b)&=6\phi _1(b)\phi _2(b)+2B_1(b)\phi _2(b),\\ U(b)&=\phi _1^2(b)B_1^{'}(b)+B_2^{'}(b)\phi _1(b)+B_3^{'}(b)-B_1^{'}(b)\phi _2^2(b),\\ V(b)&=2\phi _1(b) \phi _2(b)B_1^{'}(b)+B_2^{'}(b)\phi _2(b). \end{aligned}$$

Notice that $\phi _1(b^\star )=0,\ \phi _2(b^\star )=\sqrt{B_2(b^\star )}$, we have

$$\begin{aligned} M(b^\star )&=-2B_2(b^\star ),\ N=2B_1(b^\star ) \sqrt{B_2(b^\star )},\\ U(b^\star )&=B_3^{'}(b^\star )-B_1^{'}(b^\star )B_2(b^\star ),\ V(b^\star )=B_2^{'}(b^\star )\sqrt{B_2(b^\star )}. \end{aligned}$$

Now transversality condition

$$\begin{aligned} \left( \frac{d \varDelta }{db}\right) |_{b=b^\star }&=\frac{M(b^\star )V(b^\star )+N(b^\star )U(b^\star )}{M(b^\star )^2+N(b^\star )^2},\\&= \frac{B_1(b^\star )B_2^{'}(b^\star )-B_3^{'}(b)+B_1^{'}(b^\star )B_2(b^\star )}{2(B_2(b^\star )+B_1^2(b^\star ))},\\&\ne 0,\ \text {if}\ B_1(b^\star )B_2^{'}(b^\star )-B_3^{'}(b)+B_1^{'}(b^\star )B_2(b^\star ) \ne 0. \end{aligned}$$

Here, the transversality condition holds. Thus Hopf-bifurcation occurs at $b=b^\star$. $\square$

Numerical Simulation

In this section, long term dynamics of the model system (4) is investigated numerically. Numerical simulation is performed to examine the complex behavior of proposed predator–prey system (4). To investigate the deterministic behavior, we present phase portraits, time evolution and The bifurcation diagrams. Lyapunov exponents are calculated with the help of MATLAB 2010a to justify the dynamics of predator–prey system. We assume biologically feasible parameter values that satisfy the theoretical results. On the basis of theoretical analysis, we fix following set of parameter values :

$$\begin{aligned} {\left\{ \begin{array}{ll} r_{1}=0.5, \ r_{2}=0.9,\ w_{1}=0.8,\, w_{2}=0.25,\ \xi _{1}=0.7,\xi _{2}=0.2, \\ qE=0.04, \ a=8,\ c=7,\ b=0.025,\ K_{1}=40, K_{2}=30,\ d=0.11. \end{array}\right. } \end{aligned}$$

(25)

It is observe that for above given parameter values conditions defined in Eq. (13) satisfy since $l_1^2-4l_2=6864.5>0$. In Fig. 1, we observe a unique interior equilibrium $E^\star$ for $b=0.02$. As we increase value of parameter b, the interior equilibrium loses its stability and stable limit cycle is observed at $b=0.025$. 3D view of limit cycle around $E^\star$ and effect of time upon population densities is shown in Fig. 2a, b. Lyapunov spectrum at $b=0.025$ is shown in Fig. 2c. Three Lyapunov exponent are $-0.00042341, -0.011955$ and $-0.36929$. It is observed that one Lyapunov exponent approaches to zero as time increase and the rest two are negative, which ensures periodic behaviour of the system (4) around $E^\star$.

Fig. 1

a Phase portrait of stable focus around $E^\star$, b population time series, c Lyapunov spectrum. For these, parameter $b=0.02$ and others are defined in Eq. (25)

Fig. 2

a Phase portrait of stable limit cycle around, b population time series, c Lyapunov spectrum. For these, parameter $b=0.025$ and others are defined in Eq. (25)

Fig. 3

a 3D view of chaotic attractor, b effect of time on population density, c Lyapunov spectrum. For these, parameter $b=0.03$ and others are defined in Eq. (25)

Fig. 4

2D view of Hopf bifurcation for different values of b at a$b=0.023$, b$b=0.024$ and other parameters are defined in Eq. (25)

Fig. 5

Sensitivity analysis for different initial conditions a chaotic face at $b=0.030$, b chaotic face at $K_{2}=32$ and all other parameters are defined in Eq. (25)

Fig. 6

2D view of disappearance of chaos (chaos crises) and appearance of limit cycle for a$K_{2}=32.0$, b$K_{2}=32.5$ , c$K_{2}=33$, d$K_{2}=33.5$. For this, all other parameters are defined in Eq. (25)

Fig. 7

Bifurcation diagram for parameter $K_1$ versus a maximum of P population, b maximum of O population, c maximum of W population. For this, all parameters are defined in Eq. (25)

Fig. 8

Bifurcation diagram for parameter b versus a maximum of P population, b maximum of O population, c maximum of W population, d maximum Lyapunov exponent. For this, all parameters are defined in Eq. (25)

Fig. 9

Bifurcation diagram for parameter $K_{2}$ versus a maximum of O population, b maximum of W population. For this, all parameters are defined in Eq. (25)

Fig. 10

Bifurcation diagram for parameter qE versus a maximum of O population, b maximum of W population. For this, all parameters are defined in Eq. (25)

Fig. 11

Bifurcation diagram for parameter $r_{2}$ versus a maximum of O population, b maximum of W population. For this, all parameters are defined in Eq. (25)

A chaotic attractor and effect of time on population are shown in Fig. 3a, b, respectively. For this $b=0.03$, and other parameters are defined in Eq. (25). The Lyapunov spectrum is shown in Fig. 3c for the chaotic attractor and values of Lyapunov exponents are 0.12681, $-0.044119$ and $-0.11666$. One can easily see that one exponent is positive and two are negative. Thus the interior equilibrium shows chaotic oscillation for some higher value of b.

In Fig. 4, we observe Hopf bifurcation when defense efficiency b crosses its critical value $b^\star =0.023$. The coexisting equilibrium $E^\star$ is locally stable for $b<0.023$. At $b=0.023$, interior equilibrium is stable but system loses its stability at $b=0.024$ and periodic oscillation arises. 2D view of Hopf bifurcation is shown for $b=0.024$ in Fig. 4. Now the conditions of the Hopf-bifurcation derived in Theorem 4 at bifurcation threshold $b^\star =0.024$, and all other parameters are same as given in Eq. (25) are satisfied since $B_1(b^\star )=0.4217>0$, $B_2(b^\star )=0.1055>0$, $B_2(b^\star )=0.0445>0$, $\varDelta =B_1(b^\star )B_2(b^\star )-B_3(b^\star )=0$ with eigenvalue $\lambda _{1,2}=-0.1251 \pm 0.2998i,\ \lambda _3=-0.04217$ and transversality condition $\displaystyle \left( \frac{d \varDelta }{db}\right) |_{b=b^\star }=-2.40342<0$ holds. Hence simple Hopf-bifurcation occurs at $b=0.024$.

In Fig. 5, $F_{1}$ shows the solution of system (4) for initial condition (P(0), O(0), W(0)). $F_{2}$ shows the solution for initial condition $(P(0),O(0),W(0)+1e{-}008)$ for chaotic face and ($log=F_{1}-F_{2}$) is difference of both solutions. Figure 5a is plotted for the chaotic phase when $b=0.030$. In Fig. 5b, sensitivity analysis is numerically performed at another chaotic phase $K_{2}=32$. It is observed that a minor change in the initial condition shows drastic change in the behaviour of system (4), which ensures sensitivity dependence on its initial condition which verifies chaotic dynamics of model.

Figure 6 shows the chaos crises phenomenon in dynamical predator–prey system. In Fig. 6a, chaotic attractor is observed at $K_2=32$, all parameter values are same as given in Eq. (25). As we increase the value of $K_2$, we see disappearance of chaos and limit cycles with different periods appear (see Fig. 6c, d). In dynamical system, This phenomenon is known as chaos crises.

Bifurcation diagrams are generated for different control parameters in Figs. 7, 8, 9, 10 and 11. We have generated bifurcation diagrams to study how parameters affect the dynamics of the system. (4). In Fig. 7, the bifurcation diagram is plotted the parameter $K_1$. Prey, intermediate and top predator population is observed in [0.0, 55.0], [0.0, 25.0] and [2.0, 18.0], respectively, as the function of carrying capacity in the range $20.0<K_{1}<50.0$ of prey population. We observe that population density of prey and top predator is increasing rapidly for $K_1<34.0$ and density fluctuations are observed for $K_1>34.0$. A weak negative impact of parameter $K_1$ is observed on the density of middle predator. We also observe that the interior equilibrium becomes unstable through a period doubling cascade as carrying capacity passes some threshold value $K_1=34.0$.

In Fig. 8a–c, bifurcation diagrams are plotted for control parameter b. The population densities of prey, intermediate predator and top predator are observed in the range [0.0, 40.0], [0.0, 28.0] and [0.0, 16.0], respectively. Bifurcation diagrams for control parameter defense efficiency (b) are generated in the range $0.01\le b \le 0.05$. We have generated Lyapunov exponent bifurcation diagram to detect chaotic range for the parameter b. In Fig. 8d, effect of parameter b on the Lyapunov exponent is presented by Lyapunov exponent bifurcation diagram. In this bifurcation diagram, the range of the maximum Lyapunov exponent is observed in the range [$-0.02$, 0.2] as the function of $0.02<b<0.04$. We observe that density of prey (P) decreases for $b<0.024$ and fluctuation is observed in the range $b\in (0.024,\ 0.033)$. These fluctuations ensure existence of limit cycles in predator–prey models and represent unstable nature of equilibrium point. Thus defense efficiency has negative impact on the stability of the system. We observe extinction of prey species for $b>0.033$ (see Fig. 8a). From Fig.8b, c, positive impact of parameter b is observed on the density of predators for $b<0.024$. We observe fluctuation on the density of predators when $b>0.024$. In Fig. 8d, we observe that the maximum Lyapunov exponent is negative when $b<0.023$. One can see a positive Lyapunov exponent for some larger value of $b>0.027$. It is well known that positive Lyapunov exponent shows the chaotic behaviour of predator–prey system around $E^\star$.

We present some more bifurcation diagrams which have a great impact on the stability of the system. We have generated bifurcation diagrams for parameters including carrying capacity, growth rate and harvesting rate of middle predator. The dynamics generated by accumulation of carrying capacity ($K_{2}$) of intermediate predator is shown in Fig. 9. Bifurcation diagrams are generated for intermediate and top predator in the range $(0.0,\ 30.0)$ and $(0.0,\ 0.18)$, respectively, as the function of $K_{2} \in (25.0,35.0)$. It is observe that $K_2$ has positive effect on the density of both the predators when $K_2<29.0$. Fluctuating population is observed for $K_2>29.0$, which ensures limit cycle behavior for the given parameter space. Thus increased carrying capacity has negative effect on the stability of the system. It is able to destabilize the stable equilibrium through a period doubling cascade.

In Fig. 10, we present bifurcation diagrams as the function of harvesting rate in the range $0.03<qE< 0.10$. From Fig. 10a, b, we observe that due to harvesting, the populations of intermediate and top predator population oscillate in the ranges of (0.0, 0.20) and (5.0, 16.0), respectively. We observe that populations of middle and top predator are fluctuating in the low harvesting range. A strong negative effect can be seen on the density of top predator for some threshold harvesting of intermediate predator. Here it is also very important to note that the unstable interior equilibrium becomes stable through a period halving cascade. In Fig. 11, bifurcation diagrams for growth rate $r_{2}$ are generated in the range $0.5\le r_{2}\le 1.2$. For $r_2<1.0$, we see that population densities of intermediate and top predator are fluctuating in the range (0.0, 25.0) and (0.0, 16.0), respectively. Hence the coexisting equilibrium of system is unstable for $r_2<1.0$ and becomes stable through a period halving cascade. Thus $r_2$ has positive impact on the dynamics of the proposed model.

A keen observation on the bifurcation diagrams presented in the Figs. 7, 8, 9, 10 and 11 states that model parameters have great impact in the dynamics of the system. There are some parameters that stabilizes the system but also parameters like growth rates, defense efficiency and carrying capacity destabilize the stable interior equilibrium of proposed model. Bifurcation diagrams also give an idea about route to chaos mechanism in predator–prey models. Period doubling route to chaos is observed in Figs. 7, 8 and 9 and period halving cascades are observed in Figs. 10 and 11.

Discussion and Conclusion

In recent studies, the paradox of enrichment is observed due to induced group defense of prey species [22, 23]. But the impact of defense behaviour of intermediate predator on the population dynamics is still absent in the literature. Recently, Mishra et. al. [31] proposed a mathematical model to study the defense mechanism between prey and predators. They assumed that group defense is social behavior of prey and usually found in the most of prey species. The key novelty of this model is the inclusion of innocent prey in food chain system. In this food chain model, we assume that middle predator is a generalist type predator and is also able to defend themselves from predators. It is assumed that middle predator is depredating on innocent prey following Holling type II rate and top predator consumes middle predator following simplified Monod–Haldane functional response due to group defense of middle predator. Condition for boundedness and existence of all biological feasible equilibrium points have been established. We have performed local and global stability analysis around positive interior equilibrium $E^\star$. We observe that predator–prey system exhibits rich dynamics, including stable focus, limit cycles of different periods which are changing in chaotic nature. From the Hopf bifurcation analysis, we see that stable interior equilibrium losses its stability and periodic oscillations arise when defense efficiency b crosses its critical value $b^\star =0.023$ (see Fig. 4a, b). Sensitivity analysis shows that a small change in initial condition has a drastic change in the dynamics of the system which ensures that predator–prey system has the chaotic nature for some certain parameter values. We found crisis-limited chaotic dynamics in a simple three species predator–prey model with group defense.

Bifurcation diagrams are plotted in certain range of model parameters like growth rate, carrying capacity, defensive efficiency etc. to study dynamical behavior of system. Period halving and period doubling bifurcation diagram route to chaos are observed, which shows rich dynamics of predator–prey model (4). We see a period doubling cascade for control parameter ($K_1)$ in Fig. 7. Interior equilibrium $E^\star$ is locally asymptotically stable for $K_1<34.0$. A simple Hopf-bifurcation take place when $K_1$ passes its threshold $K_1=34.0$, and $E^\star$ equilibrium point becomes unstable, and a stable limit cycle occurs for $K_1>34.0$. We observe that high carrying capacity of prey has negative effect on the stability of system and also positive effect is observed in the density of prey and top predator population.

Bifurcation diagrams as a function of defense efficiency (b) are shown in Fig. 8a–d. A period doubling bifurcation route to chaos is observed for control parameter b in the range $0.01\le b\le 0.05$. The coexisting equilibrium remains stable for $b<0.024$. A simple Hopf bifurcation occurs at $b=0.024$ and $E^\star$ becomes unstable for $b>0.024$. From Fig. 8d, we observe that system shows chaotic behavior when defense efficiency (b) is high. Interior equilibrium shows chaotic oscillations for $b>0.027$. Thus higher value of defense efficiency of intermediate predator produces chaos in the system. We also observe that better defense has a strong negative effect on the density of prey. Prey population extinct when defense efficiency of intermediate predator is very high (see Fig. 8a). Biologically, this result shows that better group defense decreases the predation rate of top predator, and as resultant intermediate predator grows rapidly. Thus, increased population of intermediate predator consumes more prey, which may be a reason for the extinction of prey species.

In Fig. 10, a period halving cascade is observed as the function of harvesting rate qE of intermediate predator, in the range $0.03\le qE \le 0.1$. Predator–prey system around the positive equilibrium is unstable for $qE <0. 065$. The stabilizing effect of harvesting rate is observed on the dynamics of the system. The interior equilibrium becomes stable for $qE>0.065$ and a subcritical Hopf bifurcation is observed at $qE=0. 065$. These bifurcation diagrams also show that qE has a negative effect on the population density of both intermediate and top predator (see. Fig. 10a, b). The population density of both the predators rapidly decreases as harvesting rate increases.

We observe that a period doubling cascade route to chaos is generated for the control parameter $(K_ {2})$. Accumulating values of carrying capacity $(K_ {2})$ of intermediate predator destabilize the interior equilibrium and chaotic oscillation arises. Interior equilibrium exhibits a variety of attractors including stable, periodic and chaotic attractor for carrying capacity of intermediate predator. The coexisting equilibrium is locally stable for $K_2<29.7$. A sudden destruction of stable equilibrium is observed at $K_2=29.7$ (see Fig. 9). A period halving cascade is shown Fig. 11 for control parameter $r_ {2}$. For $r_ {2}=1.0$, the interior equilibrium is unstable and as $r_ {2}$ crosses its critical value $r_ {2} =1.0$, a simple bifurcation occurs and interior equilibrium becomes stable. We observe that the population of top predator increases as the growth rate of intermediate predator increases.

We conclude that group defense plays a key role in the dynamics of prey–predator system. Better group defense (i.e. high defense efficiency) can destabilize tri-trophic food chain models and is also able to produce chaos in predator–prey system. The main outcome of this study reveals that prey species suffers with abolition due to increased defense ability of middle predator which is not studied earlier. In recent work by Raw et al. [30], authors considered group defense in first two population of three species food chain model and they observed extinction of top predator due to group defense. But in the present study, we assume only group defense in generalist type middle predator and observed that extinction of prey species is possible due to group defense of middle predator. This study also reveals that unlimited growth of middle predator at one or more trophic levels with huge consumption of resources by its consumers, leads to system irregularity, which in turn, demonstrates chaos for a wide range of parameter values. Finally, this study concludes that parameters such as the growth of intermediate predator and harvesting can disrupt the destabilization effect of the parameters on the stability of the predator–prey systems.