1 Introduction

Since the groundbreaking works of Lotka [1] and Volterra [2], the predator–prey model is used to describe the dynamical interaction between two species and has been widely researched by many scholars in the fields of biology and mathematics [3, 4]. It is well known that there are many functional response functions which are used to describe the interactional effect between the prey and predator species. For instance, the Holling I–III types [5, 6], ratio-dependent type [7, 8], Beddington–DeAngelis type [9], the different types with Allee effect [10,11,12] and so on.

Notice the fact that in natural ecosystem, many species may gather together and form herds to search for food resources or to defend the predators, which means that all members of a group do not interact at a time. This behaviour is often called as the herd behaviour. Recently, the authors in [13] have proposed a more realistic predator–prey model to describe this behaviour which can be written as

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\mathrm {d} u}{\mathrm {d} t}=u(1-u)-\sqrt{u} v, \\&\frac{\mathrm {d} v}{\mathrm {d} t}={\widetilde{\gamma }} v(-{\widetilde{\beta }}+\sqrt{u}), \end{aligned}\right. \end{aligned}$$
(1)

where u(t) and v(t) stand for the prey and predator densities at time t, respectively. Furthermore, \({\widetilde{\beta }} {\widetilde{\gamma }}\) is the death rate of the predator in the absence of prey, and \({\widetilde{\gamma }}\) is the conversion or consumption rate of prey to predator. The basic idea is that the prey population often gather together in huge herds with the strongest individuals on the border and the weakest will stay in the middle region of the enclosed region. Therefore, the prey which gathers together in the boundary region will be hunted by the predator. That is to say that, the prey population that occupies the outermost positions in the herds will be hunted by the predator.

In recent years, many predator–prey models with herd behaviour without diffusion term have been studied. Saha et al. [14] have studied a predator–prey model with herd behaviour and disease in prey incorporating prey refuge, which is an eco-epidemiological model of an infected predator–prey system. They assume that the susceptible prey shows the herd behaviour. The conditions for which the equilibrium point changes their stability and also the conditions for occurring Hopf bifurcation have been analysed. Manna et al. [15] have studied the dynamics of a predator–prey model with Allee effect and herd behaviour, where both predator and prey show herd behaviour. Here, the authors used the linear functional response function. A steady-state analysis has been performed, and some conditions for Hopf bifurcation are derived. Maiti et al. [16] have studied the dynamical behaviours of a predator–prey system with herd behaviour and the Holling type II functional response function, where both the predator and prey show herd behaviours. The positivity, boundedness and stability of equilibrium point are discussed.

From [17,18,19,20], we see that various reaction–diffusion predator–prey models have been extensively studied in the last decades. By considering the fact that in real natural environment, apart from the natural dispersive force of movement of an individual which is usually referred as the self-diffusion, there exists a mutual interference between individuals and is usually referred as the cross-diffusion, see [21,22,23] for details. Recently, cross-diffusion term has appeared in different fields, such as the population dynamics and ecology [24,25,26,27] and chemical reactions [28,29,30,31]. Furthermore, cross-diffusion is taken into consideration in predator–prey model which is used to measure the situation that the prey keeps away from the predator and conversely. Notice also that in recent years, many researchers have shifted from the study of the formation of stationary spatial patterns induced by Turing instability to the study of the formation of spatiotemporal patterns. For example, by combining with the model (1) and considering the self-diffusion and cross-diffusion, Tang et al. [32] have proposed a predator–prey model with herd behaviour and cross-diffusion, and they study the spatiotemporal dynamics near Turing–Hopf bifurcation point of the proposed model. Faria [33] computed the normal forms on centre manifolds (or other invariant manifolds) for partial functional differential equations (PFDEs) near the positive constant equilibrium point. Especially, when a Hopf bifurcation occurs, Faria [33] used the obtained normal forms to study the qualitative behaviours of solutions on those manifolds. In [34], Song et al. have proposed a rigorous procedure for calculating the normal form for the codimension-two Turing–Hopf bifurcation in the general reaction–diffusion system and to investigate the dynamical classification near the Turing–Hopf bifurcation point in detail.

Notice the fact that if the herd is too small, it may not be possible for the herd to form an appropriate group defence. In other words, if the herd is too small, the boundary of the herd may consist of the total number of the population. Thus, De Assis et al. [35] have proposed a modified predator–prey model with herd behaviour by considering large prey populations which display herd behaviour, and small prey populations which display individual behaviour. From a mathematical point of view, the interaction term \(a_{1}u v\) can be used to describe a small number of prey populations, and \(a_{2} \sqrt{u} v\) can be used to describe a large prey populations. Thus, the response function of their model is described by a piecewise function

$$\begin{aligned} g(u)=\left\{ \begin{aligned}&a_{1} u,~~~~~~ \text{ if } \quad 0 \le u \le h_{1}, \\&a_{2} \sqrt{u},~~~~ \text{ if } \quad u>h_{1}, \end{aligned}\right. \end{aligned}$$

where \(h_{1}\) represents a critical threshold of group size for effective defence. By considering that in real life, a smooth transition between ineffective and effective group defence is expected, at least for some species, and the function \(g^{\prime }(u)\) is discontinuous, the authors in [35] have proposed the following response function

$$\begin{aligned} f(u)=a \frac{u}{\sqrt{u+h_{2}}}, \end{aligned}$$

where a and \(h_{2}\) are parameters for which the biological interpretation will be explained after some calculations. When \(u \rightarrow 0\) and \(u \rightarrow \infty \), the authors in [35] have created a smooth transition function f(u) that approximates g(u) by letting g(u)/f(u) goes to one. Through simple calculations, the authors in [35] obtained that \(a=a_{2}\) and \(h_{2}=\left( a_{2}/a_{1}\right) ^{2}\). Furthermore, it follows that \(h_{1}=\left( a_{2}/a_{1}\right) ^{2}\) from the continuity of g(u), i.e. the critical threshold \(h_{1}\) of group size for effective defence and the threshold \(h_{2}\) is consistent. Thus, f(u) can be seen as an approximation of g(u) with a smooth transition for the individual behaviour \(a_{1} u\) and the group defence \(a_{2} \sqrt{u}\). For convenience, the authors in [35] let \(h_{1}=h_{2}={\widetilde{h}}\). Then, the functional response function of their proposed model can be written as

$$\begin{aligned} {\widetilde{f}}(u,v)=a \frac{u}{\sqrt{u+{\widetilde{h}}}}v. \end{aligned}$$

Here, by setting \(a_{1}=1\) and \(a_{2}=\sqrt{20}\), then \(h_{1}=h_{2}={\widetilde{h}}=20\), we plot the graphs of f(u) and g(u) in Fig. 1. The coordinate of point P is (20, 20).

Fig. 1
figure 1

The graphs of response functions f(u) and g(u) for \(a_{1}=1\), \(a_{2}=\sqrt{20}\) and \(h_{1}=h_{2}={\widetilde{h}}=20\)

Thus, based on the above facts, the authors in [35] have proposed a modified predator–prey model with herd behaviour described by the following ordinary differential equations

$$\begin{aligned} \left\{ \begin{aligned} \frac{\mathrm{d} u}{\mathrm{d} t}&=ru \left( 1-\frac{u}{K}\right) -a\frac{u}{\sqrt{u+{\widetilde{h}}}}v, \\ \frac{\mathrm{d} v}{\mathrm{d} t}&=-m v+ea\frac{u}{\sqrt{u+{\widetilde{h}}}}v, \end{aligned}\right. \end{aligned}$$
(2)

where r is the intrinsic growth rates of the prey, K is the carrying capacity of the prey, a is the maximum value of prey consumed by per predator per unit time, \({\widetilde{h}}\) is a threshold for the transition between herd grouping and solitary behaviour, m is the death rate of the predator in the absence of prey, and e is the conversion or consumption rate of prey to predator.

To the best of our knowledge, there are no results on spatiotemporal dynamics near Turing–Hopf bifurcation point of the model (2). Therefore, by assuming that the prey and predator populations are in an isolate patch and neglecting the impact of migration, including immigration and emigration, and introducing the spatial diffusion into the model (2), we consider the following modified predator–prey model with herd behaviour and cross-diffusion subject to homogeneous Neumann boundary condition for \(x \in \varOmega :=(0, \ell \pi )\) with \(\ell \in \mathbb {R}^{+}\). That is

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\partial u}{\partial t}=d_{11} \varDelta u+d_{12} \varDelta v+ru\left( 1-\frac{u}{K}\right) -a\frac{u}{\sqrt{u+{\widetilde{h}}}}v, \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x \in \varOmega ,~t>0, \\&\frac{\partial v}{\partial t}=d_{21} \varDelta u+d_{22} \varDelta v-m v+ea\frac{u}{\sqrt{u+{\widetilde{h}}}}v, \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x \in \varOmega ,~t>0, \\&\frac{\partial u}{\partial {\mathbf {n}}}=\frac{\partial v}{\partial {\mathbf {n}}}=0, \quad ~~~~~~~~~~~~~~~~~~~~~~~~x \in \partial \varOmega ,~t>0, \\&u(x, 0)=\phi (x) \ge 0,~v(x, 0)=\psi (x) \ge 0, \quad ~~~x \in \varOmega , \end{aligned}\right. \end{aligned}$$
(3)

where u(xt) and v(xt) describe the prey and predator densities at the spatial location x and at time t, respectively, the nonnegative constants \(d_{11}\) and \(d_{22}\) are the self-diffusion coefficients of the prey and the predator, respectively, and the nonnegative constants \(d_{12}, d_{21}\) are the cross-diffusion coefficients, which describe the respective population fluxes of preys and predators resulting from the presence of the other species, respectively, \(\varDelta u=\partial ^{2}u/\partial x^{2}\), \(\varDelta v=\partial ^{2}v/\partial x^{2}\), and \({\mathbf {n}}\) is the outward unit normal vector at the smooth boundary \(\partial \varOmega \). Moreover, \(\phi (x)\) and \(\psi (x)\) are the initial functions. Furthermore, it is necessary to assume that \(d_{11} d_{22}>d_{12} d_{21}\), which indicates that the flux of the respective densities in the spatial domain depends more strongly on their own density than on the other [36]. Especially, we point out that the condition \(d_{11} d_{22}>d_{12} d_{21}\) can also be obtained in Sect. 3 which is one of the necessary conditions for the occurrence of Turing instability for model (4).

In this paper, by calculating the normal form for the codimension-two Turing–Hopf bifurcation in the model (4), we investigate the dynamical classification near the Turing–Hopf bifurcation point. It is necessary to point out the fact that, although we use the method of computing normal form which presented in [34], since we choose \(d_{11}\) as the bifurcation parameter and the existence of cross-diffusion, the procedure of computing \(B_{11}\) and \(B_{13}\) needs to be deduced again, see “Appendix” for details.

The remaining part of this article is organized as follows. In Sect. 2, we show the non-negativity and boundedness of solutions corresponding to the temporal model and the spatiotemporal model, respectively. Furthermore, the existence and priori boundedness of solutions corresponding to the spatiotemporal model without cross-diffusion are also researched. Section 3 is devoted to the stability analysis of the proposed spatiotemporal model, including the stability analysis for the case without self-diffusion and cross-diffusion and the stability analysis for the case with self-diffusion and cross-diffusion. Furthermore, we plot the bifurcation diagram for model (4) in \(d_{11}-\delta \) plane which can be found in Sect. 3. In Sect. 4, we compute the normal form on the centre manifold for Turing–Hopf bifurcation corresponding to the model (4) by using the method in [34]. Some numerical simulations are given to support the theoretical results in Sect. 5. Finally, we conclude this paper in Sect. 6.

2 Non-negativity and boundedness of solutions corresponding to the temporal model

With a non-dimensionalized change of variables

$$\begin{aligned} u \rightarrow \frac{1}{K} u,~~v \rightarrow \frac{1}{e K} v,~~t\rightarrow m t, \end{aligned}$$

and let

$$\begin{aligned}\begin{aligned}&d_{11}=\frac{d_{11}}{m},~~d_{12}=\frac{d_{12}}{m},~~d_{21}=\frac{d_{21}}{m},~~d_{_{22}}=\frac{d_{22}}{m}, \\&\gamma =\frac{r}{m},~~\beta =\frac{a e \sqrt{K}}{m}, \end{aligned}\end{aligned}$$

then model (3) becomes the following non-dimensiona lized model

$$\begin{aligned} \left\{ \begin{array}{ll@{\qquad }l} &{}\dfrac{\partial u}{\partial t}=d_{11} \varDelta u+d_{12} \varDelta v+\gamma u(1-u)-\dfrac{\beta u v}{\sqrt{u+h}}, &{} x \in \varOmega ,~t>0, \\ &{}\dfrac{\partial v}{\partial t}=d_{21} \varDelta u+d_{22} \varDelta v-v+\dfrac{\beta u v}{\sqrt{u+h}}, &{} x \in \varOmega ,~t>0, \\ &{}\dfrac{\partial u}{\partial {\mathbf {n}}}=\dfrac{\partial v}{\partial {\mathbf {n}}}=0, &{} x \in \partial \varOmega ,~t>0, \\ &{}u(x, 0)=\phi (x) \ge 0,~v(x, 0)=\psi (x) \ge 0, &{} x \in \varOmega , \end{array}\right. \end{aligned}$$
(4)

where \(h={\widetilde{h}}/K\) represents the critical threshold of group size for effective defence in non-dimensional scaling. Since we assumed that there is an observable group defence effect, it is reasonable to take \(0<{\widetilde{h}}<K\), and hence \(0<h<1\).

The temporal model associated with the model (4) is

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\mathrm{d}u(t)}{\mathrm{d}t}=\gamma u(t)(1-u(t))-\frac{\beta u(t) v(t)}{\sqrt{u(t)+h}},&t> 0, \\&\frac{\mathrm{d}v(t)}{\mathrm{d}t}=-v(t)+\frac{\beta u(t) v(t)}{\sqrt{u(t)+h}},&t > 0, \\&u(0) \ge 0,~v(0) \ge 0. \end{aligned}\right. \end{aligned}$$
(5)

For the temporal model (5), we have the following results.

Theorem 1

Suppose that \(\gamma>0,~\beta >0,~0<h<1\) and consider the system given by (5) and its trajectories u(t), v(t); if the initial value \(u(0) \ge 0,~v(0) \ge 0\), then the solutions u(t) and v(t) are always non-negative. Furthermore, for any solution (u(t), v(t)) of system (5), we have

$$\begin{aligned} \lim _{t \rightarrow \infty }\mathrm{{sup}}~u(t) \le \frac{(\gamma +1)^{2}}{4\gamma },~~\lim _{t \rightarrow \infty }\mathrm{{sup}}~v(t) \le \frac{(\gamma +1)^{2}}{4\gamma }. \end{aligned}$$

Proof

In fact, the system (5) is composed of the following three subsystems

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\mathrm{d}u(t)}{\mathrm{d}t}=\gamma u(t)(1-u(t))-\frac{\beta u(t) v(t)}{\sqrt{u(t)+h}},&t> 0, \\&\frac{\mathrm{d}v(t)}{\mathrm{d}t}=-v(t)+\frac{\beta u(t) v(t)}{\sqrt{u(t)+h}},&t> 0, \\&u(0)>0,~v(0)>0, \end{aligned}\right. \end{aligned}$$
(6)
$$\begin{aligned} \left\{ \begin{aligned}&u(t)=0,&t> 0, \\&\frac{\mathrm{d} v(t)}{\mathrm{d} t}=-v(t),&t > 0, \\&u(0)=0,~v(0) \ge 0 \end{aligned}\right. \end{aligned}$$
(7)

and

$$\begin{aligned} \left\{ \begin{aligned}&\frac{\mathrm{d}u(t)}{\mathrm{d}t}=\gamma u(t)(1-u(t)),&t> 0, \\&v(t)=0,&t > 0, \\&u(0)\ge 0,~v(0)=0. \end{aligned}\right. \end{aligned}$$
(8)

Since it is easy to verify that the solution of (7) will approach the origin (0, 0) along the v-axis as \(t \rightarrow \infty \) and the solution of (refh8) will tend to (1, 0) along the u-axis as \(t \rightarrow \infty \), we only need to consider the solutions of (6). Assume \(X(t)=(u(t), v(t))\) is a solution of system (6). If it does not remain in the first quadrant, then the solution either hits the u-axis or the v-axis in finite time. Thus, by combining with the analysis results of the subsystems (7) and (refh8), we can obtain that any solution (u(t), v(t)) of system (6) with non-negative initial value (u(0), v(0)) will remain positive for all \(t>0\), or will approach either the original or (1, 0) along the axes.

Next, the boundedness of solutions is confirmed. Let \({\widetilde{w}}(t)=u(t)+v(t)\), then by adding

$$\begin{aligned} \frac{\mathrm{d}u(t)}{\mathrm{d}t}=\gamma u(t)(1-u(t))-\frac{\beta u(t) v(t)}{\sqrt{u(t)+h}} \end{aligned}$$

to

$$\begin{aligned} \frac{\mathrm{d}v(t)}{\mathrm{d}t}=-v(t)+\frac{\beta u(t) v(t)}{\sqrt{u(t)+h}}, \end{aligned}$$

we have

$$\begin{aligned} \frac{\mathrm{d}{\widetilde{w}}(t)}{\mathrm{d}t}&=\gamma u(t)(1-u(t))+u(t)-{\widetilde{w}}(t)\\&\le \frac{(\gamma +1)^{2}}{4\gamma }-{\widetilde{w}}(t) \end{aligned}$$

resulting in

$$\begin{aligned} \lim _{t \rightarrow \infty }\mathrm{{sup}}~{\widetilde{w}}(t) \le \frac{(\gamma +1)^{2}}{4\gamma }. \end{aligned}$$

Furthermore, notice that \({\widetilde{w}}(t)=u(t)+v(t)\), then we have

$$\begin{aligned} \lim _{t \rightarrow \infty }\mathrm{{sup}}~u(t)&\le \frac{(\gamma +1)^{2}}{4\gamma },~~\lim _{t \rightarrow \infty }\mathrm{{sup}}~v(t)\\&\le \frac{(\gamma +1)^{2}}{4\gamma }. \end{aligned}$$

This concludes the proof. \(\square \)

2.1 Non-negativity and boundedness of solutions corresponding to the spatiotemporal model

Theorem 2

Suppose that \(\gamma>0,~\beta >0,~0<h<1\) and \(\varOmega \subset \mathbb {R}\) is a bounded domain with smooth boundary. Then for any solution (u(xt), v(xt)) of model (4), we have

$$\begin{aligned}\begin{aligned}&\lim _{t \rightarrow \infty } \mathrm{{sup}} \int _{\varOmega } u(x, t) \mathrm{d} x \le \frac{(\gamma +1)^{2}}{4\gamma } |\varOmega |, \\&\lim _{t \rightarrow \infty } \mathrm{{sup}} \int _{\varOmega } v(x, t) \mathrm{d} x \le \frac{(\gamma +1)^{2}}{4\gamma } |\varOmega |, \end{aligned}\end{aligned}$$

where \(|\varOmega |\) is the length of the bounded domain \(\varOmega \).

Proof

Denote

$$\begin{aligned} M_{1}(t)=\int _{\varOmega } u(x, t) \mathrm{d}x, \quad M_{2}(t)=\int _{\varOmega } v(x, t) \mathrm{d}x, \end{aligned}$$

then by integrating on both sides of each equation on the region \(\varOmega \) in system (4), we have

$$\begin{aligned} \begin{aligned} \frac{\mathrm{d} M_{1}(t)}{\mathrm{d} t}&= \int _{\varOmega }\left( d_{11} \varDelta u+d_{12} \varDelta v\right) \mathrm{d} x \\&\quad +\int _{\varOmega }(\gamma u(1-u)-\frac{\beta u v}{\sqrt{u+h}})\mathrm{d} x, \\ \frac{\mathrm{d} M_{2}(t)}{\mathrm{d} t}&= \int _{\varOmega }\left( d_{21} \varDelta u+d_{22} \varDelta v\right) \mathrm{d} x \\&\quad +\int _{\varOmega }(-v+\frac{\beta u v}{\sqrt{u+h}})\mathrm{d} x. \end{aligned}\end{aligned}$$
(9)

Let \(M(t)=M_{1}(t)+M_{2}(t)\), then by combining with Eq. (9) and noticing the homogeneous Neumann boundary defined in system (4), we can obtain

$$\begin{aligned}\begin{aligned} \frac{\mathrm{d} M(t)}{\mathrm{d} t}&=\frac{\mathrm{d} M_{1}(t)}{\mathrm{d} t}+\frac{\mathrm{d} M_{2}(t)}{\mathrm{d} t} \\&=\int _{\varOmega }\left( \gamma u(1-u)-v\right) \mathrm{d} x \\&=\int _{\varOmega }\left( \gamma u(1-u)+u \right) \mathrm{d}x-M(t) \\&\le \frac{(\gamma +1)^{2}}{4\gamma }|\varOmega |-M(t), \end{aligned}\end{aligned}$$

from which we obtain that

$$\begin{aligned} \lim _{t \rightarrow \infty } \mathrm{{sup}}~M(t) \le \frac{(\gamma +1)^{2}}{4\gamma }|\varOmega |. \end{aligned}$$

Furthermore, notice that \(M(t)=M_{1}(t)+M_{2}(t)\), then we have

$$\begin{aligned}\begin{aligned}&\lim _{t \rightarrow \infty } \mathrm{{sup}} \int _{\varOmega } u(x, t) \mathrm{d} x \le \frac{(\gamma +1)^{2}}{4\gamma }|\varOmega |, \\&\lim _{t \rightarrow \infty } \mathrm{{sup}} \int _{\varOmega } v(x, t) \mathrm{d} x \le \frac{(\gamma +1)^{2}}{4\gamma }|\varOmega |. \end{aligned}\end{aligned}$$

This proof is completed. \(\square \)

2.2 The existence and priori boundedness of solutions corresponding to the spatiotemporal model without cross-diffusion

In this section, we give out a sufficient condition for the existence of a positive solution of system (4) without cross-diffusion. Meanwhile, we derive a priori boundedness of the solution. When \(d_{12}=d_{21}=0\), system (4) becomes

$$\begin{aligned} \left\{ \begin{array}{ll@{\qquad }l} &{}\dfrac{\partial u}{\partial t}=d_{11} \varDelta u+\gamma u(1-u)-\dfrac{\beta u v}{\sqrt{u+h}}, &{} x \in \varOmega ,~t>0, \\ &{}\dfrac{\partial v}{\partial t}=d_{22} \varDelta v-v+\dfrac{\beta u v}{\sqrt{u+h}}, &{} x \in \varOmega ,~t>0, \\ &{}\dfrac{\partial u}{\partial {\mathbf {n}}}=\dfrac{\partial v}{\partial {\mathbf {n}}}=0, &{} x \in \partial \varOmega ,~t>0, \\ &{} u(x, 0)=\phi (x) \ge 0,~v(x, 0)=\psi (x) \ge 0, &{} x \in \varOmega . \end{array}\right. \nonumber \\ \end{aligned}$$
(10)

Theorem 3

Suppose that \(\gamma>0,~\beta >0,~0<h<1\) and \(\varOmega \subset \mathbb {R}\) is a bounded domain with smooth boundary. Then we have the following results:

  1. (i)

    For \(\phi (x) \ge 0,~\psi (x) \ge 0\) and \(\phi (x)\not \equiv 0, \psi (x) \not \equiv 0\), system (10) has a unique solution \((u(x, t), v(x, t))\) satisfying \(0<u(x, t) \le u^{*}(t),~0<v(x, t) \le v^{*}(t)\) for \(t>0\) and \(x \in \varOmega \), where \(\left( u^{*}(t), v^{*}(t)\right) \) is the unique solution of

    $$\begin{aligned} \left\{ \begin{aligned}&\frac{\mathrm {d} u}{\mathrm {d} t}=\gamma u(1-u), \qquad \qquad t> 0, \\&\frac{\mathrm {d} v}{\mathrm {d} t}=-v+\frac{\beta u v}{\sqrt{u+h}}, \qquad \quad t > 0, \\&u(0)=\phi ^{*}=\sup _{x \in {\overline{\varOmega }}} \phi (x),~v(0)=\psi ^{*}=\sup _{x \in {\overline{\varOmega }}} \psi (x), \end{aligned}\right. \end{aligned}$$

    and \({\overline{\varOmega }}\) represents the closure of \(\varOmega \);

  2. (ii)

    For any solution (u(xt), v(xt)) of system (10), we have

    $$\begin{aligned} \lim _{t \rightarrow \infty } \mathrm{{sup}}~u(x, t)&\le 1, \quad \lim _{t \rightarrow \infty } \mathrm{{sup}}~\int _{\varOmega } v(x, t) \mathrm {d} x\\&\le (\gamma +1) |\varOmega |. \end{aligned}$$

Proof

Denote

$$\begin{aligned}&f(u, v)=\gamma u(1-u)-\frac{\beta u v}{\sqrt{u+h}}, \\&g(u, v)=-v+\frac{\beta u v}{\sqrt{u+h}}, \end{aligned}$$

then we have \(f_{v} \le 0\) and \(g_{u} \ge 0\) for \((u, v) \in \mathbb {R}_{+}^{2}=\{(u, v) \mid u \ge 0, v \ge 0\}\). Thus, the system (10) is a mixed quasi-monotone system.

Furthermore, if we assume that \(\left( u_{1}(x, t), v_{1}(x, t)\right) =(0,0)\) and \(\left( u_{2}(x, t), v_{2}(x, t)\right) =\left( u^{*}(t), v^{*}(t)\right) \), then we can obtain

$$\begin{aligned}\begin{aligned}&\frac{\partial u_{2}}{\partial t}-d_{11} \varDelta u_{2}-f\left( u_{2}, v_{1}\right) \\&\quad =0 \ge 0 \\&\quad =\frac{\partial u_{1}}{\partial t}-d_{11} \varDelta u_{1}-f\left( u_{1}, v_{2}\right) , \\&\frac{\partial v_{2}}{\partial t}-d_{22} \varDelta v_{2}-g\left( u_{2}, v_{2}\right) \\&\quad =0 \ge 0 \\&\quad =\frac{\partial v_{1}}{\partial t}-d_{11} \varDelta v_{1}-g\left( u_{1}, v_{1}\right) , \end{aligned}\end{aligned}$$

and \(0 \le \phi (x) \le \phi ^{*}\), \(0 \le \psi (x) \le \psi ^{*}\), so \(\left( u_{1}(x, t), v_{1}(x, t)\right) \) and \(\left( u_{2}(x, t), v_{2}(x, t)\right) \) are the lower and upper solutions of system (10), respectively.

According to Theorem 3.3 in Section 8.3 of Chapter 8 in [37], we know that (10) has a unique globally defined solution (u(xt), v(xt)) which satisfies

$$\begin{aligned} 0 \le u(x, t) \le u^{*}(t), \quad 0 \le v(x, t) \le v^{*}(t), \quad t>0. \end{aligned}$$

The strong maximum principle implies that \(u(x, t)>0, v(x, t)>0\) when \(t>0\), for \(x \in \varOmega \). This completes the proof of conclusion (i).

From the above discussion, we can obtain that \(0<u(x, t) \le u^{*}(t),~0<v(x, t) \le v^{*}(t)\) for all \(t>0\), and \(u^{*}(t)\) is the unique solution of

$$\begin{aligned} \frac{\mathrm {d} u}{\mathrm {d} t}=\gamma u(1-u), \quad u(0)=\phi ^{*}>0. \end{aligned}$$

One can see that \(u^{*}(t) \rightarrow 1\) as \(t \rightarrow \infty \). Thus, for any \(\varepsilon >0\), there exists \(t_{0}>0\) such that \(u(x, t) \le 1+\varepsilon \) for \(t>t_{0}\) and \(x \in {\overline{\varOmega }}\), which implies that \(\underset{t_{0} \rightarrow \infty }{\lim }\sup _{t>t_{0}}~u(x, t) \le 1\).

Furthermore, if let

$$\begin{aligned}\begin{aligned}&N_{1}(t)=\int _{\varOmega } u(x, t) \mathrm {d} x, \quad N_{2}(t)=\int _{\varOmega } v(x, t) \mathrm {d} x, \\&N(t)=N_{1}(t)+N_{2}(t), \end{aligned}\end{aligned}$$

then

$$\begin{aligned}\begin{aligned}&\frac{\mathrm {d} N_{1}(t)}{\mathrm {d} t}=\int _{\varOmega } d_{11} \varDelta u \mathrm {d} x+\int _{\varOmega }\left( \gamma u(1-u)-\frac{\beta u v}{\sqrt{u+h}}\right) \mathrm {d} x, \\&\frac{\mathrm {d} N_{2}(t)}{\mathrm {d} t}=\int _{\varOmega } d_{22} \varDelta v \mathrm {d} x+\int _{\varOmega }\left( -v+\frac{\beta u v}{\sqrt{u+h}}\right) \mathrm {d} x. \end{aligned}\end{aligned}$$

By using the homogeneous Neumann boundary condition, we can obtain

$$\begin{aligned}\begin{aligned} \frac{\mathrm {d} N(t)}{\mathrm {d} t}&=\frac{\mathrm {d} N_{1}(t)}{\mathrm {d} t}+\frac{\mathrm {d} N_{2}(t)}{\mathrm {d} t} \\&=-N_{2}(t)+\gamma \int _{\varOmega } u(1-u) \mathrm {d} x \\&=\gamma \int _{\varOmega } u(1-u) \mathrm {d} x-(N_{1}(t)+N_{2}(t))+N_{1}(t) \\&=\gamma \int _{\varOmega } u(1-u) \mathrm {d} x-(N_{1}(t)+N_{2}(t))+\int _{\varOmega } u(x, t) \mathrm {d} x \\&\le -N(t)+(\gamma +1)N_{1}(t). \end{aligned}\end{aligned}$$

From \(\lim _{t_{0} \rightarrow \infty } \sup _{t>t_{0}}~u(x, t) \le 1\), we can obtain that \(\lim _{t_{0} \rightarrow \infty }\sup _{t>t_{0}}~N_{1}(t) \le |\varOmega |\). Thus for small \(\varepsilon >0\), there exists \(t_{1} \ge 0\) such that

$$\begin{aligned} \frac{\mathrm {d} N(t)}{\mathrm {d} t} \le -N(t)+(\gamma +1)|\varOmega |, \quad t>t_{1}. \end{aligned}$$
(11)

It is well known that the solution N(t) of

$$\begin{aligned} \frac{\mathrm {d} N(t)}{\mathrm {d} t}=-N(t)+(\gamma +1)(1+\varepsilon )|\varOmega | \end{aligned}$$

satisfies

$$\begin{aligned} \lim _{t \rightarrow \infty } N(t)=(\gamma +1)(1+\varepsilon )|\varOmega |. \end{aligned}$$

In terms of comparison principle and using (11) we obtain that, for \(t_{2}>t_{1}\)

$$\begin{aligned} \int _{\varOmega } v(x, t) \mathrm {d} x=N_{2}(t)<N(t)\le (\gamma +1)(1+\varepsilon )|\varOmega |+\varepsilon ,~t>t_{2}, \end{aligned}$$

which implies that

$$\begin{aligned} \lim _{t_{2} \rightarrow \infty }\sup _{t>t_{2}}~\int _{\varOmega } v(x, t) \mathrm {d} x \le (\gamma +1)|\varOmega |. \end{aligned}$$

This completes the proof of conclusion (ii). \(\square \)

3 Stability analysis of the corresponding spatiotemporal model

The system (4) has two boundary equilibrium points (0, 0) and (1, 0). Moreover, when \(0<\delta <(1/\sqrt{h+1})\), the system (4) has a unique positive constant equilibrium point \(E_{*}\left( u_{*}, v_{*}\right) \) with

$$\begin{aligned} u_{*}=\frac{\delta }{2}\left( \delta +\sqrt{\delta ^{2}+4 h}\right) ,~v_{*}=\gamma u_{*}(1-u_{*}), \end{aligned}$$

where \(\delta =1/\beta \) with \(\beta >0\), and \(0<h<1\), \(\gamma >0\).

Defining a real-valued Hilbert space

$$\begin{aligned}\begin{aligned} {\mathscr {X}}&:=\left\{ (u, v)^{T} \in H^{2}(0, \ell \pi ) \times H^{2}(0, \ell \pi ): \frac{\partial u}{\partial x}\right. \\&\left. =\frac{\partial v}{\partial x} =0 \text{ at } x=0, \ell \pi \right\} \end{aligned}\end{aligned}$$

with the inner product defined by

$$\begin{aligned} \left[ U_{1}, U_{2}\right] =\int _{0}^{\ell \pi } U_{1}^{T} U_{2}~\mathrm{d}x \end{aligned}$$

for \(U_{1}=\left( u_{1}, v_{1}\right) ^{T} \in {\mathscr {X}}\) and \(U_{2}=\left( u_{2}, v_{2}\right) ^{T} \in {\mathscr {X}}\). The linearization of system (4) at positive constant equilibrium point \(E_{*}\left( u_{*}, v_{*}\right) \) is

$$\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t}\left( \begin{array}{l} u \\ v \end{array}\right) =\mathrm {d} \varDelta \left( \begin{array}{l}u \\ v\end{array}\right) +A\left( \begin{array}{l}u \\ v\end{array}\right) \end{aligned}$$
(12)

with

$$\begin{aligned} \begin{aligned}&\begin{aligned}&\mathrm {d} \varDelta =\left( \begin{array}{l} d_{11} \frac{\partial ^{2}}{\partial x^{2}}~~d_{12} \frac{\partial ^{2}}{\partial x^{2}} \\ d_{21} \frac{\partial ^{2}}{\partial x^{2}}~~d_{22} \frac{\partial ^{2}}{\partial x^{2}} \end{array}\right) , \\ A&=\left( \begin{array}{ll} a_{11} &{} a_{12} \\ a_{21} &{} a_{22} \end{array}\right) =\left( \begin{array}{ll} \frac{\gamma u_{*}}{2\left( u_{*}+h\right) }\left( -3 u_{*}+1-2 h\right) &{}~~ -1 \\ \frac{\gamma \left( 1-u_{*}\right) \left( u_{*}+2 h\right) }{2\left( u_{*}+h\right) } &{}~~ 0 \end{array}\right) \\&=\left( \begin{array}{ll} \frac{\gamma \left( \delta ^{2}\left( -3 \delta ^{2}-8 h+1\right) +\delta \sqrt{\delta ^{2}+4 h}\left( -3 \delta ^{2}-2 h+1\right) \right) }{2 \delta \sqrt{\delta ^{2}+4 h}+2 \delta ^{2}+4 h} &{}~~ -1 \\ \frac{\gamma \left( \delta \sqrt{\delta ^{2}+4 h}\left( 2-2 \delta ^{2}-4 h\right) +\delta ^{2}\left( 2-2 \delta ^{2}-8 h\right) +8 h\right) }{4\left( \delta \sqrt{\delta ^{2}+4 h}+\delta ^{2}+2 h\right) } &{}~~ 0 \end{array}\right) . \end{aligned}\end{aligned}\nonumber \\ \end{aligned}$$
(13)

It is well known that the eigenvalue problem

$$\begin{aligned}\left\{ \begin{aligned}&\varphi ^{\prime \prime }(x)={\widetilde{\lambda }} \varphi (x), \quad x \in (0, \ell \pi ), \\&\varphi ^{\prime }(0)=\varphi ^{\prime }(\ell \pi )=0 \end{aligned}\right. \end{aligned}$$

has eigenvalues \({\widetilde{\lambda }}_{k}=-k^{2}/\ell ^{2}\) with corresponding normalized eigenfunctions

$$\begin{aligned} \gamma _{k}(x)=\frac{\cos (\frac{k x}{\ell })}{\left\| \cos (\frac{k x}{\ell })\right\| _{L^{2}}}=\left\{ \begin{array}{ll} \frac{1}{\sqrt{\ell \pi }}, &{} k=0, \\ \frac{\sqrt{2}}{\sqrt{\ell \pi }} \cos (\frac{k x}{\ell }), &{} k \ge 1, \end{array}\right. \nonumber \\ \end{aligned}$$
(14)

where \(k \in \mathbb {N}_{0}=\mathbb {N}\cup \left\{ 0\right\} \) is often called wave number, and \(\mathbb {N}_{0}=\mathbb {N}\cup \left\{ 0\right\} \) is the set of all non-negative integers, \(\mathbb {N}=\left\{ 1,2,...\right\} \) represents the set of all positive integers. Thus, the eigenfunctions of \(d \varDelta \) on \({\mathscr {X}}\) corresponding to its eigenvalues have the form

$$\begin{aligned} \beta _{k}(x)=\left( \begin{array}{l}a_{k} \\ b_{k}\end{array}\right) \gamma _{k}(x), \end{aligned}$$

where \(a_{k}, b_{k} \in \mathbb {R}\) and \(\gamma _{k}(x)\) is defined by Eq. (14).

Let

$$\begin{aligned} \left( \begin{array}{l}y_{1} \\ y_{2} \end{array}\right) =\sum _{k=0}^{\infty }\left( \begin{array}{l}a_{k} \\ b_{k}\end{array}\right) \gamma _{k}(x) \end{aligned}$$

be an eigenfunction of \(\mathrm {d} \varDelta +A\) corresponding to an eigenvalue \(\lambda \), that is

$$\begin{aligned} (\mathrm {d} \varDelta +A)(y_{1}, y_{2})^{\mathrm {T}}=\lambda (y_{1}, y_{2})^{\mathrm {T}}. \end{aligned}$$

Then, we have

$$\begin{aligned} {\mathcal {L}}_{k}\left( \begin{array}{l}a_{k} \\ b_{k}\end{array}\right) =\lambda \left( \begin{array}{l}a_{k} \\ b_{k}\end{array}\right) , \quad k \in \mathbb {N}_{0}=\mathbb {N}\cup \left\{ 0\right\} , \end{aligned}$$

where

$$\begin{aligned} {\mathcal {L}}_{k}=\left( \begin{array}{cc}a_{11}-d_{11} \frac{k^{2}}{\ell ^{2}} &{} a_{12}-d_{12}\frac{k^{2}}{\ell ^{2}} \\ a_{21}-d_{21}\frac{k^{2}}{\ell ^{2}} &{} a_{22}-d_{22}\frac{k^{2}}{\ell ^{2}} \end{array}\right) \end{aligned}$$

which follows that the eigenvalues of \(\mathrm {d} \varDelta +A\) are given by the eigenvalues of \({\mathcal {L}}_{k}\) with \(k \in \mathbb {N}_{0}=\mathbb {N}\cup \left\{ 0\right\} \). Notice that the solutions of Eq. (12) can be obtained by using the eigenvalues and eigenvectors of the matrix \(d\varDelta +A\). Then by the Fourier expansion

$$\begin{aligned} \left( \begin{array}{l}u \\ v\end{array}\right) =\sum _{k=0}^{\infty } q_{k}^{\mathrm {T}}\left( \begin{array}{l}\xi _{k}^{1} \\ \xi _{k}^{2}\end{array}\right) \mathrm {e}^{\lambda t}, \quad q_{k}=\left( \begin{array}{l}q_{k 1} \\ q_{k 2}\end{array}\right) \in \mathbb {C}^{2},\nonumber \\ \end{aligned}$$
(15)

which can be seen as a nontrivial solutions of Eq. (12), where

$$\begin{aligned} \xi _{k}^{1}(x)=\left( \begin{array}{c}\gamma _{k}(x) \\ 0\end{array}\right) ,~\xi _{k}^{2}(x)=\left( \begin{array}{c}0 \\ \gamma _{k}(x)\end{array}\right) ,~k \in \mathbb {N}_{0}=\mathbb {N}\cup \left\{ 0\right\} , \end{aligned}$$

and by substituting (15) into (12), we can obtain the following sequence of characteristic equations \(\varDelta _{k}\), i.e.

$$\begin{aligned} \lambda ^{2}-T_{k} \lambda +D_{k}=0, \end{aligned}$$
(16)

where

$$\begin{aligned} \begin{aligned} T_{k}&=a_{11}-\left( d_{11}+d_{22}\right) \frac{k^{2}}{\ell ^{2}}, \\ D_{k}&=\left( d_{11}d_{22}-d_{12}d_{21}\right) \frac{k^{4}}{\ell ^{4}} \\&\quad -\left( a_{11}d_{22}+d_{21}-d_{12}a_{21} \right) \frac{k^{2}}{\ell ^{2}}+a_{21}. \end{aligned}\end{aligned}$$
(17)

By combining with Eqs. (16) and (17), and noticing that the necessary condition for the occurrence of Turing instability is \(d_{11} d_{22}-d_{12} d_{21}>0\) and

$$\begin{aligned}&4\left( d_{11}d_{22}-d_{12}d_{21}\right) a_{21}-\left( a_{11}d_{22}+d_{21}-d_{12}a_{21}\right) ^{2}\\&\quad <0, \end{aligned}$$

then in the \(d_{11}-\delta \) plane, we can define a curve \({\mathbf {L}}\) by

$$\begin{aligned} {\mathbf {L}}: \left( a_{11}d_{22}+d_{21}-d_{12}a_{21}\right) -2\sqrt{\left( d_{11}d_{22}-d_{12}d_{21}\right) a_{21}}=0.\nonumber \\ \end{aligned}$$
(18)

According to [32], we know that if there exist a nonnegative integer \(k_{1}\) and a positive integer \(k_{2} \ne k_{1}\) such that \(\varDelta _{k_{1}}=0\) has a pair of purely imaginary roots and \(\varDelta _{k_{2}}=0\) has a simple zero root, and no other roots of (16) has a zero real part, and the corresponding transversality conditions hold, then we call the bifurcation in this case as a Turing–Hopf bifurcation.

3.1 Stability analysis for the case without self-diffusion and cross-diffusion

In the case of without self-diffusion or cross-diffusion \(\left( d_{11}=d_{12}=d_{21}=d_{22}=0\right) \), the original system (4) becomes the following ordinary differential equation

$$\begin{aligned} \left\{ \begin{aligned}&\frac{d u}{\partial t}=\gamma u(1-u)-\frac{\beta u v}{\sqrt{u+h}},&t>0, \\&\frac{d v}{\partial t}=-v+\frac{\beta u v}{\sqrt{u+h}},&t>0, \\&u(0)=u_{0} \ge 0,~v(0)=v_{0} \ge 0. \end{aligned}\right. \end{aligned}$$
(19)

Clearly, the original system (4) and (19) have the same positive constant equilibrium point \(E_{*}(u_{*}, v_{*})\). By a simple linear analysis, we can obtain the following result.

Theorem 4

For the positive constant equilibrium point \(E_{*}(u_{*},v_{*})\) of system (19) with \(\gamma>0,~\beta >0,~0<h<1\), we have the following results on its stability.

  1. (i)

    When \(1/2<h<1\), the positive constant equilibrium point \(E_{*}\left( u_{*}, v_{*}\right) \) is always asymptotically stable;

  2. (ii)

    When \(0<h \le 1/2\), the positive constant equilibrium point \(E_{*}\left( u_{*}, v_{*}\right) \) is asymptotically stable for

    $$\begin{aligned} \max \left\{ 0, \delta ^{*}\right\}<\delta <\frac{1}{\sqrt{h+1}}, \quad \delta ^{*}=\frac{1-2 h}{\sqrt{3(1+h)}}, \end{aligned}$$

    and unstable for \(0<\delta <\delta ^{*}\).

Moreover, \(\varDelta _{0}=0\) has a pair of purely imaginary roots \(\pm i \sqrt{D_{0}}\) iff \(\delta =\delta ^{*}\), which implies that system (19) undergoes a Hopf bifurcation at \(\delta =\delta ^{*}\) near the positive constant equilibrium point \(E_{*}\left( u_{*}, v_{*}\right) \), and the Hopf bifurcation curve is defined by

$$\begin{aligned} \mathbf {H_{0}}: \delta =\delta ^{*}=\frac{1-2 h}{\sqrt{3(1+h)}}. \end{aligned}$$
(20)

Proof

By combining with Eqs. (16), (17) and \(0<\delta <1/\sqrt{h+1}\), we have

$$\begin{aligned} T_{0}=a_{11},~~D_{0}=a_{21}>0. \end{aligned}$$

Indeed, if \(0<\delta <1/\sqrt{h+1}\), follows that \(D_{0}=a_{21}>0\), then for the stability analysis of \(E_{*}(u_{*}, v_{*})\), it is sufficient to study the sign of

$$\begin{aligned} T_{0}=a_{11}=\frac{\gamma u_{*}}{2\left( u_{*}+h\right) }\left( -3 u_{*}+1-2 h\right) . \end{aligned}$$

Let \(P(\delta ):=-3 u_{*}+1-2h\), then by combining with

$$\begin{aligned} u_{*}=\frac{\delta }{2}\left( \delta +\sqrt{\delta ^{2}+4 h}\right) , \end{aligned}$$

we can obtain

$$\begin{aligned} P(\delta )=-\frac{3 \delta }{2}\left( \delta +\sqrt{\delta ^{2}+4 h}\right) +1-2h. \end{aligned}$$

Notice that for \(1/2<h<1\), \(P(\delta )\) is always negative. For \(0 < h \le 1/2\), notice that \(P(\delta )\) is decreasing with respect to \(\delta \), so the sign of \(P(\delta )\) is determined by its root \(\delta ^{*}\). Thus, \(T_{0}=a_{11}\) is positive for \(\delta <\delta ^{*}\) and negative conversely. Notice also that when \(\delta =\delta ^{*}\), we can obtain \(T_{0}=a_{11}=0\). Thus, when \(\delta =\delta ^{*}\), by combining with \(\lambda ^{2}-T_{0} \lambda +D_{0}=0\), we have \(\lambda =\pm i \sqrt{D_{0}}\) iff \(\delta =\delta ^{*}\).

This completes the proof. \(\square \)

3.2 Stability analysis for the case with self-diffusion and cross-diffusion

In this section, we investigate the effect of cross-diffusion on the positive constant equilibrium point \(E_{*}(u_{*}, v_{*})\) of the original system (4). We have the following Theorem 5.

Theorem 5

Assume that \(\gamma>0,~\beta >0,~0<h<1\), \(\mathbf{{H_{0}}}\) is defined by \(\delta =\delta ^{*}=(1-2h)/\sqrt{3(1+h)}\) and \(\mathbf{{L_{k}}}\) is defined by

$$\begin{aligned} \left( d_{11}d_{22}-d_{12}d_{21}\right) \frac{k^{4}}{\ell ^{4}}&-\left( a_{11}d_{22}+d_{21}-d_{12}a_{21} \right) \frac{k^{2}}{\ell ^{2}}\nonumber \\&+a_{21}=0. \end{aligned}$$
(21)

Then

\((\mathrm {I})\)    if \(0<d_{21}<(\ell ^{2} \gamma (4h+1))/3\), then original system (4) is always stable over the Hopf bifurcation curve \(\mathbf{{H_{0}}}\), that is, system (4) doesn’t undergo Turing–Hopf bifurcation;

\((\mathrm {II})\)    if \((\ell ^{2} \gamma (4h+1))/3<d_{21}<(d_{11}d_{22})/d_{12}\), then

  1. (i)

    The Hopf bifurcation curve \(\mathbf{{{H}_{0}}}\) intersects with the Turing bifurcation curve \(\mathbf{{L_{k}}}\), and a codimension-2 Turing–Hopf bifurcation occurs at the intersect point \(\left( d_{11}^{*}, \delta ^{*}\right) \), where

    $$\begin{aligned}\begin{aligned}&d_{11}^{*}=\frac{d_{12}d_{21}\frac{k_{*}^{4}}{\ell ^{4}}+(d_{21}-d_{12}\frac{\gamma (4h+1)}{3})\frac{k_{*}^{2}}{\ell ^{2}}-\frac{\gamma (4h+1)}{3}}{d_{22}\frac{k_{*}^{4}}{\ell ^{4}}}, \\&\delta ^{*}=\frac{1-2h}{\sqrt{3(1+h)}}; \end{aligned}\end{aligned}$$
  2. (ii)

    For \(\left( d_{11}, \delta \right) =\left( d_{11}^{*}, \delta ^{*}\right) \), the equation \(\varDelta _{0}=0\) has a pair of purely imaginary roots \(\pm i \omega _{c}\) and \(\varDelta _{k_{*}}=0\) has a simple zero root, and for the system (4), there are no other roots with zero real parts, where

    $$\begin{aligned} \omega _{c}=\sqrt{a_{21}(\delta ^{*})}=\sqrt{\frac{\gamma (4 h+1)}{3}}. \end{aligned}$$

Proof

The Turing bifurcation curve \(\mathbf {L_{k}}\) defined by (21) is followed from \(D_{k}=0\), i.e.

$$\begin{aligned} \left( d_{11}d_{22}-d_{12}d_{21}\right) \frac{k^{4}}{\ell ^{4}}-\left( a_{11}d_{22}+d_{21}-d_{12}a_{21} \right) \frac{k^{2}}{\ell ^{2}}+a_{21}=0. \end{aligned}$$

By combining with Eqs. (16) and (17), we can see that \(T_{k}<0\) provided that \(a_{11}<0\), and that \(D_{k}>0\) is equivalent to

$$\begin{aligned} d_{11}>\frac{d_{12}d_{21}\frac{k^{4}}{\ell ^{4}}+(a_{11}d_{22}+d_{21}-d_{12}a_{21})\frac{k^{2}}{\ell ^{2}}-a_{21}}{d_{22}\frac{k^{4}}{\ell ^{4}}}. \end{aligned}$$

Notice that \(a_{11}(\delta ^{*})=0\), \(a_{21}(\delta ^{*})=\gamma (4h+1)/3\), and when

$$\begin{aligned} d_{12}d_{21}\frac{k^{4}}{\ell ^{4}}+(a_{11}(\delta ^{*})d_{22}+d_{21}-d_{12}a_{21}(\delta ^{*}))\frac{k^{2}}{\ell ^{2}}-a_{21}(\delta ^{*})<0, \end{aligned}$$

i.e. \(0<d_{21}<(\ell ^{2} \gamma (4h+1))/3\), the Turing bifurcation curve \(\mathbf{{L_{k}}}\) doesn’t interact with the Hopf bifurcation curve \(\mathbf{{H_{0}}}\). Here, \(a_{11}(\delta ^{*})\), \(a_{21}(\delta ^{*})\) indicating \(a_{11}\) and \(a_{21}\) are dependent on \(\delta ^{*}\), respectively. That is, system (4) is always asymptotically stable over the Hopf bifurcation curve \(\mathbf{{H_{0}}}\). Thus, the conclusion \((\mathrm {I})\) is confirmed.

When \(d_{11} d_{22}-d_{12} d_{21}>0\) and

$$\begin{aligned} d_{12}d_{21}\frac{k^{4}}{\ell ^{4}}+(a_{11}(\delta ^{*})d_{22}+d_{21}-d_{12}a_{21}(\delta ^{*}))\frac{k^{2}}{\ell ^{2}}-a_{21}(\delta ^{*})>0, \end{aligned}$$

i.e. \((\ell ^{2} \gamma (4h+1))/3<d_{21}<(d_{11}d_{22})/d_{12}\), the Turing bifurcation curve \(\mathbf{{L_{k}}}\) interact with the Hopf bifurcation curve \(\mathbf{{H_{0}}}\), then by substituting \(\delta =\delta ^{*}\) into \(D_{k}=0\) and solving \(D_{k}=0\) for \(d_{11}\), we have that

$$\begin{aligned}\begin{aligned} d_{11}(k)&=\frac{d_{12}d_{21}\frac{k^{4}}{\ell ^{4}}+(d_{21}-d_{12}\frac{\gamma (4h+1)}{3})\frac{k^{2}}{\ell ^{2}}-\frac{\gamma (4h+1)}{3}}{d_{22}\frac{k^{4}}{\ell ^{4}}}, \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~k>k^{*}, \end{aligned}\end{aligned}$$

where \(k^{*}\) is defined by Eq. (22). Furthermore, the symbol \([~\cdot ~]\) in Eq. (22) stands for the integer part function.

$$\begin{aligned} k^{*}=\left[ \ell \sqrt{\frac{(d_{12}\frac{\gamma (4h+1)}{3}-d_{21})+\sqrt{(d_{21}-d_{12}\frac{\gamma (4h+1)}{3})^{2}+\frac{4\gamma (4h+1)}{3}d_{12}d_{21}}}{2d_{12}d_{21}}}\right] . \end{aligned}$$
(22)

Let

$$\begin{aligned} f(x)=\frac{d_{12}d_{21}x^{2}+(d_{21}-d_{12}\frac{\gamma (4h+1)}{3})x-\frac{\gamma (4h+1)}{3}}{d_{22}x^{2}}, \quad x>k^{*}, \end{aligned}$$

we can obtain that

$$\begin{aligned} f^{\prime }(x)=\frac{\left( d_{12}\frac{\gamma (4h+1)}{3}-d_{21}\right) x^{2}+\frac{2\gamma (4h+1)}{3}x}{d_{22}x^{4}}, \quad x >k^{*}. \end{aligned}$$

Therefore, if \((d_{12}\gamma (4h+1))/3-d_{21}<0\), we have \(f^{\prime }(x) \ge 0\) for \(x \le x^{*}\), and \(f^{\prime }(x)<0\) for \(x>x^{*}\), where

$$\begin{aligned} x^{*}=\frac{2\gamma (4h+1)}{3d_{21}-\gamma d_{12}(4h+1)}. \end{aligned}$$

By setting

$$\begin{aligned} k_{*}=\left\{ \begin{aligned}&{[\ell \sqrt{x^{*}}],}~\text {if}~~d_{11}\left( [\ell \sqrt{x^{*}}]+1\right) \le d_{11}\left( [\ell \sqrt{x^{*}}]\right) , \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x^{*}>k^{*}, \\&{[\ell \sqrt{x^{*}}]+1,}~\text {if}~~d_{11}\left( [\ell \sqrt{x^{*}}]\right) <d_{11}\left( [\ell \sqrt{x^{*}}]+1\right) , \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x^{*}>k^{*}, \end{aligned}\right. \nonumber \\ \end{aligned}$$
(23)

then there exists a \(k_{*}>k^{*}\) such that \(d_{11}^{*}=d_{11}\left( k_{*}\right) =\max _{k>k^{*}} d_{11}(k)\).

Furthermore, if \((d_{12}\gamma (4h+1)/3)-d_{21}>0\), we have \(f^{\prime }(x) \le 0\) for \(x \le x^{*}\), and \(f^{\prime }(x)>0\) for \(x>x^{*}\). By setting

$$\begin{aligned} k_{*}=\left\{ \begin{aligned}&{[\ell \sqrt{x^{*}}]+1},~\text {if}~~d_{11}\left( [\ell \sqrt{x^{*}}]+1\right) \le d_{11}\left( [\ell \sqrt{x^{*}}]\right) , \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x^{*}>k^{*}, \\&{[\ell \sqrt{x^{*}}]},~\text {if}~~d_{11}\left( [\ell \sqrt{x^{*}}]\right) <d_{11}\left( [\ell \sqrt{x^{*}}]+1\right) , \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x^{*}>k^{*}, \end{aligned}\right. \end{aligned}$$

then there exists a \(k_{*}>k^{*}\) such that \(d_{11}^{*}=d_{11}\left( k_{*}\right) =\min _{k>k^{*}} d_{11}(k)\).

Thus, the Hopf bifurcation curve \({\mathbf {H_{0}}}\) intersects with the Turing bifurcation curve \({\mathbf {L_{k}}}\) at

$$\begin{aligned}\begin{aligned} \left( d_{11}^{*}, \delta ^{*}\right)&= \left( \frac{d_{12}d_{21}\frac{k_{*}^{4}}{\ell ^{4}}+(d_{21}-d_{12} \frac{\gamma (4h+1)}{3})\frac{k_{*}^{2}}{\ell ^{2}}-\frac{\gamma (4h+1)}{3}}{d_{22}\frac{k_{*}^{4}}{\ell ^{4}}}, \right. \\&\left. \frac{1-2h}{\sqrt{3(1+h)}}\right) , \end{aligned}\end{aligned}$$

which is called as the Turing–Hopf bifurcation point.

Next, we continue to verify the transversality condition. Taking \(d_{11}\) as a bifurcation parameter and letting \(\lambda (d_{11})\) be the root of Eq. (16) near \(d_{11}=d^{*}_{11}\) satisfying \(\lambda \left( d^{*}_{11}\right) =0\). Differentiating the two sides of the characteristic equation (16) with respect to \(d_{11}\), we obtain

$$\begin{aligned} 2\lambda \frac{d\lambda (d_{11})}{dd_{11}}-\left( \frac{dT_{k}(d_{11})}{dd_{11}}\lambda +T_{k}\frac{d\lambda (d_{11})}{dd_{11}} \right) +\frac{dD_{k}(d_{11})}{dd_{11}}=0. \end{aligned}$$

By combining with Eq. (17), we can obtain

$$\begin{aligned} \frac{d\lambda (d_{11})}{dd_{11}}=\frac{-d_{22}\frac{k^{4}}{\ell ^{4}}-\frac{k^{2}}{\ell ^{2}}\lambda }{2\lambda -T_{k}}. \end{aligned}$$
(24)

Moreover, when \(\delta >\delta ^{*}\), from the discussion in Sect. 3.1, we know that \(a_{11}<0\). Thus, when \(\delta >\delta ^{*}\), we can obtain that

$$\begin{aligned} T_{k_{*}}=a_{11}-\left( d_{11}+d_{22}\right) \frac{k_{*}^{2}}{\ell ^{2}}<0. \end{aligned}$$
(25)

By combining with Eqs. (24) and (25), we have

$$\begin{aligned}\begin{aligned}&\text { Re }\left( \left. \frac{d\lambda (d_{11})}{dd_{11}}\right| _{\mathbf {L_{k_{*}}}}\right) =\text { Re }\left( \left. \frac{-d_{22}\frac{k^{4}}{\ell ^{4}}-\frac{k^{2}}{\ell ^{2}}\lambda }{2\lambda -T_{k}}\right| _{\mathbf {L_{k_{*}}}}\right) \\&=\text { Re }\left( \frac{-d_{22}\frac{k_{*}^{4}}{\ell ^{4}}}{-T_{k_{*}}}\right) =\frac{d_{22}\frac{k_{*}^{4}}{\ell ^{4}}}{T_{k_{*}}}<0, \end{aligned}\end{aligned}$$

where the symbol \(\text {Re}(\vartheta )\) represents the real part of \(\vartheta \). This, together with the fact that

$$\begin{aligned}\begin{aligned} \mathrm{{sign}}\left( \left. \frac{\mathrm {d}\mathrm{{Re}}(\lambda (d_{11}))}{\mathrm {d} d_{11}}\right| _{\mathbf {L_{k_{*}}}}\right)&=sign\left( \text { Re }\left( \left. \frac{\mathrm{d}\lambda (d_{11})}{\mathrm{d}d_{11}}\right| _{\mathbf {L_{k_{*}}}}\right) \right) \\&<0, \end{aligned}\end{aligned}$$

we have

$$\begin{aligned} \left. \frac{\mathrm {d}Re(\lambda (d_{11}))}{\mathrm {d} d_{11}}\right| _{\mathbf {L_{k_{*}}}}<0, \end{aligned}$$

where the symbol sign represents the sign function.

Moreover, if taking \(\delta \) as a bifurcation parameter, letting \(\lambda (\delta )=\alpha (\delta )\pm i \beta (\delta )\) be the pair of roots of Eq. (16) near \(\delta =\delta ^{*}\) satisfying \(\alpha \left( \delta ^{*}\right) =0\) and \(\beta \left( \delta ^{*}\right) =\omega _{c}\), we have the following transversality condition. Differentiating the two sides of Eq. (16) with respect to \(\delta \), we obtain

$$\begin{aligned} 2\lambda \frac{\mathrm{d}\lambda (\delta )}{\mathrm{d}\delta }-\left( \frac{\mathrm{d}T_{k}(\delta )}{\mathrm{d}\delta }\lambda +T_{k}\frac{\mathrm{d}\lambda (\delta )}{\mathrm{d}\delta } \right) +\frac{\mathrm{d}D_{k}(\delta )}{\mathrm{d}\delta }=0.\nonumber \\ \end{aligned}$$
(26)

By combining with Eqs. (17) and (26), we have

$$\begin{aligned} \frac{\mathrm{d}\lambda (\delta )}{\mathrm{d}\delta }=\frac{\left( \lambda +d_{22}\frac{k^{2}}{\ell ^{2}}\right) \frac{\mathrm{d}a_{11}}{\mathrm{d}\delta }-\left( d_{12}\frac{k^{2}}{\ell ^{2}}+1\right) \frac{\mathrm{d}a_{21}}{\mathrm{d}\delta }}{2\lambda -T_{k}}.\nonumber \\ \end{aligned}$$
(27)

Furthermore, by Eqs. (17) and (27) as well as \(T_{0}(\delta ^{*})=a_{11}(\delta ^{*})=0\) and

$$\begin{aligned} \left. \frac{\mathrm{d}a_{11}(\delta )}{\mathrm{d} \delta }\right| _{\delta =\delta ^{*}}=\frac{3 \gamma (h+1)(2 h-1)}{(4 h+1) \sqrt{3(h+1)}}>0, \end{aligned}$$

we have

$$\begin{aligned}\begin{aligned}&\text { Re }\left( \left. \frac{\mathrm{d}\lambda (\delta )}{\mathrm{d}\delta }\right| _{\mathbf {H_{0}}}\right) =\text { Re }\left( \frac{i\omega _{c}\frac{\mathrm{d}a_{11}}{\mathrm{d}\delta }|_{\delta =\delta ^{*}}-\frac{\mathrm{d}a_{21}}{\mathrm{d}\delta }|_{\delta =\delta ^{*}}}{2i\omega _{c}-T_{0}(\delta ^{*})}\right) \\&=\left. \frac{1}{2}\frac{\mathrm{d}a_{11}}{\mathrm{d}\delta }\right| _{\delta =\delta ^{*}}=\frac{3 \gamma (h+1)(2 h-1)}{2(4 h+1) \sqrt{3(h+1)}}>0. \end{aligned}\end{aligned}$$

This, together with the fact that

$$\begin{aligned} \mathrm{{sign}}\left( \left. \frac{\mathrm {d}\mathrm{{Re}}(\lambda (\delta ))}{\mathrm {d} \delta }\right| _{{\mathbf {H}}_{0}}\right) =\mathrm{{sign}}\left( \text { Re }\left( \left. \frac{\mathrm{d}\lambda (\delta )}{\mathrm{d}\delta }\right| _{\delta =\delta ^{*}}\right) \right) >0, \end{aligned}$$

we have

$$\begin{aligned} \left. \frac{\mathrm {d} Re(\lambda (\delta ))}{\mathrm {d} \delta }\right| _{{\mathbf {H}}_{0}}>0. \end{aligned}$$

Thus, the proof of conclusion \((\mathrm {II})\) is completed. \(\square \)

3.3 Bifurcation diagram for system (4) in \(d_{11}-\delta \) plane

Taking \(\ell =1, \gamma =3, h=0.1, d_{12}=1, d_{21}=10, d_{22}=15\), and by combining with Eqs. (13), (18), (20), (21), (22) and (23), we can conclude that \(k^{*}=0\), \(k_{*}=1\) and

$$\begin{aligned}\begin{aligned}&{\mathbf {H_{0}}}:\delta =\frac{1-2h}{\sqrt{3(1+h)}}=0.4404, \\&a_{21} \\&=\frac{3\left( \delta \sqrt{\delta ^{2}+0.4}\left( 2-2 \delta ^{2}-0.4\right) +\delta ^{2}\left( 2-2 \delta ^{2}-0.8\right) +0.8\right) }{4\left( \delta \sqrt{\delta ^{2}+0.4}+\delta ^{2}+0.2\right) }, \\&{{\mathbf {L}}}:\frac{45\left( \delta ^{2}\left( -3 \delta ^{2}+0.2\right) +\delta \sqrt{\delta ^{2}+0.4}\left( -3 \delta ^{2}+0.8\right) \right) }{2 \delta \sqrt{\delta ^{2}+0.4}+2 \delta ^{2}+0.4} \\&~~~~-2\sqrt{(15d_{11}-10)a_{21}}+10-a_{21}=0, \\&{\mathbf {L_{1}}}:15d_{11}-\frac{45\left( \delta ^{2}\left( -3 \delta ^{2}+0.2\right) +\delta \sqrt{\delta ^{2}+0.4}\left( -3 \delta ^{2}+0.8\right) \right) }{2 \delta \sqrt{\delta ^{2}+0.4}+2 \delta ^{2}+0.4} \\&~~~~-20+2a_{21}=0. \end{aligned}\end{aligned}$$

Numerical calculation confirms that when \(k_{*}=1\), the Turing bifurcation curves \(\mathbf {L_{1}}\) are tangent to the curve \({\mathbf {L}}\) at \(P_{1}(-0.0131,0.8717)\) and \(P_{2}(0.5583,0.7399)\), see Fig. 2a for details. Moreover, the Hopf bifurcation curve \({\mathbf {H_{0}}}\) intersects with Turing bifurcation curve \({\mathbf {L_{1}}}\) at the point \(\left( d_{11}^{*},\delta ^{*}\right) =\left( 1.1467, 0.4404\right) \) and system (4) undergoes Turing–Hopf bifurcation near the positive constant equilibrium point \(E_{*}\left( 0.2667, 0.5867\right) \), see Fig. 2b for details.

Fig. 2
figure 2

Bifurcation diagram for system (4) in \(d_{11}-\delta \) plane with \(k_{*}=1\). \(\mathbf {H_{0}}\) denotes the Hopf bifurcation curve, \(\mathbf {L_{1}}\) denotes the Turing bifurcation curve and \({\mathbf {L}}\) is the curve defined by (18). The region \(R_{1}\) is the stability region of the positive constant equilibrium point \(E_{*}\), \(R_{2}\) represents the region of Turing instability, and \(R_{3}\) and \(R_{4}\) are the instability region of the positive constant equilibrium point \(E_{*}\)

4 Normal form for Turing–Hopf bifurcation in the reaction–diffusion system (4)

Notice that the method of computing normal form of general reaction–diffusion system which presented in [34] can be used to obtain the third-order truncated normal form of system (4) by a slight modification. Denote the Turing–Hopf bifurcation point as \(\left( d_{11}^{*}, \delta ^{*}\right) \). Introduce the perturbation parameters \(\mu _{1}\) and \(\mu _{2}\) by setting \(d_{11}=d_{11}^{*}+\mu _{1}\) and \(\delta =\delta ^{*}+\mu _{2}\) such that \(\left( \mu _{1}, \mu _{2}\right) =(0,0)\) is the Turing–Hopf bifurcation point in the perturbation plane of \(\mu _{1}\) and \(\mu _{2}\). Then the system (4) becomes

$$\begin{aligned} \left\{ \begin{aligned} \frac{\partial u}{\partial t}&=(d_{11}^{*}+\mu _{1}) \varDelta u+d_{12} \varDelta v+\gamma u(1-u) \\&\quad -\frac{u v}{(\delta ^{*}+\mu _{2})\sqrt{u+h}},~~~~~~x \in \varOmega ,~t>0, \\ \frac{\partial v}{\partial t}&=d_{21} \varDelta u+d_{22} \varDelta v-v+\frac{u v}{(\delta ^{*}+\mu _{2})\sqrt{u+h}}, \\&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x \in \varOmega ,~t>0. \end{aligned}\right. \nonumber \\ \end{aligned}$$
(28)

The positive constant equilibrium point of system (28) becomes

$$\begin{aligned}\begin{aligned} u_{*}&=\frac{\delta ^{*}+\mu _{2}}{2}\left( \delta ^{*}+\mu _{2}+\sqrt{(\delta ^{*}+\mu _{2})^{2}+4 h}\right) , \\ v_{*}&=\lambda u_{*}(1-u_{*}) \end{aligned}\end{aligned}$$

with \(0<\delta ^{*}+\mu _{2}<1/\sqrt{h+1}\). Making the change of variables by the translation \({\overline{u}}=u-u_{*}\) and \({\overline{v}}=v-v_{*}\), and dropping the bars, then the system (28) is transformed into

$$\begin{aligned} \left\{ \begin{aligned} \frac{\partial u}{\partial t}&=(d_{11}^{*}+\mu _{1}) \varDelta u+d_{12} \varDelta v+\gamma \left( u+u_{*}\right) \left( 1-(u+u_{*})\right) \\&\quad -\frac{\left( u+u_{*}\right) \left( v+v_{*}\right) }{(\delta ^{*}+\mu _{2})\sqrt{\left( u+u_{*}\right) +h}},~~x \in \varOmega ,~t>0, \\ \frac{\partial v}{\partial t}&=d_{21} \varDelta u+d_{22} \varDelta v-\left( v+v_{*}\right) \\&\quad +\frac{\left( u+u_{*}\right) \left( v+v_{*}\right) }{(\delta ^{*}+\mu _{2})\sqrt{\left( u+u_{*}\right) +h}},~~x \in \varOmega ,~t>0. \end{aligned}\right. \nonumber \\ \end{aligned}$$
(29)

For the system (29), when \(\mu _{1}=\mu _{2}=0\), \(\varDelta _{0}(\lambda )=0\) has a pair of purely imaginary roots \(\pm i \omega _{c}\), \(\varDelta _{k_{*}}(\lambda )=0\) has a simple zero root \(\lambda =0\) and a negative real root \(\lambda =-T_{k_{*}}\), and if \(k \ne 0, k_{*}\), all roots of \(\varDelta _{k}(\lambda )=0\) have negative real parts.

By the real-valued Hilbert space \({\mathscr {X}}\) which is defined in Sect. 3, the system (29) can be written as the following abstract ordinary differential equation (ODE), that is

$$\begin{aligned} \frac{\mathrm{d} U(t)}{\mathrm{d}t}=\mathrm{d}\varDelta U+L_{0}(U)+{\widetilde{F}}(U, \mu ) \end{aligned}$$
(30)

where \(U=(u, v)^{T},~\mu =\left( \mu _{1}, \mu _{2}\right) ,~{\widetilde{F}}(U, \mu )=L(\mu )(U)-L_{0}(U)+{\widetilde{d}}\varDelta u+F(U, \mu )\),

$$\begin{aligned} \begin{aligned}&d=\left( \begin{array}{cc} d_{11}^{*} &{}~~d_{12} \\ d_{21} &{}~~d_{22} \\ \end{array}\right) ,~~{\widetilde{d}}=\left( \begin{array}{cc} \mu _{1} &{}~~0 \\ 0 &{}~~0 \end{array}\right) , \\&L(\mu )=\left( \begin{array}{cc} a_{11}(\delta ^{*}+\mu _{2}) &{}~~-1 \\ a_{21}(\delta ^{*}+\mu _{2}) &{}~~0 \end{array}\right) . \end{aligned}\end{aligned}$$
(31)

Here, \(a_{11}(\delta ^{*}+\mu _{2})\), \(a_{21}(\delta ^{*}+\mu _{2})\) indicating \(a_{11}\) and \(a_{21}\) are dependent on \(\delta ^{*}+\mu _{2}\), respectively. Moreover, we have

$$\begin{aligned} F(U, \mu )=\left( \begin{aligned}&F^{(1)}\left( u, v, \mu _{1}, \mu _{2}\right) \\&F^{(2)}\left( u, v, \mu _{1}, \mu _{2}\right) \end{aligned}\right) \end{aligned}$$

with

$$\begin{aligned} \begin{aligned} F^{(1)}\left( u, v, \mu _{1}, \mu _{2}\right)&=\gamma \left( u+u_{*}\right) \left( 1-(u+u_{*})\right) \\&\quad -\frac{\left( u+u_{*}\right) \left( v+v_{*}\right) }{(\delta ^{*}+\mu _{2})\sqrt{\left( u+u_{*}\right) +h}}\\&\quad -a_{11}(\delta ^{*}+\mu _{2})u+v, \\ F^{(2)}\left( u, v, \mu _{1}, \mu _{2}\right)&=-\left( v+v_{*}\right) \\&\quad +\frac{\left( u+u_{*}\right) \left( v+v_{*}\right) }{(\delta ^{*}+\mu _{2})\sqrt{\left( u+u_{*}\right) +h}}\\&\quad -a_{21}(\delta ^{*}+\mu _{2})u. \end{aligned}\end{aligned}$$
(32)

For the formal Taylor expansions of L

$$\begin{aligned}\begin{aligned} L(\mu )&=L_{0}+\mu _{1} L_{1}^{(1,0)}+\mu _{2} L_{1}^{(0,1)} \\&\quad +\frac{1}{2}\left( \mu _{1}^{2} L_{2}^{(2,0)}+2 \mu _{1} \mu _{2} L_{2}^{(1,1)}+\mu _{2}^{2}, L_{2}^{(0,2)}\right) \\&\quad +\cdots , \end{aligned}\end{aligned}$$

and by Eq. (31), we can obtain

$$\begin{aligned} \begin{aligned}&L_{0}=\left( \begin{array}{cc} a_{11}(\delta ^{*}) &{}~~-1 \\ a_{21}(\delta ^{*}) &{}~~0 \end{array}\right) ,~L_{1}^{(1,0)}=\left( \begin{array}{cc} 0 &{}~~0 \\ 0 &{}~~0 \end{array}\right) , \\&L_{1}^{(0,1)}=\left( \begin{array}{cc} A &{}~~0 \\ B &{}~~0 \end{array}\right) , \end{aligned} \end{aligned}$$

where

$$\begin{aligned}\begin{aligned}&A:= \\&\frac{\gamma \left( -3(u_{*})^{2}-6u_{*}h+h-2h^{2}\right) \left( 2\delta +\sqrt{\delta ^{2}+4h}+\frac{\delta ^{2}}{\sqrt{\delta ^{2}+4h}}\right) }{4(u_{*}+h)^{2}}, \\&B:= \\&\frac{\gamma \left( -h-(u_{*})^{2}-2u_{*}h-2h^{2}\right) \left( 2\delta +\sqrt{\delta ^{2}+4h}+\frac{\delta ^{2}}{\sqrt{\delta ^{2}+4h}}\right) }{4(u_{*}+h)^{2}}. \end{aligned}\end{aligned}$$

Let

$$\begin{aligned}\begin{aligned} {\mathcal {M}}_{k}(\lambda )&=\lambda I_{2}-\left( \begin{array}{cc}-d_{11}^{*}\frac{k^{2}}{\ell ^{2}} &{} -d_{12}\frac{k^{2}}{\ell ^{2}} \\ -d_{21}\frac{k^{2}}{\ell ^{2}} &{} -d_{22}\frac{k^{2}}{\ell ^{2}}\end{array}\right) -L_{0} \\&=\left( \begin{array}{cc} \lambda +d_{11}^{*}\frac{k^{2}}{\ell ^{2}} &{}~~d_{12}\frac{k^{2}}{\ell ^{2}}+1 \\ d_{21}\frac{k^{2}}{\ell ^{2}}-\frac{\gamma (4 h+1)}{3} &{}~~\lambda +d_{22}\frac{k^{2}}{\ell ^{2}} \end{array}\right) , \end{aligned}\end{aligned}$$

where \(I_{2}\) is a \(2 \times 2\) identity matrix, then by a straightforward calculation, we can obtain that \(\xi _{0} \in \mathbb {C}^{2}\) and \(\xi _{k_{*}} \in \mathbb {R}^{2}\) are the eigenvectors associated with the eigenvalues \(i \omega _{c}\) and 0, respectively, and \(\eta _{0} \in \mathbb {C}^{2}\) and \(\eta _{k_{*}} \in \mathbb {R}^{2}\) are the corresponding adjoint eigenvectors, where

$$\begin{aligned} \begin{aligned}&\xi _{0}=\left( \begin{array}{c} 1 \\ -i \omega _{c} \end{array}\right) ,~~\eta _{0}=\frac{1}{2}\left( \begin{array}{c} 1 \\ \frac{i}{\omega _{c}} \end{array}\right) ,~~\xi _{k_{*}}=\left( \begin{array}{c} 1 \\ -\frac{d_{11}^{*}\frac{k_{*}^{2}}{\ell ^{2}}}{d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1} \end{array}\right) , \\&\eta _{k_{*}}=\left( \begin{array}{c} \frac{3d_{21}\frac{k_{*}^{2}}{\ell ^{2}}\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) -\gamma (4h+1)\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) }{3d_{21}\frac{k_{*}^{2}}{\ell ^{2}}\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) -\gamma (4h+1)\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) +3(d_{11}^{*})^{2}\frac{k_{*}^{4}}{\ell ^{4}}} \\ \frac{-3d_{11}^{*}\frac{k_{*}^{2}}{\ell ^{2}}\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) }{3d_{21}\frac{k_{*}^{2}}{\ell ^{2}}\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) -\gamma (4h+1)\left( d_{12}\frac{k_{*}^{2}}{\ell ^{2}}+1\right) +3(d_{11}^{*})^{2}\frac{k_{*}^{4}}{\ell ^{4}}} \end{array}\right) , \end{aligned} \end{aligned}$$

such that

$$\begin{aligned} \left\langle \varPsi _{1}, \varPhi _{1}\right\rangle =I_{2},~\left\langle \varPsi _{2}, \varPhi _{2}\right\rangle =1, \end{aligned}$$

where

$$\begin{aligned} \varPhi _{1}=\left( \xi _{0},{\overline{\xi }}_{0}\right) ,~\varPhi _{2}=\xi _{k_{*}},~\varPsi _{1}={\text {col}}\left( \eta _{0}^{T}, {\overline{\eta }}_{0}^{T}\right) ,~\varPsi _{2}=\eta _{k_{*}}^{T}. \end{aligned}$$

Here, \(\text {col}(.)\) represents the column vector. Furthermore, for vectors \({\widehat{\varphi }}, {\widehat{\psi }} \in \mathbb {R}^{2}\), we define their scalar product as \(\left\langle {\widehat{\psi }}^{T}, {\widehat{\varphi }}\right\rangle ={\widehat{\psi }}^{T} {\widehat{\varphi }}\).

Notice that the phase space \({\mathscr {X}}\) can be decomposed as

$$\begin{aligned} {\mathscr {X}}={\mathcal {P}} \oplus {\mathcal {Q}},~{\mathcal {P}}={\text {Im}} \pi ,~{\mathcal {Q}}={\text {Ker}} \pi , \end{aligned}$$
(33)

where \({\text {dim}} {\mathcal {P}}=3\) and \(\pi : {\mathscr {X}} \mapsto {\mathcal {P}}\) is the projection defined by

$$\begin{aligned}\begin{aligned} \pi (U)&=\left( \varPhi _{1}\left\langle \varPsi _{1},\left( \left[ U, \beta _{0}^{(1)}\right] ,\left[ U, \beta _{0}^{(2)}\right] \right) ^{T}\right\rangle \right) ^{T} \beta _{0} \\&\quad +\left( \varPhi _{2}\left\langle \varPsi _{2},\left( \left[ U, \beta _{k_{*}}^{(1)}\right] ,\left[ U, \beta _{k_{*}}^{(2)}\right] \right) ^{T}\right\rangle \right) ^{T} \beta _{k_{*}}. \end{aligned}\end{aligned}$$

According to (33), \(U \in {\mathscr {X}}\) can be decomposed as

$$\begin{aligned} \begin{aligned} U&=\left( \varPhi _{1}\left( \begin{array}{c} z_{1} \\ z_{2} \end{array}\right) \right) ^{T}\left( \begin{array}{c} \beta _{0}^{(1)} \\ \beta _{0}^{(2)} \end{array}\right) +\left( z_{3} \varPhi _{2}\right) ^{T}\left( \begin{array}{c} \beta _{k_{*}}^{(1)} \\ \beta _{k_{*}}^{(2)} \end{array}\right) +w \\&=\left( z_{1} \xi _{0}+z_{2} {\overline{\xi }}_{0}\right) \gamma _{0}(x)+z_{3} \xi _{k_{*}} \gamma _{k_{*}}(x)+w \\&=\left( \varPhi _{1}, \varPhi _{2}\right) \left( \begin{array}{c} z_{1} \gamma _{0}(x) \\ z_{2} \gamma _{0}(x) \\ z_{3} \gamma _{k_{*}}(x) \end{array}\right) +\left( \begin{array}{c} w_{1} \\ w_{2} \end{array}\right) , \end{aligned}\end{aligned}$$
(34)

where \(w=\text {col}\left( w_{1}, w_{2}\right) \) and

$$\begin{aligned}\begin{aligned} \left( \begin{array}{cc} z_{1} \\ z_{2} \end{array}\right)&=\left\langle \varPsi _{1},\left( \left[ U, \beta _{0}^{(1)}\right] ,\left[ U, \beta _{0}^{(2)}\right] \right) ^{T}\right\rangle , \\ z_{3}&=\left\langle \varPsi _{2},\left( \left[ U, \beta _{k_{*}}^{(1)}\right] ,\left[ U, \beta _{k_{*}}^{(2)}\right] \right) ^{T}\right\rangle . \end{aligned}\end{aligned}$$

By letting

$$\begin{aligned} \varPhi =\left( \varPhi _{1}, \varPhi _{2}\right) , z_{x}=\left( z_{1} \gamma _{0}(x), z_{2} \gamma _{0}(x), z_{3} \gamma _{k_{*}}(x)\right) ^{T}, \end{aligned}$$

Equation (34) can be rewritten as

$$\begin{aligned} U=\varPhi z_{x}+w. \end{aligned}$$

For the simplicity of notation, we denote

$$\begin{aligned} \left( \begin{array}{c} {\left[ {\widetilde{F}}, \beta _{v}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}, \beta _{v}^{(2)}\right] } \end{array}\right) _{v=0}^{v=k_{*}}={\text {col}}\left( \left( \begin{array}{c} {\left[ {\widetilde{F}}, \beta _{0}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}, \beta _{0}^{(2)}\right] } \end{array}\right) ,\left( \begin{array}{c} {\left[ {\widetilde{F}}, \beta _{k_{*}}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}, \beta _{k_{*}}^{(2)}\right] } \end{array}\right) \right) , \end{aligned}$$

and if let \({\mathscr {L}}U:=d \varDelta U+L_{0}(U)\) and denote by \({\mathscr {L}}_{1}\) the restriction of \({\mathscr {L}}\) to \({\mathcal {Q}}\), then system (30) is equivalent to a system of abstract ordinary differential equations (ODEs) in \(\mathbb {R}^{3} \times {\mathcal {Q}}\), with finite- and infinite-dimensional variables also separated in the linear term. That is,

$$\begin{aligned} \left\{ \begin{array}{l} \dot{z}=Bz+\varPsi \left( \begin{array}{l} {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{v}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{v}^{(2)}\right] } \end{array}\right) _{v=0}^{v=k_{*}}, \\ \dot{w}={\mathscr {L}}_{1}(w)+H(z, w, \mu ), \end{array}\right. \end{aligned}$$

where

$$\begin{aligned}\begin{aligned}&z=\left( z_{1}, z_{2}, z_{3}\right) ^{T}, B={\text {diag}}\left\{ i \omega _{c},-i \omega _{c}, 0\right\} , \\&\varPsi ={\text {diag}}\left\{ \varPsi _{1}, \varPsi _{2}\right\} , {\widetilde{F}}(z, w, \mu )={\widetilde{F}}\left( \varPhi z_{x}+w, \mu \right) , \end{aligned}\end{aligned}$$

and

$$\begin{aligned}\begin{aligned}&H(z, w, \mu )={\widetilde{F}}(z, w, \mu )-\left( \left\langle \eta _{0}^{T},\left( \begin{array}{c} {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{0}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{0}^{(2)}\right] } \end{array}\right) \right\rangle \xi _{0}\right. \\&\qquad \left. +\left\langle {\overline{\eta }}_{0}^{T},\left( \begin{array}{c} {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{0}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{0}^{(2)}\right] } \end{array}\right) \right\rangle {\overline{\xi }}_{0}\right) \gamma _{0}(x) \\&\qquad -\left\langle \eta _{k_{*}}^{T},\left( \begin{array}{c} {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{k_{*}}^{(1)}\right] } \\ ~~~ \\ {\left[ {\widetilde{F}}(z, w, \mu ), \beta _{k_{*}}^{(2)}\right] } \end{array}\right) \right\rangle \xi _{k_{*}} \gamma _{k_{*}}(x). \end{aligned}\end{aligned}$$

According to [34], by a recursive transformation, the authors obtain that the normal form for Turing–Hopf bifurcation in system (30) reads as

$$\begin{aligned} \begin{aligned}&\dot{z}=B z \\&\qquad +\left( \begin{array}{c} B_{11} \mu _{1} z_{1}+B_{21} \mu _{2} z_{1} \\ {\overline{B}}_{11} \mu _{1} z_{1}+{\overline{B}}_{21} \mu _{2} z_{1} \\ B_{13} \mu _{1} z_{3}+B_{23} \mu _{2} z_{3} \end{array}\right) \\&\qquad +\left( \begin{array}{c} B_{210} z_{1}^{2} z_{2}+B_{102} z_{1} z_{3}^{2} \\ {\overline{B}}_{210} z_{1} z_{2}^{2}+{\overline{B}}_{102} z_{2} z_{3}^{2} \\ B_{111} z_{1} z_{2} z_{3}+B_{003} z_{3}^{3} \end{array}\right) \\&\qquad +O\left( |z||\mu |^{2}+|z|^{2}|\mu |+|z|^{4}\right) . \end{aligned}\end{aligned}$$
(35)

The normal form (35) can be written in real coordinates \({\widetilde{v}}\) through the change of variables \(z_{1}={\widetilde{v}}_{1}-i {\widetilde{v}}_{2}, z_{2}={\widetilde{v}}_{1}+i {\widetilde{v}}_{2}, z_{3}={\widetilde{v}}_{3}\), and then changing to cylindrical coordinates by \({\widetilde{v}}_{1}=\rho \cos \varTheta , {\widetilde{v}}_{2}=\rho \sin \varTheta , {\widetilde{v}}_{3}=\varsigma \), we obtain, truncating at third-order terms and removing the azimuthal term

$$\begin{aligned} \begin{aligned}&{\dot{\rho }}=\alpha _{1}(\mu ) \rho +\kappa _{11} \rho ^{3}+\kappa _{12} \rho \varsigma ^{2}, \\&{\dot{\varsigma }}=\alpha _{2}(\mu )\varsigma +\kappa _{21} \rho ^{2} \varsigma +\kappa _{22} \varsigma ^{3}, \end{aligned}\end{aligned}$$
(36)

where

$$\begin{aligned}\begin{aligned}&\alpha _{1}(\mu )={\text {Re}}\left( B_{11}\right) \mu _{1}+{\text {Re}}\left( B_{21}\right) \mu _{2}, \\&\alpha _{2}(\mu )=B_{13} \mu _{1}+B_{23} \mu _{2}, \\&\kappa _{11}={\text {Re}}\left( B_{210}\right) ,~\kappa _{12}={\text {Re}}\left( B_{102}\right) ,~\kappa _{21}=B_{111}, \\&\kappa _{22}=B_{003}. \end{aligned}\end{aligned}$$

According to Section 8.6 in [38], we know that the third-order truncated normal form (36) is exactly the same to the third-order truncated normal form of the four-dimensional smooth system depending on two parameters with Hopf–Hopf bifurcation. For the so-called simple case [38], i.e. \(\kappa _{11} \kappa _{22}>0\), the dynamics of system (4) near the bifurcation value is topologically equivalent to that of normal form (36). However, for the “difficult” cases, the original system (4) is never topologically equivalent to the truncated normal form (36). In general, for the “difficult” cases, in order to obtain the whole dynamics of the original (4), five-order or higher-order normal form needs to be calculated.

With the help of MATLAB software, the explicit values of the coefficients \(B_{11}\), \(B_{21}\), \(B_{13}\), \(B_{23}\), \(B_{210}\), \(B_{102}\), \(B_{111}\) and \(B_{003}\) can be obtained for the fixed parameters. Notice that the formulas which are used to calculate the above coefficients are rather complicated and we leave them in “Appendix”.

Table 1 The twelve unfoldings of (36)
Table 2 The corresponding relationship between the equilibrium points of (37) and the solutions of original system (4)

5 Numerical simulations

In this section, the third-order truncated normal form is obtained under the given parameters and some numerical simulations are made to support the results of our theoretical analysis. More precisely, some numerical simulations about the temporal patterns, spatial patterns and spatiotemporal patterns are given.

Let \(\varOmega =(0, \pi )\) and \(\gamma =3, h=0.1, d_{12}=1, d_{21}=10, d_{22}=15\), then the system (4) at least undergoes Turing–Hopf bifurcation at the point (1.1467, 0.4404). By using the above-given parameters, the normal form truncated to the third-order terms is

$$\begin{aligned} \begin{aligned} {\dot{\rho }}&=-1.5571 \mu _{2}\rho -1.0401 \rho ^{3}-1.4138 \rho \varsigma ^{2}, \\ {\dot{\varsigma }}&=(0.9290\mu _{1}-2.5716\mu _{2})\varsigma \\&\quad -3.3790 \rho ^{2} \varsigma -2.4380 \varsigma ^{3}, \end{aligned}\end{aligned}$$
(37)

where \(\mu _{1}\) and \(\mu _{2}\) are perturbation parameters for the Turing–Hopf bifurcation point (1.1467, 0.4404). Notice that \(\rho >0\) and \(\varsigma \) is arbitrary real number. System (37) has a zero equilibrium point \(A_{0}(0,0)\) for any \(\mu _{1}, \mu _{2} \in \mathbb {R}\), three boundary equilibrium points

$$\begin{aligned} \begin{aligned}&A_{1}\left( \sqrt{\frac{-1.5571 \mu _{2}}{1.0401}}, 0\right) ,~~\text { for } -1.5571 \mu _{2}>0, \\&A_{2}^{\pm }\left( 0, \pm \sqrt{\frac{0.9290\mu _{1}-2.5716\mu _{2}}{2.4380}}\right) , \end{aligned} \end{aligned}$$

for \(0.9290\mu _{1}-2.5716\mu _{2}>0\), and two interior equilibrium points

$$\begin{aligned} A_{3}^{\pm }\left( \sqrt{\frac{1.3134\mu _{1}+0.1605\mu _{2}}{2.2415}}, \pm \sqrt{\frac{-0.9663\mu _{1}-2.5867\mu _{2}}{2.2415}}\right) \end{aligned}$$

for \(1.3134\mu _{1}+0.1605\mu _{2}>0\) and \(-0.9663\mu _{1}-2.5867\mu _{2}>0\).

Define the critical bifurcation lines are as follows:

$$\begin{aligned} \begin{aligned}&\mathbf {H_{0}}: \mu _{2}=0, \\&\mathbf {L_{1}}: \mu _{2}=0.3613 \mu _{1}, \\&\mathbf {T_{1}}: \mu _{2}=-8.1832 \mu _{1},~\mu _{1}>0, \\&\mathbf {T_{2}}: \mu _{2}=-0.3736 \mu _{1},~\mu _{1}>0. \end{aligned} \end{aligned}$$

These four lines divide the \(\mu _{1}-\mu _{2}\) parameter plane into six regions marked as \(D_{j},~j=1,2, \ldots , 6\).

Based on Section 7.5 in [39], by the different signs of \(d, b, c, d-b c\) in Table 1, the system (36) has twelve distinct types of unfoldings, which are twelve essentially distinct types of phase portraits and bifurcation diagrams. More precisely, for the system (37), we have \(d=+1,~b=1.3593>0,~c=1.3860>0,~d-bc=-0.8840<0\). That is, the unfolding of the planner system (37) corresponding to the Case \(\mathrm {I}b\) in Table 1. Thus, the phase portraits and bifurcation diagrams corresponding to Case \(\mathrm {I}b\) can be given out, see Fig. 7.5.2 in Section 7.5 in [39] for details.

The dynamics of the original reaction–diffusion system (4) can be determined by the third-order truncated normal form (37) near the neighbourhood of the Turing–Hopf bifurcation point. According to [34], the corresponding relationships between the equilibrium points of plane system (37) and the solutions of original system (4) are shown in Table 2.

Fig. 3
figure 3

Bifurcation diagram of the system (4) near the Turing–Hopf bifurcation point (1.1467, 0.4404)

Fig. 4
figure 4

When \(\left( \mu _{1}, \mu _{2}\right) =(0.60, 0.25)\) lies in region \(D_{1}\) and let \(d_{11}=1.1467, \delta =0.5004\), the positive constant steady state \(E_{*}(0.5615, 0.7386)\) is asymptotically stable. The initial value is \(u(x,0)=0.5615-0.25 \cos (x),~v(x,0)=0.7386+0.25 \cos x\)

Furthermore, by the defined critical bifurcation lines, the bifurcation diagram in the \(\mu _{1}-\mu _{2}\) parameter plane is shown in Fig. 3. The linearized equation of the system (37) at each equilibrium point is

$$\begin{aligned} \frac{d}{{d} t}\left( \begin{array}{cc}\rho \\ \varsigma \end{array}\right) =\left( \begin{array}{ll} C_{1} &{}~~-2.8276 \rho _{i} \varsigma _{i} \\ -6.7580 \rho _{i} \varsigma _{i} &{}~~C_{2} \end{array}\right) \left( \begin{array}{cc}\rho \\ \varsigma \end{array}\right) ,\nonumber \\ \end{aligned}$$
(38)

where

$$\begin{aligned}\begin{aligned} C_{1}&:=-1.5571\mu _{2}-3.1203\rho _{i}^{2}-1.4138 \varsigma _{i}^{2}, \\ C_{2}&:=0.9290\mu _{1}-2.5716\mu _{2}-3.3790\rho _{i}^{2}-7.3140 \varsigma _{i}^{2} \end{aligned}\end{aligned}$$

with \(i=1,2,3,4\) and

$$\begin{aligned}\begin{aligned}&\left( \rho _{1}, \varsigma _{1}\right) =A_{0},~\left( \rho _{2}, \varsigma _{2}\right) =A_{1}, \\&\left( \rho _{3}, \pm \varsigma _{3}\right) =A_{2}^{\pm },~\left( \rho _{4}, \pm \varsigma _{4}\right) =A_{3}^{\pm }. \end{aligned}\end{aligned}$$

More precisely, the coefficient matrices of linearized equation (38) at equilibrium points \(A_{0}, A_{1}, A_{2}^{\pm }, A_{3}^{\pm }\) are

$$\begin{aligned}\begin{aligned}&J(A_{0})=\left( \begin{array}{ll} -1.5571\mu _{2} &{}~~0 \\ 0 &{}~~0.9290\mu _{1}-2.5716\mu _{2} \end{array}\right) , \\&J(A_{1})=\left( \begin{array}{ll} 3.1142\mu _{2} &{}~~0 \\ 0 &{}~~0.9290\mu _{1}+2.4870\mu _{2} \end{array}\right) , \\&J(A_{2}^{\pm }) \\&=\left( \begin{array}{ll} -0.5387 \mu _{1}-0.0658 \mu _{2} &{}~~0 \\ 0 &{}~~-1.8580 \mu _{1}+5.1432 \mu _{2} \end{array}\right) , \\&J(A_{3}^{\pm }) \\&=\left( \begin{array}{ll} -1.2188\mu _{1}-0.1490\mu _{2} &{}~~C_{3} \\ C_{4} &{}~~2.1021\mu _{1}+5.6268\mu _{2} \end{array}\right) , \end{aligned}\end{aligned}$$

respectively, where

$$\begin{aligned}\begin{aligned} C_{3}&:=\mp 2.8276 \sqrt{-0.2526\mu _{1}^{2}-0.0826\mu _{2}^{2}-0.7070\mu _{1}\mu _{2}}, \\ C_{4}&:=\mp 6.7580 \sqrt{-0.2526\mu _{1}^{2}-0.0826\mu _{2}^{2}-0.7070\mu _{1}\mu _{2}}. \end{aligned}\end{aligned}$$

By combining with the bifurcation diagram in Fig. 3 and the linear stability theory, we can analyse the sign of the eigenvalues corresponding to the characteristic equations; thus, the stability and instability of each equilibrium point in regions \(D_{1}\)-\(D_{6}\) can be obtained.

In region \(D_{1}\), the third-order truncated normal form (37) has only one equilibrium point \(A_{0}\) and it is asymptotically stable. This implies that the positive constant steady state \(E_{*}\) of the original system (4) is asymptotically stable, as shown in Fig. 4 for \(\left( \mu _{1}, \mu _{2}\right) =(0.60, 0.25)\) and the initial value \(u(x,0)=0.5615-0.25 \cos (x),~v(x,0)=0.7386+0.25 \cos x\).

In region \(D_{2}\), the third-order truncated normal form (37) has two equilibrium points: \(A_{0}\) and \(A_{1}\). The equilibrium point \(A_{0}\) is unstable and the equilibrium point \(A_{1}\) is asymptotically stable. This means that the system (4) has a stable spatially homogeneous periodic solution. For \(\left( \mu _{1}, \mu _{2}\right) =(-0.28, -0.1)\) and initial value \(u(x, 0)=0.1802+0.01 \cos (x), v(x, 0)=0.4431+0.01 \cos x\), Fig. 5 illustrates this result.

Fig. 5
figure 5

When \(\left( \mu _{1}, \mu _{2}\right) =(-0.28, -0.1)\) lies in region \(D_{2}\) and let \(d_{11}=1.1467, \delta =0.4404\), the positive constant steady state \(E_{*}(0.1802, 0.4431)\) is unstable and there is a stable spatially homogeneous periodic solution. The initial value is \(u(x, 0)=0.1802+0.01 \cos (x), v(x, 0)=0.4431+0.01 \cos x\)

Fig. 6
figure 6

When \(\left( \mu _{1}, \mu _{2}\right) =(-0.16, -0.08)\) lies in region \(D_{3}\) and let \(d_{11}=1.1400, \delta =0.4404\), the positive constant steady state \(E_{*}(0.1961, 0.4729)\) is unstable and there is a heteroclinic orbit connecting the unstable spatially inhomogeneous steady state to stable spatially homogeneous periodic solution. The initial value is \(u(x,0)=0.1961+0.2 \cos x,~v(x,0)=0.4729+0.2 \cos x\)

In region \(D_{3}\), the third-order truncated normal form (37) has four equilibrium points: \(A_{0}, A_{1}, A_{2}^{+}\) and \(A_{2}^{-}\). The equilibrium points \(A_{0}\), \(A_{2}^{+}\) and \(A_{2}^{-}\) are unstable and the equilibrium point \(A_{1}\) is asymptotically stable. Thus, the original system (4) has an unstable positive constant steady state, two unstable spatially inhomogeneous steady states like \(\cos x\)-shape in space, and a stable spatially homogeneous periodic solution. By choosing \(\left( \mu _{1}, \mu _{2}\right) =(-0.16, -0.08)\) and the initial value \(u(x,0)=0.1961+0.2 \cos x,~v(x,0)=0.4729+0.2 \cos x\), the dynamics of the original system (4) evolves from the spatially inhomogeneous steady states to the spatially homogeneous periodic solution, as shown in Fig. 6.

Fig. 7
figure 7

When \(\left( \mu _{1}, \mu _{2}\right) =(0.005,-0.002)\) lies in region \(D_{4}\) and let \(d_{11}=1.1467, \delta =0.4364\), the positive constant steady state \(E_{*}(0.2648, 0.5840)\) is unstable and there is a heteroclinic orbit connecting the spatially inhomogeneous periodic solution to stable spatially homogeneous periodic solution. The initial value is \(u(x,0)=0.2648-0.1 \cos x,~v(x,0)=0.5840-0.15 \cos x\). a and b are transient behaviours for u and v, respectively; c and d are middle-term behaviours for u and v, respectively; e and f are long-term behaviours for u and v, respectively

Fig. 8
figure 8

When \(\left( \mu _{1}, \mu _{2}\right) =(0.23, -0.09)\) lies in region \(D_{4}\) and let \(d_{11}=1.1467, \delta =0.4364\), the positive constant steady state \(E_{*}(0.1881, 0.4581)\) is unstable and there is stable spatially homogeneous periodic solution. The initial value is \(u(x,0)=0.1881-0.02 \cos x,~v(x,0)=0.4581-0.02 \cos x\)

Fig. 9
figure 9

When \(\left( \mu _{1}, \mu _{2}\right) =(0.17, -0.03)\) lies in region \(D_{5}\) and let \(d_{11}=1.1467, \delta =0.3\), the positive constant steady state \(E_{*}(0.2389,0.5455)\) is unstable and there is a heteroclinic orbit connecting the unstable positive constant steady state to stable spatially homogeneous periodic solution. The initial value is \(u(x,0)=0.2389-0.15 \cos x,~v(x,0)=0.5455-0.15 \cos x\). When \(\left( \mu _{1}, \mu _{2}\right) =(0.31,-0.05)\) lies in region \(D_{5}\) and let \(d_{11}=0.9, \delta =0.4004\), the positive constant steady state \(E_{*}(0.2213,0.5169)\) is unstable and there is a heteroclinic orbit connecting the unstable positive constant steady state to stable spatially inhomogeneous steady states. The initial value is \(u(x,0)=0.2213-0.2 \cos x,~v(x,0)=0.5169+0.2 \cos x\)

Fig. 10
figure 10

When \(\left( \mu _{1}, \mu _{2}\right) =(0.75, 0.1675)\) lies in region \(D_{6}\) and let \(d_{11}=0.9, \delta =0.5004\), the positive constant steady state \(E_{*}(0.4514,0.7429)\) is unstable and there are two stable spatially inhomogeneous steady states like \(\cos x\) in space. a and b: the initial value is \(u(x,0)=0.4514-0.15 \cos x,~v(x,0)=0.7429+0.15 \cos x\); c and d: the initial value is \(u(x,0)=0.4514+0.15 \cos x,~v(x,0)=0.7429-0.15 \cos x\)

In region \(D_{4}\), the third-order truncated normal form (37) has six equilibrium points: \(A_{0}, A_{1}, A_{2}^{\pm }\) and \(A_{3}^{\pm }\). The equilibrium point \(A_{0}\) is unstable, the equilibrium points \(A_{1}\), \(A_{2}^{\pm }\) and \(A_{3}^{\pm }\) are asymptotically stable. This implies that the original system (4) has an unstable positive constant steady state, a stable spatially homogeneous periodic solution, two stable spatially inhomogeneous steady states like \(\cos x\)-shape in space and two stable spatially inhomogeneous periodic solution like \(\cos x\)-shape in space. Taking the parameter \(\left( \mu _{1}, \mu _{2}\right) =(0.005,-0.002)\) and the initial value \(u(x,0)=0.2648-0.1 \cos x,~v(x,0)=0.5840-0.15 \cos x\) close to the unstable positive constant steady state, the dynamics of the original system (4) evolves from unstable spatially inhomogeneous steady states, spatially inhomogeneous periodic solution to stable spatially homogeneous periodic solution, as shown in Fig. 7. Furthermore, by taking the parameter \(\left( \mu _{1}, \mu _{2}\right) =(0.23, -0.09)\) and the initial value \(u(x,0)=0.1881-0.02 \cos x,~v(x,0)=0.4581-0.02 \cos x\) close to the unstable positive constant steady state, the dynamics of the original system (4) evolves from unstable spatially inhomogeneous steady states to stable spatially homogeneous periodic solution, as shown in Fig. 8.

In region \(D_{5}\), the third-order truncated normal form has four equilibrium points: \(A_{0}, A_{1}, A_{2}^{+}\) and \(A_{2}^{-}\). The equilibrium point \(A_{0}\) is unstable and the equilibrium points \(A_{1}\), \(A_{2}^{+}\) and \(A_{2}^{-}\) are asymptotically stable. Thus, the original system (4) has an unstable positive constant steady state, a stable spatially homogeneous periodic solution and two stable spatially inhomogeneous steady states like \(\cos x\)-shape in space. Taking the parameter \(\left( \mu _{1}, \mu _{2}\right) =(0.17, -0.03)\) and the initial value \(u(x,0)=0.2389-0.15 \cos x,~v(x,0)=0.5455-0.15 \cos x\) close to the unstable positive constant steady state, the dynamics of the original system (4) evolves from unstable positive constant steady state to stable spatially homogeneous periodic solution, as shown in Fig. 9a and b. Taking the parameter \(\left( \mu _{1}, \mu _{2}\right) =(0.31,-0.05)\) and the initial value \(u(x,0)=0.2213-0.2 \cos x,~v(x,0)=0.5169+0.2 \cos x\) close to the unstable positive constant steady state, the dynamics of the original system (4) evolves from unstable positive constant steady state to stable spatially inhomogeneous steady states, as shown in Fig. 9c and d.

In region \(D_{6}\), the third-order truncated normal form (37) has three equilibrium points: \(A_{0}, A_{2}^{+}\) and \(A_{2}^{-}\). The equilibrium point \(A_{0}\) is unstable and the equilibrium points \(A_{2}^{+}\) and \(A_{2}^{-}\) are asymptotically stable. Thus, the original system (4) has an unstable positive constant steady state and two stable non-constant steady states like \(\cos x\)-shape in space. For the fixed parameter \(\left( \mu _{1}, \mu _{2}\right) =(0.75, 0.1675)\) and choosing different initial values, the original system (4) can converge to one of these two stable non-constant steady states, as shown in Fig. 10a and b for the initial value \(u(x,0)=0.4514-0.15 \cos x,~v(x,0)=0.7429+0.15 \cos x\) and Fig. 10c and d for the initial value \(u(x,0)=0.4514+0.15 \cos x,~v(x,0)=0.7429-0.15 \cos x\).

6 Conclusion and discussion

In this paper, a predator–prey model with cross-diffusion considering the prey individual behaviour and herd behaviour transition with homogeneous Neumann boundary condition is investigated. We first show that the non-negativity and boundedness of solutions corresponding to the model without self-diffusion and cross-diffusion and the model with self-diffusion and cross-diffusion. Then we show the existence and priori boundedness of solutions corresponding to the spatiotemporal model without cross-diffusion. In order to classify the possible dynamical classification near the Turing–Hopf bifurcation point, by using the method of computing the normal form presented in [34], the third-order truncated normal form (37) is given. By the obtained third-order truncated normal form (37), we obtain a zero equilibrium point corresponding to the positive constant steady state of the original system (4), three boundary equilibrium points \(A_{1}\) and \(A_{2}^{\pm }\) in which \(A_{1}\) corresponding to the spatially homogeneous periodic solution of the original system (4) and \(A_{2}^{\pm }\) corresponding to the two spatially inhomogeneous steady states with \(\cos x\)-like shape in space of the original system (4). Furthermore, two interior equilibrium points \(A_{3}^{\pm }\) is also obtained corresponding to the two spatially inhomogeneous periodic solutions with \(\cos x\)-like shape in space of the original system (4). Moreover, we obtain four inequality which is used to ensure the existence of these different types of equilibrium points. Notice that the four inequality can be seen as the four critical bifurcation lines; thus, by the defined critical bifurcation lines, the bifurcation diagram in the \(\mu _{1}\)-\(\mu _{2}\) parameters plane which includes six different regions is shown in Fig. 3.

By the numerical simulations, the rich dynamics such as the positive constant steady state, the spatially homogeneous periodic solution, spatially inhomogeneous steady states and spatially inhomogeneous periodic solutions have been found, which can be seen in Figs. 4, 5, 6, 7, 8, 9 and 10. Especially, we would like to mention that the interaction between the Turing bifurcation curve and the Hopf bifurcation curve may leads to the emergence of the spatially inhomogeneous periodic solutions, see Fig. 7a and b for details.

We have to point out the fact that the method of computing normal form developing in this paper can also be used to the case without cross-diffusion by a slight modification. Furthermore, in order to the simplicity of notation for further normal form computation and for the convenience of carrying out the numerical simulations, we let \(\varOmega :=(0, \pi )\). However, the spatial \(\varOmega \) can also be taken \((0, \ell \pi )\) with \(\ell \in \mathbb {R}^{+}\). Notice also that the general open interval \(({\widetilde{a}},{\widetilde{b}})\) can be transformed to \((0, \pi )\) by a translation and rescaling.