1 Introduction

Competition models, having enormous impacts on various fields including biological, ecological and biochemical processes, enrich modern research to a large extent [1,2,3]. In biology, competition models are widely regarded as the crucial tools to understand the mechanisms leading to biodiversity. During the past years, the competition models derived from interactions of several species have been extensively studied. Among those models, the Lotka-Volterra competition model is considered as the basis of a model reflecting competitive interactions between species [4,5,6,7,8,9]. However, there is a limitation in classical Lotka-Volterra model that has the competitive interaction of two populations: with the increase of one competitor’s density, its competitive capacity will increase and tend to infinity. But in reality, this capacity between different species should be upper-bounded. To overcome this deficiency, several types of models have been proposed. For example, this deficiency can be remedied by the following model with nonlinear functional response:

$$ \left \{ \textstyle\begin{array}{l} \frac{\partial u}{\partial t}=u(a_{1}-b_{1}u-\frac{c_{1}v}{1+v}), \\ \frac{\partial v}{\partial t}=v(a_{2}-b_{2}v-\frac{c_{2}u}{1+u}), \end{array}\displaystyle \right . $$
(1.1)

where \(u\) and \(v\) stand for the densities of two competing species, \(a_{1}\) and \(a_{2}\) refer to their intrinsic growth rates, \(b_{1}\) and \(b_{2}\) account for their logistic growth rates, \(c_{1}\) and \(c_{2}\) are their maximum inter-specific interaction coefficients. Here, all parameters are positive constants. The terms \(\frac{c_{1}uv}{1+v}\) and \(\frac{c_{2}uv}{1+u}\) represent the functional response, their limits \(\lim \limits _{v\rightarrow \infty }\frac{c_{1}uv}{1+v}=c_{1}u\) and \(\lim \limits _{u\rightarrow \infty }\frac{c_{2}uv}{1+u}=c_{2}v\) imply that the competitive capacity of species cannot increase at an infinitely great rate when the density of its competitor increases. In the past few years, the models based on (1.1) have been well studied, and some valuable results have been obtained, see [10, 11] for examples.

In the field of population dynamics, the diffusion phenomenon of different species in the environment is a very universal survival and life style. Therefore, a large number of models of the multi-species interacting populations are described by reaction-diffusion systems. For example, Barabanova [12] studied a reaction-diffusion system with exponential nonlinearity. He discussed the global existence of nonnegative solutions and the asymptotic behavior of global solutions for system. Jia [13] considered a reaction-diffusion population model with predator-prey-dependent functional response. He investigated the conditions which ensure the model has a unique positive constant solution, and studied the dynamical properties of the model, including the large time behaviors of the nonconstant solutions and the local and global asymptotic stability of the positive constant solution. For more detailed backgrounds of reaction-diffusion systems, one can see [4, 14,15,16] and the references therein.

In recent decades, there has been considerable interest in being able to reveal the dynamics of reaction-diffusion models with nonlinear diffusion. Just because of this, the cross-diffusion terms are introduced into a system of reaction-diffusion equations to model the situation that one species influence the movement of another species, which was proposed firstly by Kerner [17] and applied firstly to biological models by Shigesada et al. [18]. Recently, many researchers have devoted to the study of the population models with cross-diffusion from various mathematical viewpoints. For example, in [19], the authors presented a general instability analysis on cross-diffusion system with two species. They showed that cross-diffusion can destabilize a uniform equilibrium which is stable for the kinetic and self-diffusion-reaction systems; On the other hand, cross-diffusion can also stabilize a uniform equilibrium which is stable for the kinetic system but unstable for the self-diffusion-reaction system. Bendahmane [20] discussed a predator-prey model with cross-diffusion. He established the existence of weak and classical solutions for model by means of an approximation system, the Faedo-Galerkin method, and the compactness method. Paper [21] studied three species food chain model with a Holling type-II functional response involving cross-diffusions. The authors presented the equilibrium solutions of the model and proved the stability of positive coexistence equilibrium, and conducted Turing instability induced by cross-diffusion. In [22], the authors investigated the Shigesada-Kawasaki-Teramoto model for two competing species with triangular cross-diffusion. By using the scalar maximum principle and the Hopf boundary point lemma, they determined explicit parameter ranges within which the model exclusively possesses constant steady state solutions. Moreover, there have many valuable surveys on the mathematical developments of cross-diffusion equations arising from various research fields, one can see [6, 23,24,25,26,27,28,29,30,31] and the references therein.

Based on model (1.1), in [28], Li et al. proposed the following model with cross-diffusion

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} \frac{\partial u}{\partial t}-\Delta (d_{11}u+d_{12}v)=u(a_{1}-b_{1}u- \frac{c_{1}v}{1+v}), & (x,t)\in \Omega \times (0,\infty ), \\ \frac{\partial v}{\partial t}-\Delta (d_{21}u+d_{22}v)=v(a_{2}-b_{2}v- \frac{c_{2}u}{1+u}), & (x,t)\in \Omega \times (0,\infty ), \\ \partial _{\textbf{n}}u=\partial _{\textbf{n}}v=0, & (x,t)\in \partial \Omega \times (0,\infty ), \\ u(x,0)=u_{0}(x)>0, v(x,0)=v_{0}(x)>0, & x\in \Omega , t=0, \end{array}\displaystyle \right . $$
(1.2)

where \(\Omega \) is a bounded domain in \(\mathbb{R}^{n}\ (n\geq 1)\) with smooth boundary \(\partial \Omega \), \(\textbf{n}\) is the unit outward normal vector on the boundary \(\partial \Omega \), \(a_{i}\), \(b_{i}\), \(c_{i}\), \(i=1,2\) have the same biological meaning as in model (1.1), \(d_{11}\) and \(d_{22}\) are the self-diffusion coefficients of two species, \(d_{12}\) and \(d_{21}\) denote their cross-diffusion pressures. Here, \(d_{11},d_{12},d_{21},d_{22}\) are positive parameters. In [28], the authors studied the existence and stability of positive equilibrium, and gave the Turing bifurcation critical value and the condition for the occurrence of Turing pattern to model (1.2).

In [18], Shigesada et al. described that the movement of two species in the actual ecological environment is affected by nonlinear diffusion forces. So, it is more realistic to consider the nonlinear diffusion effects to model (1.2). With what in mind, by making appropriate modifications, model (1.2) can be revised as the following form with nonlinear diffusion effects and Dirichlet boundary conditions

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} \frac{\partial u}{\partial t}-\Delta [(\alpha _{1}+\beta _{11}u+ \beta _{12}v)u]=u(a_{1}-b_{1}u-\frac{c_{1}v}{1+v}), & (x,t)\in \Omega \times (0,\infty ), \\ \frac{\partial u}{\partial t}-\Delta [(\alpha _{2}+\beta _{21}u+ \beta _{22}v)v]=v(a_{2}-b_{2}v-\frac{c_{2}u}{1+u}), & (x,t)\in \Omega \times (0,\infty ), \\ u=v=0, & (x,t)\in \partial \Omega \times (0,\infty ), \\ u(x,0)=u_{0}(x)\geq 0,\not \equiv 0, v(x,0)=v_{0}(x)\geq 0,\not \equiv 0, & x\in \Omega , t=0, \end{array}\displaystyle \right . $$
(1.3)

where all parameters are positive constants, the parameters \(a_{i},b_{i},c_{i},i=1,2\) have the same biological meaning as in model (1.1), \(\alpha _{1}\) and \(\alpha _{2}\) are the diffusion rates of two species, \(\beta _{11}\) and \(\beta _{22}\) are their self-diffusion pressures, \(\beta _{12}\) and \(\beta _{21}\) are their cross-diffusion pressures, the nonlinear terms \(\Delta (\alpha _{1}u)\) and \(\Delta (\alpha _{2}v)\) model the situation that the two species move in random ways, and the nonlinear terms \(\Delta [(\beta _{11}u+\beta _{12}v)u]\) and \(\Delta [(\beta _{21}u+\beta _{22}v)v]\) describe that the movement of two species is under influence of population pressure caused by intra- and inter-species interferences, \(u_{0}\) and \(v_{0}\) are continuous functions. Obviously, compared with model (1.2), the model (1.3) is more logical and close to real situations.

In this paper, we focus on the existence of steady state solutions of model (1.3), that is, the existence of classical positive solutions of the following elliptic system

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [(\alpha _{1}+\beta _{11}u+\beta _{12}v)u]=u(a_{1}-b_{1}u- \frac{c_{1}v}{1+v}), & x\in \Omega , \\ -\Delta [(\alpha _{2}+\beta _{21}u+\beta _{22}v)v]=v(a_{2}-b_{2}v- \frac{c_{2}u}{1+u}), & x\in \Omega , \\ u=v=0, & x\in \partial \Omega . \end{array}\displaystyle \right . $$
(1.4)

Since the cross-diffusion terms are introduced, one interesting problem is that whether their increase will affect the possibility of existence of positive solutions for model (1.4) or not. Therefore, the main purpose of this paper is to consider the effects of cross-diffusion pressures on the positive solutions of model (1.4). Our approach to the proof is the fixed point index theory [31]. It should be pointed out that we extend the conclusions in [31] in analyzing the existence of positive solution since the standard conclusion frameworks in [31] are not comprehensive for our model (it only can obtain that system (1.4) admits a positive solution if the signs of the first eigenvalues of suitable operators are the same). More specifically, we not only prove that (1.4) has a positive solution when the signs of the first eigenvalues of suitable operators are the same, but also show that (1.4) has a positive solution when one of these first eigenvalues is less than 0 and one is equal to 0. Due to the cross-diffusion pressures are related to these first eigenvalues, and therefore they affect the existence of positive solution of model. Moreover, we also show that the inter-specific competition pressures are also related to the existence of positive solution of model (1.4).

The rest of this paper is organized as follows. Section 2 states some known results about eigenvalue problem, a scalar equation and the fixed point index theory. In Sect. 3, we briefly discuss the stability of trivial and semi-trivial solutions of (1.4) by spectrum analysis. In Sect. 4, we first give the boundedness of positive solution of (1.4), and then present the sufficient conditions which ensure (1.4) having no positive solution. Using the fixed point index theory, the existence of positive solutions of (1.4) is investigated in Sect. 5. Section 6 gives the conclusion to end the investigation.

2 Preliminaries

In this section, we first consider a certain eigenvalue problem and a scalar equation, and then give some known results for fixed point index theory.

2.1 Eigenvalue Problem

For \(a(x)>0\) in \(C^{2}(\overline{\Omega })\) and \(b(x)\in L^{\infty }(\Omega )\), consider the eigenvalue problem

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} \Delta [a(x)u]+b(x)u=\lambda u, & x\in \Omega , \\ u=0, & x\in \partial \Omega , \end{array}\displaystyle \right . $$
(2.1)

where \(\Omega \) is the same as \(\Omega \) in (1.2). By [31], we obtain that the problem (2.1) has an infinite sequence of eigenvalues \(\{\lambda _{i}(\Delta a(x)+b(x))\}\) such that \(\lambda _{i}(\Delta a(x)+b(x))\geq \lambda _{i+1}(\Delta a(x)+b(x))\) with corresponding eigenfunctions \(\phi _{i},\phi _{i+1},\ldots \), \(i=1,2,\ldots \), where \(\lim \limits _{i\rightarrow \infty }\lambda _{i}(\Delta a(x)+b(x))=- \infty , i\geq 1\).

Denote by \(\|\cdot \|_{L^{2}}\) the usual \(L^{2}\)-norm in \(L^{2}(\Omega )\). From [31] we have

$$ \lambda _{1}(\Delta a(x)+b(x))=\sup _{u\in W^{1,2}(\Omega )} \frac{\int _{\Omega }(-|\nabla [a(x)u]|^{2}+a(x)b(x)u^{2})\mbox{d}x}{\|\sqrt{a(x)}u\|^{2}_{L^{2}}}. $$
(2.2)

Clearly, \(\lambda _{1}(\Delta a(x)+b(x))\) is increasing in \(b(x)\). The following Lemma 2.1 and Lemma 2.2 can also be obtained from [31].

Lemma 2.1

Let \(a(x)>0\) in \(C^{2}(\overline{\Omega }), b(x)\in L^{\infty }(\Omega )\), and \(u\geq 0,\not \equiv 0\) in \(\Omega \) with \(u=0\) on \(\partial \Omega \). Then the following conclusions hold:

  1. (i)

    If \((\Delta a(x)+b(x))u\geq ,\not \equiv 0\), then \(\lambda _{1}(\Delta a(x)+b(x))>0\);

  2. (ii)

    If \((\Delta a(x)+b(x))u\leq ,\not \equiv 0\), then \(\lambda _{1}(\Delta a(x)+b(x))<0\);

  3. (iii)

    If \((\Delta a(x)+b(x))u\equiv 0\), then \(\lambda _{1}(\Delta a(x)+b(x))=0\).

Lemma 2.2

Assume that \(b_{1}(x)/a_{1}(x)>b_{2}(x)/a_{2}(x)\), where \(a_{i}(x)>0\) in \(C^{2}(\overline{\Omega }), b_{i}(x)\in L^{\infty }(\Omega )\) for \(i=1,2\).

  1. (i)

    If \(\lambda _{1}(\Delta a_{1}(x)+b_{1}(x))\leq 0\), then \(\lambda _{1}(\Delta a_{2}(x)+b_{2}(x))<0\);

  2. (ii)

    If \(\lambda _{1}(\Delta a_{2}(x)+b_{2}(x))\geq 0\), then \(\lambda _{1}(\Delta a_{1}(x)+b_{1}(x))>0\).

Let \(T:E\rightarrow E\) be a linear operator on a Banach space \(E\) and denote by \(r(T)\) the spectral radius of \(T\). Then we have the following statements for \(r(T)\), which can be shown by the similar manner in [32, Lemma 2].

Lemma 2.3

Let \(a(x)>0\) in \(C^{2}(\overline{\Omega }), b(x)\in L^{\infty }(\Omega )\), and \(M\) be a positive constant such that \(b(x)+Ma(x)>0\) for all \(x\in \overline{\Omega }\). Then we have

  1. (i)

    If \(\lambda _{1}(\Delta a(x)+b(x))>0\), then \(r[\frac{1}{a(x)}(-\Delta +M)^{-1}(b(x)+Ma(x))]>1\);

  2. (ii)

    If \(\lambda _{1}(\Delta a(x)+b(x))<0\), then \(r[\frac{1}{a(x)}(-\Delta +M)^{-1}(b(x)+Ma(x))]<1\);

  3. (iii)

    If \(\lambda _{1}(\Delta a(x)+b(x))=0\), then \(r[\frac{1}{a(x)}(-\Delta +M)^{-1}(b(x)+Ma(x))]=1\).

2.2 A Scalar Equation

In this subsection, we consider the scalar equation

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [\varphi (u)u]=uf(u), & x\in \Omega , \\ u=0, & x\in \partial \Omega , \end{array}\displaystyle \right . $$
(2.3)

where \(\Omega \) is a bounded domain in \(\mathbb{R}^{n}\ (n\geq 1)\) with smooth boundary \(\partial \Omega \). The functions \(\varphi :[0,\infty )\rightarrow [0,\infty )\) and \(f:[0,\infty )\rightarrow \mathbb{R}\) are assumed to satisfy the following hypotheses:

  1. (H2.1)

    \(\varphi (0)>0\) and \(\varphi (u)\) is \(C^{2}\)-function in \(u\) with \(\varphi '(u)\geq 0\) for all \(u\geq 0\);

  2. (H2.2)

    \(f(u)\) is \(C^{1}\)-function in \(u\) with \(f'(u)<0\) for all \(u\geq 0\);

  3. (H2.3)

    \(f(0)>0\) and \(f(u)<0\) on \((C_{0},\infty )\) for some positive constant \(C_{0}\).

Now we give the existence and uniqueness theorem of positive solutions of (2.3), which can be proved by the similar technique in [31, Theorem 2.11].

Theorem 2.1

Consider the scalar equation (2.3) with hypotheses (H2.1)-(H2.3).

  1. (i)

    If \(\lambda _{1}(\varphi (0)\Delta +f(0))\leq 0\), then (2.3) has no positive solution;

  2. (ii)

    If \(\lambda _{1}(\varphi (0)\Delta +f(0))>0\), then (2.3) has a unique positive solution.

2.3 Fixed Point Index Theory in Banach Space

Let \(E\) be a real Banach space and \(W\subset E\) a closed convex set. \(W\) is called a total wedge if \(\alpha W\subset W\) for all \(\alpha \geq 0\) and \(\overline{W-W}=E\). A wedge is said to be a cone if \(W\cap (-W)=\{0\}\). For \(y\in W\), define \(W_{y}=\{x\in E: y+\gamma x\in W\ \mbox{for some}\ \gamma >0\}, S_{y}= \{x\in \overline{W}_{y}: -x\in \overline{W}_{y}\}\). Then \(\overline{W}_{y}\) is a wedge containing \(W,y,-y\), and \(S_{y}\) is a closed subspace of \(E\) containing \(y\).

Let \(T\) be a compact linear operator on \(E\) which satisfies \(T(\overline{W}_{y})\subset \overline{W}_{y}\). We say that \(T\) has property \(\alpha \) on \(\overline{W}_{y}\) if there is \(t\in (0,1)\) and \(w\in \overline{W}_{y}\setminus S_{y}\) such that \(w-tTw\in S_{y}\). Let \(F: W\rightarrow W\) is a compact operator with a fixed point \(y\in W\) and \(F\) is Fréchet differentiable at \(y\). Let \(L=F'(y)\) be the Fréchet derivative of \(F\) at \(y\). Then \(L\) maps \(\overline{W}_{y}\) into itself. For an open subset \(U\subset W\), define \(\mbox{index}_{W}(F,U)=\mbox{index}(F,U,W)=\mbox{index}_{W}(I-F,U,0)\), where \(I\) is the identity map. If \(y\) is an isolated fixed point of \(F\), then the fixed point index of \(F\) at \(y\) in \(W\) is defined by \(\mbox{index}_{W}(F,y)=\mbox{index}(F,y,W)=\mbox{index}_{W}(F,U(y),W)\), where \(U(y)\) is a small open neighborhood of \(y\) in \(W\).

Next, we represent two results that will be useful in proving the existence of positive solutions of system (1.4). Theorem 2.2 is due to Li [33], and Theorem 2.3 is due to Amann [34].

Theorem 2.2

Assume that \(I-L\) is invertible on \(\overline{W}_{y}\).

  1. (i)

    If \(L\) has property \(\alpha \) on \(\overline{W}_{y}\), then \(\textit{index}_{W}(F,y)=0\).

  2. (ii)

    If \(L\) does not have property \(\alpha \) on \(\overline{W}_{y}\), then \(\textit{index}_{W}(F,y)=(-1)^{\gamma }\), where \(\gamma \) is the sum of multiplicities of all the eigenvalues of \(L\) which are greater than 1.

Suppose that \(B\) is an open unit ball of \(E\), \(V\) is a real vector space, \(P\) is a nonempty subset of \(V\). Denote by \((E,P)\) an arbitrary ordered Banach space. For every \(\rho >0\), denote \(P_{\rho }=\rho B\cap P\). Then we have the following statements.

Theorem 2.3

Let \(F: \bar{P}_{\rho }\rightarrow P\) be a compact map such that \(F(0)=0\). Suppose that \(F\) has a right derivative \(F'_{+}(0)\) at zero such that 1 is not an eigenvalue of \(F'_{+}(0)\) to a positive eigenvector. Then there exists a constant \(\sigma _{0}\in (0,\rho ]\) such that for every \(\sigma \in (0,\sigma _{0}]\),

  1. (i)

    if \(F'_{+}(0)\) has no positive eigenvector to an eigenvalue greater than one, then \(\textit{index}_{W}(F, P_{\sigma })=1\).

  2. (ii)

    if \(F'_{+}(0)\) possesses a positive eigenvector to an eigenvalue greater than one, then \(\textit{index}_{W}(F, P_{\sigma })=0\).

3 Stability of Trivial and Semi-Trivial Solutions

This section focuses on the stability of trivial and semi-trivial solutions of model (1.4). The arguments are based on the spectrum analysis of the linearized operators.

Clearly, model (1.4) has a trivial solution \((0,0)\). With Theorem 2.1, we know that the problem

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [(\alpha _{1}+\beta _{11}u)u]=u(a_{1}-b_{1}u), & x\in \Omega , \\ u=0, & x\in \partial \Omega \end{array}\displaystyle \right . $$

has a unique positive solution \(u^{\ast }\) for \(\lambda _{1}(\alpha _{1}\Delta +a_{1})>0\). Thus (1.4) has semi-trivial solution \((u^{\ast },0)\) when \(\lambda _{1}(\alpha _{1}\Delta +a_{1})>0\).

Similarly, (1.4) has semi-trivial solution \((0,v^{\ast })\) when \(\lambda _{1}(\alpha _{2}\Delta +a_{2})>0\).

Theorem 3.1

The solution \((0,0)\) is asymptotically stable if \(\lambda _{1}(\alpha _{1}\Delta +a_{1})<0\) and \(\lambda _{1}(\alpha _{2}\Delta +a_{2})<0\), whereas it is unstable if \(\lambda _{1}(\alpha _{1}\Delta +a_{1})>0\) or \(\lambda _{1}(\alpha _{2}\Delta +a_{2})>0\).

Proof

The linearized operator of (1.4) at \((0,0)\) is

$$ G_{1}=\left [ \textstyle\begin{array}{c@{\quad }c} \alpha _{1}\Delta +a_{1} & 0 \\ 0 & \alpha _{2}\Delta +a_{2} \end{array}\displaystyle \right ]. $$

In view of [35], we know that all eigenvalues of \(G_{1}\) are \(\{\lambda _{i}(\alpha _{1}\Delta +a_{1})\}\cup \{\lambda _{i}( \alpha _{2}\Delta +a_{2})\},i=1,2,\ldots {}\). Thus the conclusion can be obtained directly by spectral analysis. □

Theorem 3.2

(i) The solution \((u^{\ast },0)\) is asymptotically stable if \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})<0\), whereas it is unstable if \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})>0\).

(ii) The solution \((0,v^{\ast })\) is asymptotically stable if \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})<0\), whereas it is unstable if \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})>0\).

Proof

We only prove the (i) since we can make a similar argument for (ii). The linearized operator of (1.4) at \((u^{\ast },0)\) is

$$ G_{2}=\left [ \textstyle\begin{array}{c@{\quad }c} \Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{\ast } & \beta _{12}\Delta u^{\ast }-c_{1}u^{\ast } \\ 0 & \Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }} \end{array}\displaystyle \right ]. $$

Similarly, by [35] we derive that all eigenvalues of \(G_{2}\) are \(\{\lambda _{i}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })\}\cup \{\lambda _{i}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\},i=1,2,\ldots {}\). Clearly, \(\lambda _{i}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })\leq \lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })<0\) for any \(i\geq 1\). Combining the spectral analysis, we can obtain the conclusion directly. □

4 Boundedness and Non-existence of Positive Solutions

This section deals with the boundedness and non-existence of positive solutions of (1.4), which play a critical role in proving the existence result in Sect. 5.

Denote

$$\begin{gathered} \varphi (u,v)=\alpha _{1}+\beta _{11}u+\beta _{12}v,\quad \psi (u,v)= \alpha _{2}+\beta _{21}u+\beta _{22}v, \\ f(u,v)=a_{1}-b_{1}u-\frac{c_{1}v}{1+v},\quad g(u,v)=a_{2}-b_{2}v- \frac{c_{2}u}{1+u}. \end{gathered}$$

Then model (1.4) becomes

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [\varphi (u,v)u]=uf(u,v), & x\in \Omega , \\ -\Delta [\psi (u,v)v]=vg(u,v), & x\in \Omega , \\ u=v=0, & x\in \partial \Omega . \end{array}\displaystyle \right . $$
(4.1)

The following lemma is useful in the calculation of the priori upper bound of positive solution to model (1.4), which is an immediate result of the proof of Lemma 3.2 in [31].

Lemma 4.1

Let \((u,v)\) be a positive solution of (4.1). If \(\varphi (u,v)u\) and \(\psi (u,v)v\) attain their maximum at \(x=x_{0}\) and \(x=x_{1}\) over \(\overline{\Omega }\), respectively, then \(f(u(x_{0}),v(x_{0}))\geq 0\) and \(g(u(x_{1}),v(x_{1}))\geq 0\).

Theorem 4.1

Suppose \(c_{1}\geq a_{1}\), \(c_{2}\geq a_{2}\). Then there exist constants \(M_{1},M_{2}>0\) such that every positive classical solution \((u,v)\) of (1.4) satisfies

$$ u(x)\leq M_{1}, \quad v(x)\leq M_{2}. $$

Proof

Assume that \((u, v)\) is a positive solution of (1.4). Let \(x_{0}\) be a point such that \(\varphi (u(x_{0}),v(x_{0}))u(x_{0})=\max \limits _{x\in \overline{\Omega }}\{\varphi (u,v)u\}\). Applying Lemma 4.1 to the first equation of model (4.1), we have \(f(u(x_{0}),v(x_{0}))\geq 0\). Since \(f_{u},f_{v}<0\), we have

$$ f(u(x_{0}),0))\geq f(u(x_{0}),v(x_{0}))\geq 0\quad \mbox{and}\quad f(0,v(x_{0}))) \geq f(u(x_{0}),v(x_{0}))\geq 0. $$

This means that \(u(x_{0})\leq \frac{a_{1}}{b_{1}}\) and \(v(x_{0})\leq \frac{a_{1}}{c_{1}-a_{1}}\) by given condition \(c_{1}\geq a_{1}\). With \(\varphi _{u},\varphi _{v}>0\), we have

$$ \max \limits _{x\in \overline{\Omega }}\{\varphi (u,v)u\}\leq \varphi ( \frac{a_{1}}{b_{1}},\frac{a_{1}}{c_{1}-a_{1}})\frac{a_{1}}{b_{1}} = \frac{a_{1}(c_{1}-a_{1})(\alpha _{1}b_{1}+a_{1}\beta _{11})+a^{2}_{1}b_{1}\beta _{12}}{b^{2}_{1}(c_{1}-a_{1})}, $$

and so

$$ u(x)\leq \frac{1}{\alpha _{1}}\varphi (u,v)u\leq \frac{1}{\alpha _{1}}\max \limits _{x\in \overline{\Omega }}\{\varphi (u,v)u \}\leq \frac{a_{1}(c_{1}-a_{1}) (\alpha _{1}b_{1}+a_{1}\beta _{11})+a^{2}_{1}b_{1}\beta _{12}}{\alpha _{1}b^{2}_{1}(c_{1}-a_{1})} \triangleq M_{1} $$

for all \(x\in \overline{\Omega }\).

By the similar reason, we can show that there exists a positive constant \(M_{2}\) such that \(v(x)\leq M_{2}\) for all \(x\in \overline{\Omega }\) when \(c_{2}\geq a_{2}\), where

$$ M_{2}= \frac{a_{2}(c_{2}-a_{2})(\alpha _{2}b_{2}+a_{2}\beta _{22})+a^{2}_{2}b_{2}\beta _{21}}{\alpha _{2}b^{2}_{2}(c_{2}-a_{2})}. $$

Thus, the proof is finished. □

In the following, we give the non-existence of positive solutions for (1.4). By Sect. 2.1, we know that when \(a(x)=1\) and \(b(x)=0\) in (2.1), \(\lambda _{i}(\Delta a(x)+b(x))\) is the eigenvalue of \(\Delta \), denoted as \(\lambda _{i}\). In this case, the principle eigenvalue of \(\Delta \) in \(\Omega \) with the homogeneous Dirichlet boundary condition is \(\lambda _{1}\), which will be used many times in the later. In addition, for any \(\varphi \in L^{1}(\Omega )\), we let \(\overline{\varphi }=\frac{1}{|\Omega |}\int _{\Omega }\varphi \mbox{d}x\).

Theorem 4.2

Suppose that \(A_{i}, D_{i}, i=1,2\) are given positive constants. Then there exist positive constants \(D^{0}_{1},D^{0}_{2}\) such that model (1.4) has no positive solution if

$$ \alpha _{1}>D_{1}^{0},\ \alpha _{2}>D_{2}^{0},\ D_{1}\leq \beta _{12}< A_{1}, \ D_{2}\leq \beta _{21}< A_{2}. $$

Proof

Assume, on the contrary, that \((u,v)\) is a positive solution of (1.4). Multiply the first equation of (1.4) by \((u-\overline{u})\) and the second equation by \((v-\overline{v})\), then integrate over \(\Omega \) by parts, and then add them together to yield

$$\begin{aligned} &-\int _{\Omega }\Delta [(\alpha _{1}+\beta _{11}u+\beta _{12}v)u](u- \overline{u})\mbox{d}x-\int _{\Omega }\Delta [(\alpha _{2}+\beta _{21}u+ \beta _{22}v)v](v-\overline{v})\mbox{d}x \\ =&\int _{\Omega }\{2\beta _{11}u|\nabla u|^{2}+\beta _{12}u\nabla u \nabla v+\alpha _{1}|\nabla u|^{2}+\beta _{12}v|\nabla u|^{2}+2\beta _{22}v| \nabla v|^{2}+\beta _{21}v\nabla u\nabla v \\ &+\alpha _{2}|\nabla v|^{2}+\beta _{21}u|\nabla v|^{2}\}\mbox{d}x \\ \\ =&\int _{\Omega }u(a_{1}-b_{1}u-\frac{c_{1}v}{1+v})(u-\bar{u})\mbox{d}x+ \int _{\Omega }v(a_{2}-b_{2}v- \frac{c_{2}u}{1+u})(v-\overline{v}) \mbox{d}x \\ =&\int _{\Omega }\{[a_{1}-b_{1}(u+\overline{u})- \frac{c_{1}\overline{v} +c_{1}v\overline{v}}{(1+v)(1+\overline{v})}](u-\overline{u})^{2}- \frac{c_{1}u(v-\overline{v}) (u-\overline{u})}{(1+v)(1+\overline{v})}\}\mbox{d}x \\ &+\int _{\Omega }\{[a_{2}-b_{2}(v+\overline{v})- \frac{c_{2}\overline{u} +c_{2}u\overline{u}}{(1+u)(1+\overline{u})}](v-\overline{v})^{2}- \frac{c_{2}v(u-\overline{u})(v- \overline{v})}{(1+u)(1+\overline{u})}\}\mbox{d}x. \end{aligned}$$
(4.2)

With (4.2), Young’s inequality, Theorem 4.1 and the given conditions \(\beta _{12}< A_{1}, \beta _{21}< A_{2}\), we have

$$\begin{aligned} &\int _{\Omega }\{\alpha _{1}|\nabla u|^{2}+\alpha _{2}|\nabla v|^{2} \}\mbox{d}x \\ =&\int _{\Omega }\{2\beta _{11}u|\nabla u|^{2}+\beta _{12}u\nabla u \nabla v+\alpha _{1}|\nabla u|^{2}+\beta _{12}v|\nabla u|^{2}+2\beta _{22}v| \nabla v|^{2}+\beta _{21}v\nabla u\nabla v \\ &+\alpha _{2}|\nabla v|^{2}+\beta _{21}u|\nabla v|^{2}\}\mbox{d}x- \int _{\Omega }\{2\beta _{11}u|\nabla u|^{2}+\beta _{12}u\nabla u \nabla v+\beta _{12}v|\nabla u|^{2}+2\beta _{22}v|\nabla v|^{2} \\ &+\beta _{21}v\nabla u\nabla v+\beta _{21}u|\nabla v|^{2}\}\mbox{d}x \\ \leq &\int _{\Omega }\{2\beta _{11}u|\nabla u|^{2}+\beta _{12}u\nabla u \nabla v+\alpha _{1}|\nabla u|^{2}+\beta _{12}v|\nabla u|^{2}+2\beta _{22}v| \nabla v|^{2}+\beta _{21}v\nabla u\nabla v \\ &+\alpha _{2}|\nabla v|^{2}+\beta _{21}u|\nabla v|^{2}\}\mbox{d}x- \int _{\Omega }\{\beta _{12}u\nabla u\nabla v+\beta _{21}v\nabla u \nabla v\}\mbox{d}x \\ =&\int _{\Omega }\{[a_{1}-b_{1}(u+\overline{u})- \frac{c_{1}\overline{v} +c_{1}v\overline{v}}{(1+v)(1+\overline{v})}](u-\overline{u})^{2}- \frac{c_{1}u(v-\overline{v}) (u-\overline{u})}{(1+v)(1+\overline{v})}\}\mbox{d}x \\ &+\int _{\Omega }\{[a_{2}-b_{2}(v+ \overline{v}) -\frac{c_{2}\overline{u}+c_{2}u\overline{u}}{(1+u)(1+\overline{u})}](v- \overline{v})^{2}- \frac{c_{2}v (u-\overline{u})(v-\overline{v})}{(1+u) (1+\overline{u})}\}\mbox{d}x\\ &-\int _{\Omega }\{\beta _{12}u\nabla u \nabla v+\beta _{21}v\nabla u\nabla v\}\mbox{d}x \\ \leq &\int _{\Omega }[a_{1}+(\frac{c_{1}u}{(1+v)(1+\overline{v})})^{2}/ (4\varepsilon _{1})+\varepsilon _{2}](u-\overline{u})^{2}\mbox{d}x \\ &+\int _{\Omega }[a_{2}+(\frac{c_{2}v}{(1+u) (1+\overline{u})})^{2}/(4\varepsilon _{2})+\varepsilon _{1}]\cdot (v- \overline{v})^{2}\mbox{d}x \\ &+\int _{\Omega }[\frac{\beta _{12}^{2}u^{2}}{4\varrho _{1}}|\nabla u|^{2}+ \varrho _{1}|\nabla v|^{2}]\mbox{d}x+\int _{\Omega }[\varrho _{2}| \nabla u|^{2}+\frac{\beta _{21}^{2}v^{2}}{4\varrho _{2}}|\nabla v|^{2}] \mbox{d}x \\ \leq &\int _{\Omega }[(a_{1}+C(\varepsilon _{1})+\varepsilon _{2})(u- \overline{u})^{2}+ (a_{2}+C(\varepsilon _{2})+\varepsilon _{1})(v- \overline{v})^{2}+(\frac{A_{1}^{2}M_{1}^{2}}{4\varrho _{1}}+\varrho _{2})| \nabla u|^{2} \\ &+(\varrho _{1}+\frac{A_{2}^{2}M_{2}^{2}}{4\varrho _{2}})|\nabla v|^{2}] \mbox{d}x, \end{aligned}$$

where, \(C(\varepsilon _{i})=C(\varepsilon _{i})(M_{1},M_{2}), \varepsilon _{i}>0, \varrho _{i}>0, i=1,2, u\leq M_{1}, v\leq M_{2}\), \(M_{1}, M_{2}\) are the same as defined in Theorem 4.1.

It follows from the Poincaré inequality that

$$ \begin{aligned} \int _{\Omega }(\alpha _{1}|\nabla u|^{2}+\alpha _{2}|\nabla v|^{2}) \mbox{d}x \leq &\int _{\Omega }[(- \frac{a_{1}+C(\varepsilon _{1})+\varepsilon _{2}}{\lambda _{1}}+ \frac{A_{1}^{2}M_{1}^{2}}{4\varrho _{1}}+\varrho _{2})|\nabla u|^{2} \\ &+(-\frac{a_{2}+C(\varepsilon _{2})+\varepsilon _{1}}{\lambda _{1}}+ \varrho _{1}+\frac{A_{2}^{2}M_{2}^{2}}{4\varrho _{2}})|\nabla v|^{2}]\mbox{d}x. \end{aligned} $$
(4.3)

We choose \(D^{0}_{1}\) and \(D^{0}_{2}\) satisfying

$$ D^{0}_{1}\geq - \frac{a_{1}+C(\varepsilon _{1})+\varepsilon _{2}}{\lambda _{1}}+ \frac{A_{1}^{2}M_{1}^{2}}{4\varrho _{1}}+\varrho _{2} \quad \mbox{and} \quad D^{0}_{2}\geq - \frac{a_{2}+C(\varepsilon _{2})+\varepsilon _{1}}{\lambda _{1}}+ \varrho _{1}+ \frac{A_{2}^{2}M_{2}^{2}}{4\varrho _{2}}, $$

respectively, then (4.3) is a contradiction with assumptions \(\alpha _{1}>D_{1}^{0}, \alpha _{2}>D_{2}^{0}\). Thus, when \(\alpha _{1}>D_{1}^{0}, \alpha _{2}>D_{2}^{0}, D_{1}\leq \beta _{12}< A_{1}, D_{2}\leq \beta _{21}< A_{2}\), model (1.4) has no positive solution. □

5 Existence of the Positive Solutions

In this section, we investigate the existence of positive solutions of (1.4) by calculating the fixed point’s index. We assume that the following hypothesis always hold.

  1. (H5.1)

    \(\lambda _{1}(\alpha _{1}\Delta +a_{1})>0\) and \(\lambda _{1}(\alpha _{2}\Delta +a_{2})>0\).

Obviously, the hypothesis (H5.1) shows that the model (1.4) has semi-trivial solutions \((u^{*},0)\) and \((0,v^{*})\).

Let \(G(u,v)=(\varphi (u,v)u,\psi (u,v)v), S=(S_{1},S_{2})\), where

$$\begin{gathered} S_{1}(u,v)=u(a_{1}-b_{1}u-\frac{c_{1}v}{1+v}+M(\alpha _{1}+\beta _{11}u+ \beta _{12}v)), \\ S_{2}(u,v)=v(a_{2}-b_{2}v-\frac{c_{2}u}{1+u}+M(\alpha _{2}+\beta _{21}u+ \beta _{22}v)) \end{gathered}$$

with \(M\) being a sufficiently large positive constant so that \(S_{1}\) is monotone increasing with respect to \(u\) and \(S_{2}\) is monotone increasing with respect to \(v\) for all \((u,v)\in [0,M_{1}]\times [0,M_{2}]\). The existence of \(M\) follows from \(\alpha _{1}>0\) and \(\alpha _{2}>0\).

Since the Jacobian determinant \(J=\frac{\partial G(u,v)}{\partial (u,v)}\) satisfies

$$\begin{aligned} J=\frac{\partial G(u,v)}{\partial (u,v)}&=\left | \textstyle\begin{array}{c@{\quad }c} \alpha _{1}+2\beta _{11}u+\beta _{12}v & \beta _{12}u \\ \beta _{21}v & \alpha _{2}+\beta _{21}u+2\beta _{22}v \end{array}\displaystyle \right | \\ &=(\alpha _{1}+2\beta _{11}u+\beta _{12}v)(\alpha _{2}+\beta _{21}u+2 \beta _{22}v)-\beta _{12}\beta _{21}uv>0, \end{aligned}$$

G is invertible and denote the inverse of \(G\) by \(G^{-1}\). Define operator \(H: C(\overline{\Omega })\times C(\overline{\Omega })\rightarrow C( \overline{\Omega })\times C(\overline{\Omega })\) by \(H(u,v)=((-\Delta +M)^{-1}S_{1}(u,v), (-\Delta +M)^{-1}S_{2}(u,v))\). Then \(H\) is compact. Simple calculation gives that \((u,v)\) is a solution of (1.4) is equivalent to \((u,v)\) satisfies \((u,v)=(G^{-1}\circ H)(u,v)\). Denote \(F= G^{-1}\circ H\) throughout this section.

We introduce the following notations:

  • \(C_{0}(\overline{\Omega }):=\{u\in C(\overline{\Omega }): u=0\ \mbox{on}\ \partial \Omega \},\quad E:=C_{0}(\overline{\Omega })\oplus C_{0}( \overline{\Omega })\),

  • \(D:=\{(u,v)\in C_{0}(\overline{\Omega })\oplus C_{0}(\overline{\Omega }): u\leq M_{1}+1, v\leq M_{2}+1\}\),

  • \(K:=\{u\in C_{0}(\overline{\Omega }): 0\leq u(x), x\in \overline{\Omega }\},\quad W:=K\oplus K\),

  • \(Q_{\rho '}:= \{(u,v)\in W: u\leq \rho ', v\leq \rho ', \rho '=\max \{M_{1},M_{2}\}+\varepsilon ,\varepsilon >0\}\),

  • \(D':=\{(\mbox{int}D)\cap W\}\) for \(\rho '>0\).

Note that \(D'\) is open in \(W\) and every positive solution of (1.4) is a fixed point of \(F\) in \(D'\). To show that model (1.4) has a strictly positive solution \((u, v)\), we prove that \(F\) has a nontrivial fixed point in \(D'\).

Let

$$ h_{1}=\frac{1}{\alpha _{1}}(a_{1}-\frac{c_{1}v^{\ast }}{1+v^{\ast }})- \frac{a_{1}-b_{1}u^{\ast }}{\alpha _{1}+\beta _{11}u^{\ast }}, \quad h_{2}= \frac{1}{\alpha _{2}}(a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }})- \frac{a_{2}-b_{2}v^{\ast }}{\alpha _{2}+\beta _{22}v^{\ast }}. $$

Now we state the existence theorem of positive solutions to model (1.4), which will be proved in the later.

Theorem 5.1

(i) If the first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) have the same signs (i.e., if both of them are either positive or negative or zero), or one of these first eigenvalues is less than 0 and one is equal to 0, then model (1.4) has a positive solution.

(ii) If \(\beta _{12}=\beta _{21}=0\) and both \(h_{1}\), \(h_{2}\) have the same constant sign, or one of \(h_{1}\) and \(h_{2}\) is less than 0 and one is equal to 0 on \((0,\overline{M})\), where \(\overline{M}=\max \{\frac{a_{1}}{b_{1}}, \frac{a_{2}}{b_{2}}\}\), then the conditions in (i) are necessary and sufficient for the existence of positive solutions to model (1.4).

Remark 5.1

By the formula of the first eigenvalue (i.e., (2.2)), we know that \(\beta _{12}, \beta _{21}\) influence the signs of \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\). So the changes of \(\beta _{12}, \beta _{21}\) can lead to all the cases in Theorem 5.1 arising, and therefore the cross-diffusion pressures affect the existence of positive solution of model (1.4). In Theorem 5.1(i), we give a sufficient condition for the existence of positive solutions of model (1.4). Biologically, this implies that the two competition species \(u\) and \(v\) can coexist, and the cross-diffusion pressures have an important effect on the coexistence of the two species. Theorem 5.1(ii) shows a necessary and sufficient condition for the existence of positive solutions of model (1.4) under \(\beta _{12}=\beta _{21}=0\). Biologically, this means that the coexistence of two competition species \(u\) and \(v\) is not affected by the cross-diffusion pressures when their self-diffusion, inter-specific competition pressures and growth rates meet certain conditions.

In order to complete the proof of Theorem 5.1, we first give the following Lemmas 5.1-5.4.

Lemma 5.1

\(\textit{Index}_{W}(F,D')=1\).

Proof

Clearly, \(\partial D\) contains no fixed points of \(F\). Thus \(\mbox{index}_{W}(F,D')\) is well-defined. Define an operator \(F_{\mu }\) by \(G^{-1}\circ H_{\mu }\) for \(\mu \in [0,1]\), where

$$\begin{gathered} H_{\mu }(u,v)=((-\Delta +M)^{-1}S_{1,\mu }(u,v), (-\Delta +M)^{-1}S_{2, \mu }(u,v)), \\ S_{1,\mu }=u(\mu (a_{1}-b_{1}u-\frac{c_{1}v}{1+v})+M(\alpha _{1}+ \beta _{11}u+\beta _{12}v)), \\ S_{2,\mu }=v(\mu (a_{2}-b_{2}v-\frac{c_{2}u}{1+u})+M(\alpha _{2}+ \beta _{21}u+\beta _{22}v)). \end{gathered}$$

Then clearly \(F=F_{1}\) and, for each \(\mu \), \((u,v)\) is the fixed point of \(F_{\mu }\) if and only if \((u,v)\) is the solution of the following problem

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [(\alpha _{1}+\beta _{11}u+\beta _{12}v)u]=\mu u(a_{1}-b_{1}u- \frac{c_{1}v}{1+v}), & x\in \Omega , \\ -\Delta [(\alpha _{2}+\beta _{21}u+\beta _{22}v)v]=\mu v(a_{2}-b_{2}v- \frac{c_{2}u}{1+u}), & x\in \Omega , \\ u=v=0, & x\in \partial \Omega . \end{array}\displaystyle \right . $$
(5.1)

As in Theorem 4.1, we can see that every fixed point of \(F_{\mu }\) satisfies \(u(x)\leq M_{1}\) and \(v(x)\leq M_{2}\) in \(\overline{\Omega }\) for each \(\mu \in [0,1]\), and so every fixed point of \(F_{\mu }\) is in \(D\) but not on \(\partial D\). Further, the homotopy invariance property of degree shows that \(\mbox{index}_{W}(F_{\mu },D')\) is independent of \(\mu \). So

$$ \mbox{index}_{W}(F,D')=\mbox{index}_{W}(F_{1},D')=\mbox{index}_{W}(F_{0},D'). $$

Noting that if \(\mu =0\), then (5.1) has only the trivial solution \((0,0)\), we get \(\mbox{index}_{W}(F_{0},D')=\mbox{index}_{W}(F_{0},(0,0))\). Moreover, by the definition of \(\lambda _{1}\), we know that \(\alpha _{1}\lambda _{1}<0\) and \(\alpha _{2}\lambda _{1}<0\).

For the point \(y=(0,0)\), we observe that \(\overline{W}_{y}=K\oplus K\), \(S_{y}=\{0\}\oplus \{0\}\), and \(\overline{W}_{y}\backslash S_{y}=(K\oplus K)\backslash \{(0,0)\}\). Set \(L_{1}=F'_{0}(0,0)\). Assume that \(\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )\) is an eigenfunction of \(L_{1}\) corresponding to some eigenvalue \(\lambda \geq 1\). Then we have

$$ \left \{ \textstyle\begin{array}{l} (-\Delta +M)^{-1} (M\alpha _{1}\xi )=\alpha _{1}(\lambda \xi ), \\ (-\Delta +M)^{-1} (M\alpha _{2}\eta )=\alpha _{2}(\lambda \eta ). \end{array}\displaystyle \right . $$

If \(\eta \not \equiv 0\), then we have \(r(\frac{1}{\alpha _{2}}(-\Delta +M)^{-1}(M\alpha _{2}))<1\) by \(\alpha _{2}\lambda _{1}<0\) and Lemma 2.3(ii), which contradicts \(\lambda \geq 1\). So \(\eta \equiv 0\). Similarly, we can derive \(\xi \equiv 0\) by \(\alpha _{1}\lambda _{1}<0\) and Lemma 2.3(ii). This implies that \(I-L_{1}\) is invertible on \(\overline{W}_{y}\) and \(L_{1}\) does not have an eigenvalue which is greater than or equal to one.

Now we suppose that \(L_{1}\) has property \(\alpha \) on \(\overline{W}_{y}\). Then there exist \(0 < t < 1\) and \((\phi ^{*}_{1},\phi ^{*}_{2})\in \overline{W}_{y}\backslash S_{y}\) such that \((I-tL_{1})\left ( \begin{array}{c} \phi ^{*}_{1} \\ \phi ^{*}_{2} \end{array} \right )\in S_{y}\). So we get

$$ \phi ^{*}_{2}-\frac{t}{\alpha _{2}}(-\Delta +M)^{-1}(\alpha _{2}M) \phi ^{*}_{2}=0. $$

Since \(\phi ^{*}_{2}\in K \backslash \{0\}\), we may conclude that \(\frac{1}{t}>1\) is an eigenvalue of \(\frac{1}{\alpha _{2}}(-\Delta +M)^{-1}(M\alpha _{2})\), which contradicts the above conclusion. This shows that \(L_{1}\) does not have property \(\alpha \) on \(\overline{W}_{y}\). Then we conclude that \(\mbox{index}_{W}(F_{0},(0,0))=1\) by Theorem 2.2(ii). Therefore, \(\mbox{index}_{W}(F,D')=1\). □

Lemma 5.2

\(\textit{Index}_{W}(F,(0,0))=0\).

Proof

Clearly, \(F(0,0)=(0,0)\) and \(F\) is compact in \(Q_{\rho '}\). Let \(L_{2}=F'(0,0)\), where \(F'(0,0)\) is the Fréchet derivative of \(F\) at \((0, 0)\). Then by calculation, we have

$$\begin{aligned} L_{2}\left ( \textstyle\begin{array}{c} \xi \\ \eta \end{array}\displaystyle \right )=\left ( \textstyle\begin{array}{c} \frac{1}{\alpha _{1}}(-\Delta +M)^{-1}[(a_{1}+M\alpha _{1})\xi ] \\ \frac{1}{\alpha _{2}}(-\Delta +M)^{-1}[(a_{2}+M\alpha _{2})\eta ] \end{array}\displaystyle \right ) \end{aligned}$$

for each \((\xi ,\eta )\in E\).

We first show that 1 is not an eigenvalue of \(L_{2}\) corresponding to a positive eigenfunction \(\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )\). Assume that \(L_{2}\) has an eigenvalue 1, i.e., \(L_{2}\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )=\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )\). This can be written as follows

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta (\alpha _{1}\xi )=a_{1}\xi , & x\in \Omega , \\ -\Delta (\alpha _{2}\eta )=a_{2}\eta , & x\in \Omega , \\ \xi =\eta =0, & x\in \partial \Omega . \end{array}\displaystyle \right . $$

By Lemma 2.1(iii) we know that if \(\xi >0\) or \(\eta >0\), then \(\lambda _{1}(\alpha _{1}\Delta +a_{1})=0\) or \(\lambda _{1}(\alpha _{2}\Delta +a_{2})=0\), which contradicts the hypothesis (H5.1). Thus 1 is not an eigenvalue of \(L_{2}\) corresponding to a positive eigenfunction.

Next we calculate \(\mbox{index}_{W}(F,(0,0))\). Since \(\lambda _{1}(\alpha _{1}\Delta +a_{1})>0\), we get \(r(T_{1})>1\) by Lemma 2.3(i), where

$$ T_{1}:=\frac{1}{\alpha _{1}}(-\Delta +M)^{-1}(a_{1}+M\alpha _{1}). $$

Then using the Krein-Rutman theorem, one can see that \(r(T_{1})\) is an eigenvalue of \(T_{1}\) with a positive eigenfunction \(\phi \). That is, if we consider the pair \(\left ( \begin{array}{c} \phi \\ 0 \end{array} \right )\) and \(\lambda =r(T_{1})>1\), then there is an eigenvalue greater than one with a positive eigenfunction. By Theorem 2.3, there exists a \(\sigma '_{0}\in (0,\rho ')\) such that \(\mbox{index}_{W}(F,Q_{\sigma '})=0\) for any \(0<\sigma '<\sigma '_{0}\). On the other hand, since \((0,0)\) is isolated, there exists \(\delta >0\) such that \((0,0)\) is the only fixed point of \(F\) in \(Q_{\delta }\). If we take \(\sigma '<\min \{\sigma '_{0},\delta \}\), then

$$ \mbox{index}_{W}(F,(0,0))=\mbox{index}_{W}(F,Q_{\sigma '})=0. $$

 □

Lemma 5.3

(i) If \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})>0\), then \(\textit{index}_{W}(F,(u^{\ast },0))=0\).

(ii) If \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})>0\), then \(\textit{index}_{W}(F,(0,v^{\ast }))=0\).

Proof

(i) For the point \(y=(u^{\ast }, 0)\), we observe that \(\overline{W}_{y}=C_{0}(\overline{\Omega })\oplus K\). Set \(L_{3}=F'(u^{\ast },0)\). By calculation, we have

$$\begin{gathered} L_{3}=\left ((-\Delta +M)\left ( \textstyle\begin{array}{c@{\quad }c} \alpha _{1}+2\beta _{11}u^{\ast } & \beta _{12}u^{\ast } \\ 0 & \alpha _{2}+\beta _{21}u^{\ast } \end{array}\displaystyle \right )\right )^{-1}\left ( \textstyle\begin{array}{c@{\quad }c} \alpha & \beta \\ 0 & \gamma \end{array}\displaystyle \right ), \\ \left \{ \textstyle\begin{array}{l} \alpha =a_{1}-2b_{1}u^{\ast }+M(\alpha _{1}+2\beta _{11}u^{\ast }), \\ \beta =u^{\ast }(-c_{1}+M\beta _{12}), \\ \gamma =a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+\beta _{21}u^{ \ast }). \end{array}\displaystyle \right . \end{gathered}$$
(5.2)

First we prove that \(I-L_{3}\) is invertible on \(\overline{W}_{y}\). Suppose that there is \((\xi ,\eta )\in \overline{W}_{y}\) such that \((I-L_{3})\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )=\left ( \begin{array}{c} 0 \\ 0 \end{array} \right )\). Then we have

$$ \left \{ \textstyle\begin{array}{l} (-\Delta +M)^{-1}(\alpha \xi +\beta \eta )=(\alpha _{1}+2\beta _{11}u^{ \ast })\xi +\beta _{12}u^{\ast }\eta , \\ (-\Delta +M)^{-1}[(a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+ \beta _{21}u^{\ast }))\eta ]=(\alpha _{2}+ \beta _{21}u^{\ast })\eta . \end{array}\displaystyle \right . $$
(5.3)

The second equation in (5.3) implies

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [(\alpha _{2}+\beta _{21}u^{\ast })\eta ]=(a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\eta , & x\in \Omega , \\ \eta =0, & x\in \partial \Omega , \end{array}\displaystyle \right . $$

where \(\eta \in K\). If \(\eta \not \equiv 0\), then we can consider \(\eta \) as a positive eigenfunction of \(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+(a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\), and so \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})=0\), which contradicts our assumption. Thus \(\eta \equiv 0\). Substituting \(\eta =0\) into the first equation of (5.3), we have

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} \Delta [(\alpha _{1}+2\beta _{11}u^{\ast })\xi ]+(a_{1}-2b_{1}u^{\ast }) \xi =0, & x\in \Omega , \\ \xi =0, & x\in \partial \Omega . \end{array}\displaystyle \right . $$

If \(\xi \not \equiv 0\), then 0 is an eigenvalue of \(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+(a_{1}-2b_{1}u^{\ast })\), and so we have \(\lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })\geq 0\), which contradicts the fact that \(\lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })<0\). Thus \(\xi \equiv 0\), i.e., \((\xi ,\eta )=(0,0)\), and so \(I-L_{3}\) is invertible on \(\overline{W}_{y}\).

Next we show that \(L_{3}\) has property \(\alpha \) on \(\overline{W}_{y}\). Observe that \(S_{y}=C_{0}(\overline{\Omega })\oplus \{0\}\) and \(\overline{W}_{y}\backslash S_{y}=C_{0}(\overline{\Omega })\oplus \{K \backslash \{0\}\}\). Since \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})>0\) from the assumption, \(r(T_{2})>1\) by Lemma 2.3(i), where

$$ T_{2}:=\frac{1}{\alpha _{2}+\beta _{21}u^{\ast }}(-\Delta +M)^{-1}[a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }} +M(\alpha _{2}+\beta _{21}u^{\ast })], $$

and so \(r(T_{2})\) is an eigenvalue of \(T_{2}\) with a corresponding positive eigenfunction \(\phi ^{*}_{3}\in K\setminus \{0\}\) by the Krein-Rutman theorem. Set \(t=1/r(T_{2})\). Then \(t\in (0,1)\) and \((0,\phi ^{*}_{3})\in \overline{W}_{y}\backslash S_{y}\). Thus

$$\begin{aligned} (I-tL_{3})\left ( \textstyle\begin{array}{c} 0 \\ \phi ^{*}_{3} \end{array}\displaystyle \right )&=\left ( \textstyle\begin{array}{c} \frac{t\beta _{12}u^{\ast }(-\Delta +M)^{-1}((a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+\beta _{21}u^{\ast }))\phi ^{*}_{3}) }{(\alpha _{1}+2\beta _{11}u^{\ast })(\alpha _{2}+\beta _{21}u^{\ast })} - \frac{t(-\Delta +M)^{-1}((-c_{1}+M\beta _{12})u^{\ast }\phi ^{*}_{3})}{\alpha _{1}+2\beta _{11}u^{\ast }} \\ \phi ^{*}_{3}- \frac{t(-\Delta +M)^{-1}((a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+\beta _{21}u^{\ast }))\phi ^{*}_{3})}{\alpha _{2}+\beta _{21}u^{\ast }} \end{array}\displaystyle \right ) \\ &=\left ( \textstyle\begin{array}{c} \frac{t\beta _{12}u^{\ast }(-\Delta +M)^{-1}((a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+\beta _{21}u^{\ast }))\phi ^{*}_{3}) }{(\alpha _{1}+2\beta _{11}u^{\ast })(\alpha _{2}+\beta _{21}u^{\ast })} - \frac{t(-\Delta +M)^{-1}((-c_{1}+M\beta _{12})u^{\ast }\phi ^{*}_{3})}{\alpha _{1}+2\beta _{11}u^{\ast }} \\ \phi ^{*}_{3}-\frac{1}{r(T_{2})}T_{2}\phi ^{*}_{3} \end{array}\displaystyle \right ) \\ &=\left ( \textstyle\begin{array}{c} \frac{t\beta _{12}u^{\ast }(-\Delta +M)^{-1}((a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+\beta _{21}u^{\ast }))\phi ^{*}_{3}) }{(\alpha _{1}+2\beta _{11}u^{\ast })(\alpha _{2}+\beta _{21}u^{\ast })} - \frac{t(-\Delta +M)^{-1}((-c_{1}+M\beta _{12})u^{\ast }\phi ^{*}_{3})}{\alpha _{1}+2\beta _{11}u^{\ast }} \\ 0 \end{array}\displaystyle \right ) \\ &\in S_{y}, \end{aligned}$$

i.e., \(L_{3}\) has property \(\alpha \). Therefore \(\mbox{index}_{W}(F,(u^{\ast },0))=0\) by Theorem 2.2(i).

Using the similar technique, we can prove (ii). □

Lemma 5.4

(i) If \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\leq 0\), then \(\textit{index}_{W}(F,(u^{\ast },0))=1\).

(ii) If \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\leq 0\), then \(\textit{index}_{W}(F,(0,v^{\ast }))=1\).

Proof

We only prove (i) since we can make a similar argument for (ii). Consider the following two cases:

  1. (a)

    \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})<0\);

  2. (b)

    \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})=0\).

Case (a). Note that \(\overline{W}_{y}=C_{0}(\overline{\Omega })\oplus K\), \(S_{y}=C_{0}(\overline{\Omega })\oplus \{0\}\), \(\overline{W}_{y}\backslash S_{y}=C_{0}(\overline{\Omega })\oplus \{K \backslash \{0\}\}\). Assume that \(\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )\) is an eigenfunction of \(L_{3}\) corresponding to some eigenvalue \(\lambda \geq 1\). Then we have

$$ \left \{ \textstyle\begin{array}{l} (-\Delta +M)^{-1} (\alpha \xi +\beta \eta )=(\alpha _{1}+2\beta _{11}u^{ \ast })(\lambda \xi )+\beta _{12}u^{\ast }(\lambda \eta ), \\ (-\Delta +M)^{-1}[(a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+ \beta _{21}u^{\ast }))\eta ]=(\alpha _{2}+ \beta _{21}u^{\ast })( \lambda \eta ). \end{array}\displaystyle \right . $$
(5.4)

By Lemma 2.3(ii), our assumption (a) implies

$$ r(\frac{1}{\alpha _{2}+\beta _{21}u^{\ast }}(-\Delta +M)^{-1}(a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }}+M(\alpha _{2}+ \beta _{21}u^{\ast })))< 1, $$

and so \(\eta \equiv 0\). Substituting \(\eta =0\) into the first equation of (5.4), we can similarly derive \(\xi \equiv 0\) by \(\lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })<0\) and Lemma 2.3(ii). This implies that \(I-L_{3}\) is invertible on \(\overline{W}_{y}\) and \(L_{3}\) does not have an eigenvalue which is greater than or equal to one. Further, as in Lemma 5.1, one can easily check that \(L_{3}\) does not have property \(\alpha \), so \(\mbox{index}_{W}(F,(u^{\ast },0))=1\).

Case (b). Define \(F_{\mu _{1}}=G^{-1}\circ H_{\mu _{1}}\) for \(\mu _{1}\in [0,1]\), where

$$\begin{gathered} H_{\mu _{1}}(u,v)=((-\Delta +M)^{-1}S_{1}(u,v), (-\Delta +M)^{-1}S_{2, \mu _{1}}(u,v)), \\ S_{1}(u,v)=u(a_{1}-b_{1}u-\frac{c_{1}v}{1+v}+M(\alpha _{1}+\beta _{11}u+ \beta _{12}v)), \\ S_{2,\mu _{1}}(u,v)=v(a_{2}-b_{2}v-\frac{c_{2}u}{1+u}-\mu _{1}+M( \alpha _{2}+\beta _{21}u+\beta _{22}v)). \end{gathered}$$

Clearly, \((u^{\ast },0)\) is a fixed point of \(F_{\mu _{1}}\) for each \(\mu _{1}\in [0,1]\) and \(F_{0}=F\). Also we can easily verify that every fixed point of \(F_{\mu _{1}}\) satisfies \(u(x)\leq M_{1}\) and \(v(x)\leq M_{2}\). Hence \(F_{\mu _{1}}\) has no fixed points on \(\partial D\times [0,1]\). By the homotopy invariance property of degree, \(\mbox{index}_{W}(F,(u^{\ast },0))=\mbox{index}_{W}(F_{\mu _{1}},(u^{ \ast },0))\).

Now we show that \(\mbox{index}_{W}(F_{\mu _{1}},(u^{\ast },0))=1\). Set \(L_{\mu _{1}}=F_{\mu _{1}}'(u^{\ast },0)\). Then we have

$$ L_{\mu _{1}}=\left ((-\Delta +M)\left ( \textstyle\begin{array}{c@{\quad }c} \alpha _{1}+2\beta _{11}u^{\ast } & \beta _{12}u^{\ast } \\ 0 & \alpha _{2}+\beta _{21}u^{\ast } \end{array}\displaystyle \right )\right )^{-1}\left ( \textstyle\begin{array}{c@{\quad }c} \alpha & \beta \\ 0 & \gamma ^{*} \end{array}\displaystyle \right ), $$

where \(\alpha , \beta \) are defined as in (5.2) and \(\gamma ^{*}=a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}-\mu _{1}+M( \alpha _{2}+\beta _{21}u^{\ast })\). Fix \(\mu _{1}>0\). Suppose that \(\left ( \begin{array}{c} \xi \\ \eta \end{array} \right )\) is an eigenfunction of \(L_{\mu _{1}}\) corresponding to some eigenvalue \(\lambda \geq 1\). Then \(\eta \) satisfies \(\lambda \eta (\alpha _{2}+\beta _{21}u^{\ast })=(-\Delta +M)^{-1}( \gamma ^{*}\eta )\), i.e., \(\Delta ((\alpha _{2}+\beta _{21}u^{\ast })\eta )+B^{*}\eta =0\) in \(\Omega \) and \(\eta =0\) on \(\partial \Omega \), where

$$ B^{*}=a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+ \frac{1-\lambda }{\lambda }(a_{2}-\frac{c_{2}u^{\ast }}{1+u^{\ast }}+ M( \alpha _{2}+\beta _{21}u^{\ast }))-\frac{\mu _{1}}{\lambda }, $$

\(\eta \in K\). If \(\eta \not \equiv 0\), then we can consider \(\eta \) as a positive eigenfunction of \(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+B^{*}\). This implies \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+B^{*})=0\). Since \(\lambda \geq 1\) and \(\mu _{1}>0\), we have \(0=\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+B^{*})< \lambda _{1}(a_{2}+\beta _{21}u^{\ast }+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) by Lemma 2.2(i), which contradicts our assumption (b). So \(\eta \equiv 0\). Thus \(\xi \) satisfies \(\lambda (\alpha _{1}+2\beta _{11}u^{\ast })\xi =(-\Delta +M)^{-1}( \alpha \xi )\), and so

$$ \Delta ((\alpha _{1}+2\beta _{11}u^{\ast })\xi )+(a_{1}-2b_{1}u^{*}+ \frac{1-\lambda }{\lambda }\alpha )\xi =0 $$

in \(\Omega \) and \(\xi =0\) on \(\partial \Omega \). If \(\xi \not \equiv 0\), then 0 is an eigenvalue of \(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+(a_{1}-2b_{1}u^{*}+ \frac{1-\lambda }{\lambda }\alpha )\), and so \(\lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{*}+ \frac{1-\lambda }{\lambda }\alpha )\geq 0\). Since \(\lambda \geq 1\), we get

$$ \lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{*}) \geq 0 $$

by Lemma 2.2(ii), which also contradicts the fact that \(\lambda _{1}(\Delta (\alpha _{1}+2\beta _{11}u^{\ast })+a_{1}-2b_{1}u^{ \ast })<0\). Hence \(I-L_{\mu _{1}}\) is invertible in \(\overline{W}_{y}\) and \(L_{\mu _{1}}\) has no eigenvalue greater than or equal to one. As in Lemma 5.1, one can easily check that \(L_{\mu _{1}}\) does not have property \(\alpha \) on \(\overline{W}_{y}\). Thus we conclude that \(\mbox{index}_{W}(F_{\mu _{1}},(u^{*},0))=1\) by Theorem 2.2(ii). □

Combing Lemma 5.1-Lemma 5.4, we give the proof of Theorem 5.1.

Proof of Theorem 5.1

(i) By Theorem 4.1 we obtain that \((0,0),(u^{\ast },0),(0,v^{\ast })\in D'\). Suppose that \(F\) has no positive fixed point in \(D'\). Then by Lemma 5.1 and the additivity of index, we have

$$ \mbox{index}_{W}(F,(0,0))+\mbox{index}_{W}(F,(u^{\ast },0))+\mbox{index}_{W}(F,(0,v^{ \ast }))=\mbox{index}_{W}(F,D')=1. $$
(5.5)

If \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})>0\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})>0\), then by Lemmas 5.2 and 5.3 we have

$$ \mbox{index}_{W}(F,(0,0))+\mbox{index}_{W}(F,(u^{\ast },0))+\mbox{index}_{W}(F,(0,v^{ \ast }))=0, $$

which contradicts (5.5). By the similar arguments, when the signs of first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) are both negative or zero, or one of these first eigenvalues is less than 0 and one is equal to 0, we can also derive a contradiction by using Lemmas 5.1, 5.2 and 5.4. Therefore model (1.4) must have a positive solution in \(D'\). This concludes the proof of Theorem 5.1(i).

(ii) If \(\beta _{12}=\beta _{21}=0\), then model (1.4) becomes

$$ \left \{ \textstyle\begin{array}{l@{\quad }l} -\Delta [(\alpha _{1}+\beta _{11}u)u]=u(a_{1}-b_{1}u- \frac{c_{1}v}{1+v}), & x\in \Omega , \\ -\Delta [(\alpha _{2}+\beta _{21}u)v]=v(a_{2}-b_{2}v- \frac{c_{2}u}{1+u}), & x\in \Omega , \\ u=v=0, & x\in \partial \Omega . \end{array}\displaystyle \right . $$
(5.6)

Similar to the proof of Theorem 5.1(i), we can show the sufficiency of the conditions in Theorem 5.1(ii). We now prove the necessity. Here we assume \(h_{1}<0\) and \(h_{2}<0\). The other cases are proved in the same way.

Let \((\bar{u},\bar{v})\) be a positive solution to system (5.6). Then \(\bar{u}\) solves

$$ \Delta [(\alpha _{1}+\beta _{11}\bar{u})\bar{u}]+\bar{u}(a_{1}-b_{1} \bar{u}-\frac{c_{1}\bar{v}}{1+\bar{v}})=0, \quad x\in \Omega , \quad \bar{u}=0, \quad x\in \partial \Omega . $$

Hence \(\bar{u}\) is a positive lower solution to

$$ \Delta [(\alpha _{1}+\beta _{11}u)u]+u(a_{1}-b_{1}u)=0, \quad x\in \Omega , \quad u=0, \quad x\in \partial \Omega . $$
(5.7)

Obviously, \(\frac{a_{1}}{b_{1}}\) is an upper solution to (5.7). So (5.7) has a positive solution \(u^{\ast }\), and hence

$$ \lambda _{1}(\Delta (\alpha _{1}+\beta _{11}u^{\ast })+a_{1}-b_{1}u^{ \ast })=0. $$

By the similar technique, we can obtain that

$$ \Delta [(\alpha _{2}+\beta _{22}v)v]+v(a_{2}-b_{2}v)=0, \quad x\in \Omega , \quad v=0, \quad x\in \partial \Omega $$
(5.8)

has a positive solution \(v^{\ast }\). Hence

$$ \lambda _{1}(\Delta (\alpha _{2}+\beta _{22}v^{\ast })+a_{2}-b_{2}v^{ \ast })=0. $$

By \(h_{1}<0\), \(h_{2}<0\) and Lemma 2.2, we have

$$ \lambda _{1}(\alpha _{1}\Delta +a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})< \lambda _{1}(\Delta (\alpha _{1}+ \beta _{11}u^{\ast })+a_{1}-b_{1}u^{\ast })=0 $$

and

$$ \lambda _{1}(\alpha _{2}\Delta +a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})< \lambda _{1}(\Delta (\alpha _{2}+ \beta _{22}v^{\ast })+a_{2}-b_{2}v^{\ast })=0. $$

Therefore the signs of \(\lambda _{1}(\alpha _{1}\Delta +a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\alpha _{2}\Delta +a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) are all negative. Similarly, we can also prove that the other cases are true. This completes the proof of (ii). □

Remark 5.2

If the signs of the first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) are opposite, or one of these first eigenvalues is greater than 0 and one is equal to 0, then by Lemmas 5.2-5.4, we have

$$ \mbox{index}_{W}(F,(0,0))+\mbox{index}_{W}(F,(u^{\ast },0))+\mbox{index}_{W}(F,(0,v^{ \ast }))=1. $$

By Lemma 5.1, we get \(\mbox{index}_{W}(F, D')=1\), hence it cannot show that whether (1.4) has a positive solution or not. However, if \(\alpha _{1}, \alpha _{2}, \beta _{12}, \beta _{21}\) also satisfy \(\alpha _{1}>D_{1}^{0}, \alpha _{2}>D_{2}^{0}, D_{1}\leq \beta _{12}< A_{1}, D_{2}\leq \beta _{21}< A_{2}\), then by Theorem 4.2 we know that model (1.4) has no positive solution. Biologically, this implies that there may be at least one species cannot persist. From the nature point of view, it is also reasonable.

According to Theorem 5.1(i), if we can find some conditions such that the signs of the first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) satisfy any of the situations in Theorem 5.1(i), then the existence of positive solution of (1.4) can be obtained. In fact, the following Corollary 5.1 and Corollaries 5.2-5.3 give some sufficient conditions for the signs of the first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) to be negative and positive, respectively. From Theorem 5.1(i), it follows naturally that those conditions are sufficient for system (1.4) to have a positive solution. We give the specific analysis as follows.

Corollary 5.1

System (1.4) has a positive solution either if

  1. (i)

    the cross-diffusion pressures \(\beta _{12}\) and \(\beta _{21}\) are sufficiently large for fixed \(\alpha _{i}, a_{i}, b_{i}, c_{i}, \beta _{ii}, i=1,2\), or

  2. (ii)

    the inter-specific competition pressures \(c_{1}\) and \(c_{2}\) are sufficiently large for fixed \(\alpha _{i}, a_{i}\), \(b_{i}, \beta _{ij}, i,j=1,2\).

Proof

(i) By (2.2), we have

$$ \begin{aligned} &\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }}) \\ =&\sup _{u\in W^{1,2}(\Omega )} \frac{\int _{\Omega }(-|\nabla [(\alpha _{1}+\beta _{12}v^{\ast })u]|^{2}+(\alpha _{1}+\beta _{12}v^{\ast }) (a_{1}-\frac{c_{1}v^{\ast }}{1+v^{\ast }})u^{2})\mbox{d}x}{\|\sqrt{(\alpha _{1}+\beta _{12}v^{\ast })}u\|^{2}_{L^{2}}}. \end{aligned} $$

Obviously, there exists a constant \(m_{1}>0\) such that

$$ \lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})< 0 $$

for all \(\beta _{12}>m_{1}\). Similarly, there also exists a constant \(m_{2}>0\) such that

$$ \lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})< 0 $$

for all \(\beta _{21}>m_{2}\). Thus the result follows from Theorem 5.1(i).

Similarly, (ii) can also be proved. □

Corollary 5.2

If \(\lambda _{1}>- \frac{a_{1}b_{2}-c_{1}a_{2}/(a_{2}+b_{2})}{\alpha _{1}b_{2}+a_{2}\beta _{12}}\), \(\lambda _{1}>- \frac{a_{2}b_{1}-c_{2}a_{1}/(a_{1}+b_{1})}{\alpha _{2}b_{1}+a_{1}\beta _{21}}\), \(c_{1}< a_{1}, c_{2}< a_{2}\), \(\beta _{12}<\frac{\alpha _{1}(a_{1}-c_{1})}{a_{1}}, \beta _{21}< \frac{\alpha _{2}(a_{2}-c_{2})}{a_{2}}\), then model (1.4) has a positive solution.

Proof

From the proof of Theorem 5.1(ii), we know that \(\frac{a_{1}}{b_{1}}\) and \(\frac{a_{2}}{b_{2}}\) are the positive upper solutions of (5.7) and (5.8), respectively. Since \(u^{\ast }\) and \(v^{\ast }\) are the unique positive solution of (5.7) and (5.8), respectively, \(u^{\ast }\leq \frac{a_{1}}{b_{1}}, v^{\ast }\leq \frac{a_{2}}{b_{2}}\). From the assumptions \(\lambda _{1}>- \frac{a_{1}b_{2}-c_{1}a_{2}/(a_{2}+b_{2})}{\alpha _{1}b_{2}+a_{2}\beta _{12}}\) and \(\lambda _{1}>- \frac{a_{2}b_{1}-c_{2}a_{1}/(a_{1}+b_{1})}{\alpha _{2}b_{1}+a_{1}\beta _{21}}\), we get

$$ \lambda _{1}(\Delta (\alpha _{1}+\beta _{12}(a_{2}/b_{2}))+a_{1}- \frac{c_{1}(a_{2}/b_{2})}{1+a_{2}/b_{2}})>0 $$

and

$$ \lambda _{1}(\Delta (\alpha _{2}+\beta _{21}(a_{1}/b_{1}))+a_{2}- \frac{c_{2}(a_{1}/b_{1})}{1+a_{1}/b_{1}})>0. $$

By assumptions \(c_{1}< a_{1}, c_{2}< a_{2}\), \(\beta _{12}<\frac{\alpha _{1}(a_{1}-c_{1})}{a_{1}}, \beta _{21}< \frac{\alpha _{2}(a_{2}-c_{2})}{a_{2}}\), we get that \(\frac{a_{1}-c_{1}v/(1+v)}{\alpha _{1}+\beta _{12}v}\) and \(\frac{a_{2}-c_{2}u/(1+u)}{\alpha _{2}+\beta _{21}u}\) are monotone decreasing in \(v\geq 0\) and \(u\geq 0\), respectively. So

$$ \frac{a_{1}-c_{1}v^{\ast }/(1+v^{\ast })}{\alpha _{1}+\beta _{12}v^{\ast }}> \frac{a_{1}-\frac{c_{1}(a_{2}/b_{2})}{1+a_{2}/b_{2}}}{\alpha _{1}+\beta _{12}(a_{2}/b_{2})} \quad \mbox{and} \quad \frac{a_{2}-c_{2}u^{\ast }/(1+u^{\ast })}{\alpha _{2}+\beta _{21}u^{\ast }}> \frac{a_{2}-\frac{c_{2}(a_{1}/b_{1})}{1+a_{1}/b_{1}}}{\alpha _{2}+\beta _{21}(a_{1}/b_{1})}, $$

and so \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})>0\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})>0\) by Lemma 2.2(ii). Therefore model (1.4) has a positive solution by Theorem 5.1(i). □

Corollary 5.3

If \(\lambda _{1}>- \frac{a_{2}b_{1}-c_{2}a_{1}/(a_{1}+b_{1})}{\alpha _{2}b_{1}+a_{1}\beta _{21}}\), \(c_{1}< a_{1}, c_{2}< a_{2}\), \(\beta _{12}<\frac{\alpha _{1}(a_{1}-c_{1})}{a_{1}}, \beta _{21}< \frac{\alpha _{2}(a_{2}-c_{2})}{a_{2}}\), \(\frac{a_{1}}{a_{2}}>\frac{\alpha _{1}}{\alpha _{2}}>\max \{ \frac{\beta _{12}}{\beta _{22}}, \frac{c_{1}\beta _{11}}{b_{1}\beta _{22}}\}\) and \(\frac{b_{1}}{b_{2}}<\frac{\beta _{11}}{\beta _{22}}\), then model (1.4) has a positive solution.

Proof

Similar to the proof of Corollary 5.2, we can prove that

$$ \lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})>0 $$

when \(\lambda _{1}>- \frac{a_{2}b_{1}-c_{2}a_{1}/(a_{1}+b_{1})}{\alpha _{2}b_{1}+a_{1}\beta _{21}}, c_{2}< a_{2}\) and \(\beta _{21}<\frac{\alpha _{2}(a_{2}-c_{2})}{a_{2}}\). Substituting \(u=\frac{\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }\) into (5.7) and using the given conditions \(\frac{a_{1}}{a_{2}}>\frac{\alpha _{1}}{\alpha _{2}}, \frac{b_{1}}{b_{2}}<\frac{\beta _{11}}{\beta _{22}}\), we get

$$ \begin{aligned} \Delta [(\alpha _{1}+\beta _{11}u)u]+u(a_{1}-b_{1}u)=&\ \alpha _{1} \Delta [(1+\frac{\beta _{11}}{\alpha _{1}}u)u]+u(a_{1}-b_{1}u) \\ =&\ \alpha _{1}\Delta [(1+\frac{\beta _{22}}{\alpha _{2}}v^{\ast }) \frac{\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }] + \frac{\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }(a_{1}- \frac{b_{1}\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }) \\ =&\ \frac{\alpha ^{2}_{1}\beta _{22}}{\alpha ^{2}_{2}\beta _{11}}v^{ \ast }(b_{2}v^{\ast }-a_{2})+ \frac{\alpha _{1} \beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }(a_{1}- \frac{b_{1}\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }) \\ =&\ \frac{\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }[a_{1}- \frac{a_{2}\alpha _{1}}{\alpha _{2}}+ ( \frac{b_{2}\alpha _{1}}{\alpha _{2}}- \frac{b_{1}\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}})v^{\ast }] \\ >&\ 0. \end{aligned} $$

So, \(\frac{\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }\) is a positive lower solution to (5.7), and \(\frac{\alpha _{1}\beta _{22}}{\alpha _{2}\beta _{11}}v^{\ast }< u^{ \ast }\).

By assumptions \(c_{1}< a_{1}\) and \(\beta _{12}<\frac{\alpha _{1}(a_{1}-c_{1})}{a_{1}}\), we know that \(\frac{a_{1}-c_{1}v/(1+v)}{\alpha _{1}+\beta _{12}v}\) is monotone decreasing with respect to \(v\geq 0\), so by Lemma 2.2(ii) and given condition \(\frac{\alpha _{1}}{\alpha _{2}}>\max \{ \frac{\beta _{12}}{\beta _{22}}, \frac{c_{1}\beta _{11}}{b_{1}\beta _{22}}\}\), we have

$$ \begin{aligned} \lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }}) >{}& \lambda _{1}(\Delta (\alpha _{1} {+} \frac{\alpha _{2}\beta _{11}\beta _{12}}{\alpha _{1}\beta _{22}}u^{ \ast }) {+} a_{1} {-} \frac{c_{1}\alpha _{2}\beta _{11}u^{\ast }/(\alpha _{1}\beta _{22})}{1+\alpha _{2}\beta _{11}u^{\ast }/ (\alpha _{1}\beta _{22})}) \\ >{}& \lambda _{1}(\Delta (\alpha _{1}+ \frac{\alpha _{2}\beta _{11}\beta _{12}}{\alpha _{1}\beta _{22}}u^{ \ast }) +a_{1}-c_{1}\alpha _{2}\beta _{11}u^{\ast }/(\alpha _{1}\beta _{22})) \\ >{}& \lambda _{1}(\Delta (\alpha _{1}+\beta _{11}u^{\ast })+a_{1}-b_{1}u^{ \ast }) \\ ={}& 0. \end{aligned} $$

Therefore, by Theorem 5.1(i) we know that model (1.4) has a positive solution. □

6 Conclusion

A rigorous investigation on the dynamics of a competitive model with cross-diffusion and nonlinear functional response subject to the homogeneous Dirichlet boundary condition is discussed. We first prove the stability of trivial and semi-trivial solutions of model (1.4) by spectrum analysis. Then the boundedness and non-existence of positive solution of (1.4) are obtained. By using the fixed point index theory in a positive cone, we prove that the existence of positive solutions of model (1.4) can be characterized by the signs of the first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\). Combining with the previous analysis in this paper, we get the following conclusions:

(I) Assume that the cross-diffusion pressures of two competition species \(u\) and \(v\) are controlled in the certain range. (i) If the signs of the first eigenvalues \(\lambda _{1}(\Delta (\alpha _{1}+\beta _{12}v^{\ast })+a_{1}- \frac{c_{1}v^{\ast }}{1+v^{\ast }})\) and \(\lambda _{1}(\Delta (\alpha _{2}+\beta _{21}u^{\ast })+a_{2}- \frac{c_{2}u^{\ast }}{1+u^{\ast }})\) are the same, or one of these first eigenvalues is less than 0 and one is equal to 0, then (1.4) has a positive solution (see Theorem 5.1(i)); (ii) Suppose that the signs of the first eigenvalues above are opposite, or one of these first eigenvalues is greater than 0 and one is equal to 0. If the self- and cross-diffusion pressures also meet additional conditions (i.e., \(\alpha _{1}>D_{1}^{0}, \alpha _{2}>D_{2}^{0}, D_{1}\leq \beta _{12}< A_{1}, D_{2}\leq \beta _{21}< A_{2}\)), then model (1.4) has no positive solution (see Remark 5.2). Biologically, the former case implies that the two competition species \(u\) and \(v\) can coexist, and the later case implies that there may be at least one species cannot persist. From the nature point of view, it is also reasonable.

(II) Assume that the cross-diffusion pressures \(\beta _{12}=\beta _{21}=0\), and some certain conditions also hold (i.e. \(h_{1}\) and \(h_{2}\) have the same signs on \((0,\overline{M})\)). Then the conditions in Theorem 5.1(i) are necessary and sufficient for the existence of positive solutions to system (1.4) (see Theorem 5.1(ii)). Biologically, this means that the two competition species \(u\) and \(v\) can coexist, and when their self-diffusion, inter-specific competition pressures and growth rates meet certain conditions, their coexistence is not affected by the cross-diffusion pressures.

(III) Either if the cross-diffusion pressures \(\beta _{12}\) and \(\beta _{21}\) or the inter-specific competition pressures \(c_{1}\) and \(c_{2}\) are sufficiently large, then system (1.4) has a positive solution (see Corollary 5.1). Biologically, this means that the two competition species \(u\) and \(v\) can coexist when their pressures of cross-diffusion or inter-specific competition are sufficiently large. Moreover, we also prove that when cross-diffusion pressures are controlled within a certain range, two species can also coexist (see Corollary 5.2 and Corollary 5.3).

The methods and results in the present paper may enrich the research of dynamics in the competition model. Further studies are necessary to analyze the behavior of more complex spatial models such as competition model with time delay or other kinds of cross-diffusion terms and functional responses.