Ground state solutions for a fractional system involving critical non-linearities

Abstract

The aim of this paper is to study a fractional system involving critical non-linearities. Using the Mountain Pass Theorem, the existence of ground state solutions for our problem is obtained in two cases.

1 Introduction

In the past few decades, Laplace equations or systems have been extensively studied, and there are many results about multiple positive solutions, ground state solutions, sign-change solutions and so on (see [8,9,10, 14, 15, 17] and references therein). In addition, the coupled Schrödinger system involving Laplacian appears in several branches of physics. It can accurately describe the multiplicate chemical reaction catalyzed by the catalyst grains under constant or variant temperature. Moreover, it can describe the interaction between the non-linear Schrödinger field and the electromagnetic field. The author in [3] studied a class of coupled quasi-linear semilinear Schrödinger system

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u+a(x)u=F_u(x,u,v),\quad x\in {\mathbb {R}}^N,\\ -\Delta v+b(x)v=F_v(x,u,v),\quad x\in {\mathbb {R}}^N, \end{array}\right. } \end{aligned}$$

where $F\in C^1({\mathbb {R}}^N\times {\mathbb {R}}^2,{\mathbb {R}})$. With the help of the generalized mountain pass lemma, the author proved that the system has a nontrivial solution. Since then, the coupled quasi-linear Schrödinger system has attracted more and more attention from related scholars (see [1, 2] and the references therein).

In recent years, fractional differential equations have played an important role in many fields such as science, electrical circuits, engineering and applied mathematics (see [11, 12]). Compared with the Laplace problem, the fractional Laplace problem is non-local and faces greater research difficulty. In recent years, both elliptic fractional and non-local operators have received great attention in the research of pure mathematics and the practical application of mathematics (see [6, 7, 16] and references therein). Therefore, the study of coupled systems is natural. Consider the following fractional system:

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su+\mu u=|u|^{p-1}u+\lambda v, \quad x\in {\mathbb {R}}^N,\\ (-\Delta )^sv+\nu v=|v|^{2^*-2}v+\lambda u, \quad x\in {\mathbb {R}}^N, \end{array}\right. } \end{aligned}$$

where $(-\Delta )^s$ is the fractional Laplacian, $\mu$, $\nu$ and $\lambda$ are parameters, $0<s<1$, $N>2s$, $\lambda <\sqrt{\mu \nu }$, $1<p<2^*-1$, $2^*=\frac{2N}{N-2s}$ is the Sobolev critical exponent. The authors in [19] proved that there exists a $\mu _0 \in (0,1)$, such that when $0 < \mu \le \mu _0$, the system has a positive ground state solution. When $\mu > \mu _0$, there exists a $\lambda _{\mu ,\nu }\in [ \sqrt{(\mu -\mu _0)\nu } ,\sqrt{\mu \nu })$, such that if $\lambda >\lambda _{\mu ,\nu }$, the system has a positive ground state solution; if $\lambda <\lambda _{\mu ,\nu }$, the system has no ground state solution.

In [13], the authors studied the small energy solutions of the coupled fractional Schrödinger system with critical growth. Using a variant of fountain theorem, when the Ambrosetti–Rabinowitz (AR) condition is not satisfied, the criterion for the existence of an infinite number of small energy solutions was explained.

As far as we know, there are few research results on concave–convex non-linear fractional elliptic systems. In [4], the authors studied the multiple solutions of fractional equations that satisfy the homogeneous Dirichlet boundary conditions. They obtained multiple solutions for the following fractional elliptic system:

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su=\lambda |u|^{q-2}u+\frac{2\alpha }{\alpha +\beta }|u|^{\alpha -2}u|v|^\beta , &{}\quad x\in \Omega ,\\ (-\Delta )^sv=\mu |v|^{q-2}v+\frac{2\beta }{\alpha +\beta }|u|^{\alpha }|v|^{\beta -2}v, &{}\quad x\in \Omega ,\\ u=v=0, &{}\quad x\in {\mathbb {R}}^N\setminus \Omega , \end{array}\right. } \end{aligned}$$

where $\Omega$ is a smooth bounded set in ${\mathbb {R}}^N$, $n>2s$, with $s\in (0,1)$, $(-\Delta )^s$ is the fractional Laplace operator; $\lambda ,\mu >0$ are two parameters; the exponent $\frac{n}{n-2s}\le 2$; $\alpha >1$, $\beta >1$ satisfy $2<\alpha +\beta =2^*_s$, $2^*_s=\frac{2n}{n-2s}(n>2s)$ is the fractional critical Sobolev exponent.

In [18], the authors focused on the following critical case fractional Laplacian system:

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su+\lambda _1u=\mu _1 |u|^{2^*-2}u+\frac{\alpha \gamma }{2^*}|u|^{\alpha -2}u|v|^\beta \quad &{}\text {in} \ \Omega ,\\ (-\Delta )^sv+\lambda _2v=\mu _2 |v|^{2^*-2}v+\frac{\beta \gamma }{2^*}|u|^{\alpha }|v|^{\beta -2}v \quad &{}\text {in} \ \Omega ,\\ u=v=0 \quad &{}\text {in}\ {\mathbb {R}}^N\setminus \Omega , \end{array}\right. } \end{aligned}$$

where $(-\Delta )^s$ is the fractional Laplacian, $0<s<1$, $\mu _1,\mu _2>0$, $2^*=\frac{2N}{N-2s}$ is a fractional critical Sobolev exponent, $N>2s$, $1<\alpha , \beta <2$, $\alpha +\beta =2^*$, $\Omega$ is an open-bounded set of ${\mathbb {R}}^N$ with Lipschitz boundary and $\lambda _1,\lambda _2>-\lambda _{1,s}(\Omega )$, $\lambda _{1,s}$ is the first eigenvalue of the non-local operator $(-\Delta )^s$ with homogeneous Dirichlet boundary datum. Using the Nehari mainfold, the authors proved the existence of a positive ground state solution of the system for all $\gamma >0$. Then, the asymptotic behaviors of the positive ground state solutions are analyzed when $\gamma \rightarrow 0$.

Recently, the positive ground states for a system of Schrödinger equations with critically growing non-linearities have been studied by many authors. At the same time, new difficulties have arisen. Due to the non-linearities, sometimes traditional methods have lost their effectiveness. Therefore, from the perspective of the Palais–Smale sequence of the functional, the authors in [5] studied the following system:

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u=\lambda u+|u|^{2^*-2}v &{}\text {in}\ \Omega ,\\ -\Delta v=\mu |v|^{2^*-2}v+|u|^{2^*-2}u &{}\text {in}\ \Omega ,\\ u>0,v>0&{}\text {in} \ \Omega ,\\ u=v=0 &{}\text {on}\ \partial \Omega , \end{array}\right. } \end{aligned}$$

where $\Omega$ is a bounded domain of ${\mathbb {R}}^N$, $N\ge 4$, $2^*=\frac{2N}{N-2}$, $\lambda \in {\mathbb {R}}$ and $\mu \ge 0$ . They obtained existence and nonexistence results, depending on the value of the parameters $\lambda$ and $\mu$.

Motivated by the above works, especially by [5], we propose the problem

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su=\lambda u+u^{2_s^*-2}v, &{}x\in \Omega ,\\ (-\Delta )^sv=\mu v^{2_s^*-1}+u^{2_s^*-1}, &{}x\in \Omega ,\\ u>0,v>0,&{}x\in \Omega ,\\ u=v=0, &{}x\in {\mathbb {R}}^N\setminus \Omega , \end{array}\right. } \end{aligned}$$

(1)

where $0<s<1, N\ge 4s$, $2_s^*:=\frac{2N}{N-2s}$ is the fractional Sobolev critical exponent, $\Omega$ is an open-bounded domain of ${\mathbb {R}}^N$ with Lipschitz boundary, $\lambda$, $\mu$ are parameters and $\mu \ge 0$.

The fractional Laplacian operator can be defined by

$$\begin{aligned} \begin{aligned} (-\Delta )^su(x)&=C_{N,s}\text {P.V.}\int _{{\mathbb {R}}^{N}} \frac{u(x)-u(y)}{|x-y|^{N+2s}}\text {d}y\\&=C_{N,s}\lim _{\varepsilon \rightarrow 0^+}\int _{B_\varepsilon ^c(x)} \frac{u(x)-u(y)}{|x-y|^{N+2s}}\text {d}y\\&=-\frac{1}{2}C_{N,s}\int _{{\mathbb {R}}^N}\frac{u(x+y)+u(x-y)-2u(x)}{|y|^{N+2s}} \text {d}y, \end{aligned} \end{aligned}$$

where $C_{N,s}$ is given by

$$\begin{aligned} C_{N,s}=\left( \int _{{\mathbb {R}}^N}\frac{1-\cos (\zeta _1)}{|\zeta |^{N+2s}} \text {d}\zeta \right) ^{-1}, \end{aligned}$$

and P.V. is the principle value defined by the latter formula.

Define Hilbert space $D^s(\Omega )$ as the completion of $C_c^\infty (\Omega )$ with respect to the norm $\Vert \cdot \Vert _{D^s}$ induced by the following scalar product:

$$\begin{aligned} \langle u,v\rangle _{D^s}:=\frac{C_{N,s}}{2}\int _{{\mathbb {R}}^{2N}}\frac{\big (u(x)-u(y)\big ) \big (v(x)-v(y)\big )}{|x-y|^{N+2s}}\text {d}x\text {d}y. \end{aligned}$$

If $\Omega$ is an open-bounded Lipschitz domain, then $D^s(\Omega )$ coincides with the Sobolev space

$$\begin{aligned} X_0:=\{f\in X:f=0\text { a.e. in } \Omega ^c\}, \end{aligned}$$

where X is a linear space of Lebesgue measurable functions from ${\mathbb {R}}^N$ to ${\mathbb {R}}$, such that the restriction to $\Omega$ of any function f in X belongs to $L^2(\Omega )$ and the map $(x,y)\mapsto \big (f(x)-f(y)\big )|x-y|^{-\frac{N}{2}+s}$ is in $L^2\left( {\mathbb {R}}^{2N}\setminus (\Omega ^c\times \Omega ^c),\text { d}x\text {d}y\right)$, and $\Omega ^c$ is the complement of $\Omega$ in ${\mathbb {R}}^N$. Consider fractional Sobolev space

$$\begin{aligned} H^s({\mathbb {R}}^N):=\left\{ u\in L^2({\mathbb {R}}^N):\frac{|u(x)-u(y)|}{|x-y|^{\frac{N}{2}+s}} \in L^2({\mathbb {R}}^{2N})\right\} , \end{aligned}$$

equipped the Gagliardo seminorm

$$\begin{aligned}{}[u]^2_{H^s({\mathbb {R}}^N)}:=\frac{C_{N,s}}{2}\int _{{\mathbb {R}}^{2N}} \frac{|u(x)-u(y)|^2}{|x-y|^{N+2s}}\text {d}x\text {d}y. \end{aligned}$$

Define the fractional Sobolev space

$$\begin{aligned} H^s(\Omega ):= & {} \left\{ x\in H^s({\mathbb {R}}^N):u=0\ \text { a.e. in } \Omega ^c\right\} , \\ \Vert u\Vert _{H^s(\Omega )}:= & {} \left( \lambda \int _\Omega |u|^2\text {d}x+\frac{C_{N,s}}{2} \int _{{\mathbb {R}}^{2N}\setminus (\Omega ^c\times \Omega ^c)} \frac{|u(x)-u(y)|^2}{|x-y|^{N+2s}}\text {d}x\text {d}y\right) ^\frac{1}{2}, \end{aligned}$$

which was introduced in [14]. From $u=0$ a.e. in $\Omega ^c$, it is easy to see that

$$\begin{aligned} |u|_2^2:=\int _\Omega |u|^2\text {d}x&=\int _{{\mathbb {R}}^N}|u|^2\text {d}x,\\ \int _{{\mathbb {R}}^{2N}\setminus (\Omega ^c\times \Omega ^c)} \frac{|u(x)-u(y)|^2}{|x-y|^{N+2s}}\text {d}x\text {d}y&=\int _{{\mathbb {R}}^{2N}} \frac{|u(x)-u(y)|^2}{|x-y|^{N+2s}}\text {d}x\text {d}y. \end{aligned}$$

Hence, we just denote $\Vert u\Vert _{H^s(\Omega )}$ by

$$\begin{aligned} \Vert u\Vert _{H^s}:=\left( \lambda \int _{{\mathbb {R}}^N}|u|^2\text {d}x+\frac{C_{N,s}}{2}\int _{{\mathbb {R}}^{2N}} \frac{|u(x)-u(y)|^2}{|x-y|^{N+2s}}\text {d}x\text {d}y\right) ^\frac{1}{2}. \end{aligned}$$

and $\big (H^s(\Omega ),\Vert \cdot \Vert _{H^s}\big )$ is a Hilbert space.

Indeed, the solutions of problem (1) correspond to critical points of the $C^1-$functional $J:H^s(\Omega )\times H^s(\Omega )\rightarrow {\mathbb {R}}$ given by

$$\begin{aligned} \begin{aligned} J(u,v)=&\frac{2^*_s-1}{2}\Vert u\Vert ^2_{D^s}-\frac{2^*_s-1}{2}\lambda \int _{\Omega }|u|^2\text {d}x+\frac{1}{2}\Vert v\Vert ^2_{D^s}\\&-\frac{1}{2^*_s}\mu \int _{\Omega }|v|^{2^*_s}\text {d}x-\int _{\Omega }|u|^{2^*_s-2}uv\text {d}x. \end{aligned} \end{aligned}$$

We are interested in nontrivial solutions of (1), namely solutions $(u,v)\in H^s(\Omega )\times H^s(\Omega )$ with both $u\not \equiv 0$ and $v\not \equiv 0$, especially positive ground states of (1). As it is known, the ground state solutions are the solutions of (1) that minimize J on the Nehari manifold

$$\begin{aligned} {\mathcal {N}}=\bigg \lbrace (u,v)\in H^s(\Omega )\times H^s(\Omega )\setminus \{(0,0)\}:G(u,v)=(0,0)\bigg \rbrace , \end{aligned}$$

where

$$\begin{aligned} \begin{aligned} G(u,v)=\left( \Vert u\Vert ^2_{D^s}-\lambda \int _{\Omega }|u|^2\text {d}x-\int _{\Omega }|u|^{2^*_s-2}uv\text {d}x,\right. \\ \left. \Vert v\Vert ^2_{D^s}-\mu \int _{\Omega }|v|^{2^*_s}\text {d}x-\int _{\Omega }|u|^{2^*_s-2}uv\text {d}x\right) . \end{aligned} \end{aligned}$$

The paper is organized as follows. In Sect. 2, we consider the limit case $(\Omega ={\mathbb {R}}^N$ and $\lambda =0)$. We provide results concerning the limiting problem for $N>4s$ and the remaining problem $N=4s$. Finally, in Sect. 3, we investigate the existence of ground states for problem (1) and we prove our main result.

Theorem 1.1

If $\mu > 0$ and $\lambda \in (0,\lambda _1(\Omega ))$, then problem (1) has a ground state solution.

To estimate the energy levels of J, now, we consider the limit system

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su=|u|^{2_s^*-2}v, &{}x \in {\mathbb {R}}^N,\ \\ (-\Delta )^sv=\mu |v|^{2_s^*-2}v+|u|^{2_s^*-2}u, &{}x\in {\mathbb {R}}^N,\\ u,v, \in D^s({\mathbb {R}}^N). \end{array}\right. } \end{aligned}$$

(2)

We search for nontrival solutions of (2) as critical points of the functional

$$\begin{aligned} J_0(u,v)=\frac{2^*_s-1}{2}\Vert u\Vert ^2_{D^s}+\frac{1}{2}\Vert v\Vert ^2_{D^s}-\frac{1}{2^*_s}\mu \int _{{\mathbb {R}}^N }|v|^{2^*_s}\text {d}x-\int _{{\mathbb {R}}^N }|u|^{2^*_s-2}uv\text {d}x \end{aligned}$$

defined in $D^s({\mathbb {R}}^N)\times D^s({\mathbb {R}}^N)$. In particular, we investigate ground state solutions of (2) of the form $(ku_\epsilon ,lu_\epsilon )$ with $k,l>0$, where the definition of $u_\epsilon$ is given by (3) in Sect. 2. Therefore, we consider

$$\begin{aligned} {\mathcal {N}}_0:=\bigg \lbrace (u,v)\in \big (D^s({\mathbb {R}}^N)\times D^s({\mathbb {R}}^N)\big )\setminus \{(0,0)\}:G_0(u,v)=(0,0)\bigg \rbrace , \end{aligned}$$

where

$$\begin{aligned} \begin{aligned} G_0(u,v)=\left( \Vert u\Vert ^2_{D^s}-\int _{{\mathbb {R}}^N }|u|^{2^*_s-2}uv\text {d}x,\right. \\ \left. \Vert v\Vert ^2_{D^s}-\mu \int _{{\mathbb {R}}^N }|v|^{2^*_s}\text {d}x-\int _{{\mathbb {R}}^N }|u|^{2^*_s-2}uv\text {d}x\right) \end{aligned} \end{aligned}$$

and

$$\begin{aligned} {\mathcal {N}}^\prime _0:=\bigg \lbrace (u,v)\in \big (D^s({\mathbb {R}}^N)\times D^s({\mathbb {R}}^N)\big )\setminus \{(0,0)\}:H_0(u,v)=0\bigg \rbrace , \end{aligned}$$

where

$$\begin{aligned} H_0(u,v)=(2^*_s-1)\Vert u\Vert ^2_{D^s}+\Vert v\Vert ^2_{D^s}-2^*_s\int _{{\mathbb {R}}^N }|u|^{2^*_s-2}uv\text {d}x-\mu \int _{{\mathbb {R}}^N }|v|^{2^*_s}\text {d}x. \end{aligned}$$

It is known that ${\mathcal {N}}_0$ and ${\mathcal {N}}^\prime _0$ are of class $C^1$. For the limit case, we define that $A:=\inf \limits _{(u,v)\in {\mathcal {N}}_0}J_0(u,v)$ and $A^\prime :=\inf \limits _{(u,v)\in {\mathcal {N}}^\prime _0}J_0(u,v)$, then we obtain the following.

Theorem 1.2

Suppose that $N>4s$ and $\mu > 0$ hold. Let $\epsilon >0$, then $\left( m_0^\frac{1}{2^*_s-2}u_\epsilon ,m_0^\frac{3-2^*_s}{2^*_s-2}u_\epsilon \right)$ is a ground state solution of (2) and

$$\begin{aligned} J_0\left( m_0^\frac{1}{2^*_s-2}u_\epsilon ,m_0^\frac{3-2^*_s}{2^*_s-2}u_\epsilon \right) =A=A^\prime =\frac{s}{N}\big ((2^*_s-1)k_0^2+l_0^2\big )S_s^{N/{(2s)}}, \end{aligned}$$

where $\left( k_0,l_0\right)$ is a solution of (5).

Theorem 1.3

Suppose that $N=4s$ and $\mu > 0$ hold. Let $\epsilon >0$, then $\left( \sqrt{m_0}u_\epsilon ,\frac{1}{\sqrt{m_0}}u_\epsilon \right)$ is a ground state solution of (2) and

$$\begin{aligned} J_0\left( \sqrt{m_0}u_\epsilon ,\frac{1}{\sqrt{m_0}}u_\epsilon \right) =A=A^\prime =\frac{1}{4}(3{\tilde{k}}^2+{\tilde{l}}^2)S_s^2, \end{aligned}$$

where ${\tilde{k}},{\tilde{l}}$ is the unique solution of (5).

2 The limit problem

Before starting to prove, let us clarify some facts. Let $S_s$ be the best constant, such that

$$\begin{aligned} S_s := \inf \limits _{u\in D^s({\mathbb {R}}^N)\setminus \{0\}}\frac{\Vert u\Vert ^2_{D^s}}{|u|^2_{2^*_s, {\mathbb {R}}^N}}, \end{aligned}$$

then $S_s$ is attained by

$$\begin{aligned} {\widetilde{u}}(x)=k(\epsilon ^2+|x-x_0|)^{\frac{N-2s}{2}}, \end{aligned}$$

that is

$$\begin{aligned} S_s=\frac{\Vert {{\widetilde{u}}}\Vert ^2_{D^s}}{|{{\widetilde{u}}}|^2_{2^*_s,{\mathbb {R}}^N}}. \end{aligned}$$

Normalizing ${\widetilde{u}}$ by $| {\widetilde{u}}|_{2^*_s,{\mathbb {R}}^N}$, we obtain that

$$\begin{aligned} {\overline{u}}=\frac{ {\widetilde{u}}}{| {\widetilde{u}}|_{2^*_s,{\mathbb {R}}^N}}. \end{aligned}$$

Thus

$$\begin{aligned} S_s=\inf \limits _{\begin{array}{c} u\in D^s({\mathbb {R}}^N)\\ |u|_{2^*_s,{\mathbb {R}}^N=1} \end{array}}\Vert u\Vert ^2_{D^s}=\Vert {\overline{u}}\Vert ^2_{D^s}, \end{aligned}$$

and ${\overline{u}}$ is a positive ground state solution of

$$\begin{aligned} (-\Delta )^su=S_s|u|^{2^*_s-2}u\quad \text {in }{\mathbb {R}}^N. \end{aligned}$$

Let

$$\begin{aligned} u_\epsilon (x)=\epsilon ^{-\frac{N-2s}{2}}u_1(\frac{x}{\epsilon }), \end{aligned}$$

(3)

where $u_1=S_s^{\frac{1}{2^*_s-2}}{\overline{u}}$ is a positive ground state solution of

$$\begin{aligned} (-\Delta )^su=|u|^{2^*_s-2}u\quad \text {in}\ {\mathbb {R}}^N, \end{aligned}$$

satisfying

$$\begin{aligned} \Vert u_1\Vert ^2_{D^s}=|u_1|^{2^*_s}_{2^*_s,\ {\mathbb {R}}^N}=S_s^{N/{(2s)}}. \end{aligned}$$

2.1 The limit problem for $N>4s$

Define a function $f_N:(0,+\infty )\rightarrow {\mathbb {R}}$

$$\begin{aligned} f_N(m)=m^{2^*_s-1}-m^{2^*_s-3}+\mu . \end{aligned}$$

Then, the function $f_N$ is strictly increasing and satisfies

$$\begin{aligned} \lim \limits _{m\rightarrow 0^+}f_N(m)=-\infty \ \text {and} \lim \limits _{m\rightarrow +\infty }f_N(m)=+\infty . \end{aligned}$$

Lemma 2.1

$f_N$ has at least one zero point. Let $k,l>0$ satisfy

$$\begin{aligned} (2^*_s-1)k^2+l^2\le 2^*_sk^{2^*_s-1}l+\mu l^{2^*_s}. \end{aligned}$$

(4)

Considering the system

$$\begin{aligned} {\left\{ \begin{array}{ll} k^{2^*_s-3}l=1,\\ \mu l^{2^*_s-1}+k^{2^*_s-1}=l,\\ k,l>0, \end{array}\right. } \end{aligned}$$

(5)

then

$$\begin{aligned} (2^*_s-1)k^2_0+l^2_0=\min \limits _{i=1,2\dots ,n}\{(2^*_s-1)k^2_i+l^2_i\}\le (2^*_s-1)k^2+l^2, \end{aligned}$$

where $(k_i,l_i)$ are solutions of system (5), and $(k_0,l_0)$ is a particular solution of system (5).

Proof

Multiplying the second equation of system (5) by $l^{1-2^*_s}$, and then brought the first equation of system (5) into it and simplifying, we obtain

$$\begin{aligned} \left( \frac{k}{l}\right) ^{2^*_s-1}-\left( \frac{k}{l}\right) ^{2^*_s-3}+\mu =0. \end{aligned}$$

Obviously, $f_N$ has a finite number of solutions and system (5) has some solutions correspondingly.

(i) If $f_N$ has a unique zero point $m_1$, then system (5) has a unique solution denoted as $(k_1,l_1)$.

(ii) If $f_N$ has n zero points, which are denoted as $m_i(i=1,2,\ldots ,n)$, then system (5) has n solutions correspondingly denoted as

$$\begin{aligned} (k_i,l_i)=\left( m_i^{\frac{1}{2^*_s-2}},m_i^{\frac{3-2^*_s}{2^*_s-2}}\right) . \end{aligned}$$

Assume $m_1<m_2<\dots <m_n$, then there exists a minimum one, which is denoted as $(k_0,l_0):=\left( m_0^{\frac{1}{2^*_s-2}},m_0^{\frac{3-2^*_s}{2^*_s-2}}\right)$. Then, we have

$$\begin{aligned} (2^*_s-1)k_0^2+l_0^2:=\min \limits _{i=1,2,\dots ,n}\{(2^*_s-1)k_i^2+l_i^2\}. \end{aligned}$$

Fix $k,l>0$ satisfying (4) and

$$\begin{aligned} k_i=kh^{\frac{1}{2^*_s-2}},\quad l_i=lh ^{\frac{1}{2^*_s-2}}, \end{aligned}$$

where $h:= \frac{(2^*_s-1)k^2+l^2}{l(2^*_sk^{2^*_s-1}+\mu l^{2^*_s-1})}$. Then

$$\begin{aligned} \frac{k_i}{l_i}=\frac{k}{l}, \end{aligned}$$

so $(k_i,l_i)$ are solutions of system (5). Since

$$\begin{aligned} 0<k_i\le k,\ 0<l_i\le l, \end{aligned}$$

we conclude that

$$\begin{aligned} (2^*_s-1)k_i^2+l_i^2 \le (2^*_s-1)k^2+l^2. \end{aligned}$$

Thus

$$\begin{aligned} (2^*_s-1)k^2_0+l^2_0=\min \limits _{i=1,2\ldots ,n}\{(2^*_s-1)k^2_i+l^2_i\}\le (2^*_s-1)k^2+l^2. \end{aligned}$$

$\square$

Proof of Theorem 1.2

For $(au_\epsilon ,bu_\epsilon )\in {\mathcal {N}}_0$, we know that $G_0(au_\epsilon ,bu_\epsilon )=(0,0)$ , that is

$$\begin{aligned} {\left\{ \begin{array}{ll} \Vert au_\epsilon \Vert ^2_{D^s}-\int _{{\mathbb {R}}^N }|au_\epsilon |^{2^*_s-1}bu_\epsilon \text {d}x=0,\\ \Vert bu_\epsilon \Vert ^2_{D^s}-\mu \int _{{\mathbb {R}}^N }|bu_\epsilon |^{2^*_s}\text {d}x-\int _{{\mathbb {R}}^N }|au_\epsilon |^{2^*_s-1}bu_\epsilon \text {d}x=0, \end{array}\right. } \end{aligned}$$

it yields that

$$\begin{aligned} {\left\{ \begin{array}{ll} \frac{\Vert u_\epsilon \Vert ^2_{D^s}}{\int _{{\mathbb {R}}^N }| u_\epsilon |^{2^*_s}\text {d}x} =a^{2^*_s-3}b,\\ b^{2-2^*_s}\cdot \frac{\Vert u_\epsilon \Vert ^2_{D^s}}{\int _{{\mathbb {R}}^N }|u_\epsilon |^{2^*_s}\text {d}x}-\mu -\frac{a^{2^*_s-1}}{b^{2^*_s-1}}=0, \end{array}\right. } \end{aligned}$$

then

$$\begin{aligned} \left( \frac{a}{b}\right) ^{2^*_s-1} -\left( \frac{a}{b}\right) ^{2^*_s-3}+\mu =0. \end{aligned}$$

Since $f_N$ admits a minimum nontrivial zero point $m_0$, then $m_0=\frac{a}{b}$. We derive

$$\begin{aligned} a= & {} \left[ m_0\Vert u_\epsilon \Vert ^2_{D^s}\left( \int _{{\mathbb {R}}^N }|u_\epsilon |^{2^*_s}\text {d}x\right) ^{-1}\right] ^{\frac{1}{2^*_s-2}}, \\ b= & {} \left[ m_0^{3-2^*_s}\Vert u_\epsilon \Vert ^2_{D^s}\left( \int _{{\mathbb {R}}^N }|u_\epsilon |^{2^*_s}\text {d}x\right) ^{-1}\right] ^{\frac{1}{2^*_s-2}}, \end{aligned}$$

which ensures that

$$\begin{aligned} \begin{aligned} \left( au_\epsilon , bu_\epsilon \right) \in {\mathcal {N}}_0, \end{aligned} \end{aligned}$$

and system (5) has a solution $( k_0,l_0) =\left( m_0^{\frac{1}{2^*_s-2}},m_0^{\frac{3-2^*_s}{2^*_s-2}}\right)$. Since ${\mathcal {N}}_0\subset {\mathcal {N}}^\prime _0$, then $A^\prime \le A$. Also, by $J_0^\prime \left( k_0u_\epsilon ,l_0u_\epsilon \right) =0$ and $\left( k_0u_\epsilon ,l_0u_\epsilon \right) \in {\mathcal {N}}_0\subset {\mathcal {N}}_0^\prime$, we have

$$\begin{aligned} A^\prime \le A\le J_0\left( m_0^\frac{1}{2^*_s-2}u_\epsilon ,m_0^{\frac{3-2^*_s}{2^*_s-2}}u_\epsilon \right) =\frac{s}{N}\left( (2^*_s-1)k_0^2+l_0^2\right) S_s^{N/{(2s)}}. \end{aligned}$$

Take $\{(u_n,v_n)\}\subset {\mathcal {N}}^\prime _0$ a minimizing sequence, we get $J_0(u_n,v_n)\rightarrow A^\prime$. Using Sobolev embedding and Hölder inequality, we deduce that

$$\begin{aligned} \begin{aligned} S_s&\left[ (2^*_s-1)|u_n|^2_{2^*_s}+|v_n|^2_{2^*_s}\right] \\&\le (2^*_s-1)\Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s}\\&=2^*_s\int _{{\mathbb {R}}^N}|u_n|^{2^*_s-1}v_n\text {d}x+\mu \int _{{\mathbb {R}}^N}|v_n|^{2^*_s}\text {d}x\\&\le 2^*_s\int _{{\mathbb {R}}^N }|u_n|^{\frac{2^*_s-1}{2^*_s}}\text {d}x\int _{{\mathbb {R}}^N }|v_n|^{2^*_s}\text {d}x+\mu \int _{{\mathbb {R}}^N }|v_n|^{2^*_s}\text {d}x. \end{aligned} \end{aligned}$$

Thereby, we obtain that

$$\begin{aligned} \begin{aligned}&(2^*_s-1)\left( S_s^{\frac{2s-N}{4s}}|u_n|_{2^*_s}\right) ^2+\left( S_s^{\frac{2s-N}{4s}}|v_n|_{2^*_s}\right) ^2\\&\quad \le 2^*_s\left( S_s^{\frac{2s-N}{4s}}|u_n|_{2^*_s}\right) ^{2^*_s-1}\left( S_s^{\frac{2s-N}{4s}}|v_n|_{2^*_s}\right) +\mu \left( S_s^{\frac{2s-N}{4s}}|u_n|_{2^*_s}\right) ^{2^*_s}; \end{aligned} \end{aligned}$$

by Lemma 2.1, it is easy to verify that

$$\begin{aligned} (2^*_s-1)k_0^2+l_0^2\le S_s^{1-\frac{N}{2s}}\left[ (2^*_s-1)|u_n|^2_{2^*_s}+|v_n|^2_{2^*_s}\right] , \end{aligned}$$

which leads to

$$\begin{aligned} \begin{aligned} A^\prime&+o_n(1)=J_0(u_n,v_n)\\&=\frac{2^*_s-1}{2}\Vert u_n\Vert ^2_{D^s}+\frac{1}{2}\Vert v_n\Vert ^2_{D^s}-\frac{1}{2^*_s}\mu \int _{{\mathbb {R}}^N }|v_n|^{2^*_s}\text {d}x-\int _{{\mathbb {R}}^N }|u_n|^{2^*_s-2}u_nv_n\text {d}x\\&=\frac{s}{N}\left[ (2^*_s-1)\Vert u_n\Vert ^2_{D^s}+\Vert v_n|^2_{D^s}\right] \\&\ge \frac{s\cdot S_s}{N}\left[ (2^*_s-1)\left( \int _{{\mathbb {R}}^N }|u_n|^{2^*_s}\text {d}x\right) ^{\frac{2}{2^*_s}}+\left( \int _{{\mathbb {R}}^N }|v_n|^{2^*_s}\text {d}x\right) ^{\frac{2}{2^*_s}}\right] \\&\ge \frac{s}{N}\left( (2^*_s-1)k_0^2+l_0^2\right) S_s^{N/{(2s)}}. \end{aligned} \end{aligned}$$

It follows that:

$$\begin{aligned} A^\prime =\frac{s}{N}\left( (2^*_s-1)k_0^2+l_0^2\right) S_s^{N/{(2s)}}. \end{aligned}$$

Thus, $\left( m_0^\frac{1}{2^*_s-2}u_\epsilon ,m_0^{\frac{3-2^*_s}{2^*_s-2}}u_\epsilon \right)$ is a ground state solution of system (2). $\square$

2.2 The limit problem for $N=4s$

In this subsection, we consider the limit problem for a general $N=4s$. We notice that in the previous subsection, the key points consist of the existence of a zero of the function $f_N$ and the solution of the system (5). Similarly as before, it is easy to see that

$$\begin{aligned} f(m)=m^3-m+\mu ,\ m>0, \\ {\left\{ \begin{array}{ll} kl=1,\\ \mu l^3+k^3=l,\\ k,l>0. \end{array}\right. } \end{aligned}$$

To prove Theorem 1.3, we give the following property.

Proposition 2.2

Suppose that $\mu \in \bigg [0,\frac{\sqrt{3}}{10}\bigg )$ holds.

(i) For $\mu =0$, ${\mathcal {N}}^\prime _0$ does not contain semitrivial couples.

(ii) For $\mu \in (0,\frac{\sqrt{3}}{10})$, ${\mathcal {N}}^\prime _0$ does not contain semitrivial couples (u, 0) and $A^\prime < \inf \limits _{(0,v)\in {\mathcal {N}}^\prime _0}J_0(0,v)$.

Proof

(i) For $\mu =0$, a straightforward computation yields that

$$\begin{aligned} H_0(u,v)=(2^*_s-1)\Vert u\Vert ^2_{D^s}+\Vert v\Vert ^2_{D^s}-2^*_s\int _{{\mathbb {R}}^{N}}|u|^{2^*_s-2}uv\text {d}x. \end{aligned}$$

For any $(u,v)\in {\mathcal {N}}^\prime _0$, if $u=0$ and $v\ne 0$, then $H_0(u,v)=H_0(0,v)=\Vert v\Vert ^2_{D^s}$, which is a contradiction with the definition of ${\mathcal {N}}^\prime _0$. Likewise, if $v=0$, $u\ne 0$, we also get a contradition.

(ii) It is obvious that if $\mu \in (0,\frac{\sqrt{3}}{10})$, ${\mathcal {N}}^\prime _0$ does not contain semitrivial couples (u, 0). Next, we prove the second part of (ii). For any $(0,v)\in {\mathcal {N}}^\prime _0$, we get

$$\begin{aligned} H_0(0,v)=\Vert v\Vert ^2_{D^s}-\mu \int _{{\mathbb {R}}^{N}}|v|^{2^*_s}\text {d}x=0 \end{aligned}$$

and

$$\begin{aligned} J_0(0,v)=\frac{1}{2}\Vert v\Vert ^2_{D^s}-\frac{1}{2^*_s}\mu \int _{{\mathbb {R}}^{N}}|v|^{2^*_s}\text {d}x=\frac{1}{2}\Vert v\Vert ^2_{D^s}-\frac{1}{2^*_s}\Vert v\Vert ^2_{D^s}=\frac{1}{4}\Vert v\Vert ^2_{D^s}. \end{aligned}$$

For every $r>0$, $\big (t(r)rv,t(r)v\big )\in {\mathcal {N}}^\prime _0$ with $t(r)=\left( \frac{(3r^2+1)\mu }{4r^3+\mu }\right) ^{\frac{1}{2}}$, and then

$$\begin{aligned} A^\prime \le J_0\Big (t(r)rv,t(r)v\Big )=\frac{1}{4}\frac{(3r^2+1)^2\mu }{4r^3+\mu }\Vert v\Vert ^2_{D^s}; \end{aligned}$$

so, according to the definition of infimum, we deduce that

$$\begin{aligned} A^\prime \le \frac{(3r^2+1)^2\mu }{4r^3+\mu }\inf \limits _{(0,v)\in {\mathcal {N}}^\prime _0}J_0(0,v). \end{aligned}$$

For $\mu \in (0,\frac{\sqrt{3}}{10})$ and $r=\frac{3}{10\mu }$, we have $\frac{(3r^2+1)^2\mu }{4r^3+\mu }<1$; therefore, $A^\prime <\inf \limits _{(0,v)\in {\mathcal {N}}^\prime _0}J_0(0,v)$. $\square$

Proof of Theorem 1.3

By the same argument as the proof of Theorem 1.2, for every $(au_\epsilon ,bu_\epsilon )\in {\mathcal {N}}_0$, we have $G_0(au_\epsilon ,bu_\epsilon )=(0,0)$, that is

$$\begin{aligned} {\left\{ \begin{array}{ll} \Vert au_\epsilon \Vert ^2_{D^s}-\int _{{\mathbb {R}}^{4s} }|au_\epsilon |^3bu_\epsilon \text {d}x=0,\\ \Vert bu_\epsilon \Vert ^2_{D^s}-\mu \int _{{\mathbb {R}}^{4s} }|bu_\epsilon |^4\text {d}x-\int _{{\mathbb {R}}^{4s} }|au_\epsilon |^3bu_\epsilon \text {d}x=0; \end{array}\right. } \end{aligned}$$

similarly, as in the proof of Theorem 1.2, it is easy to check that

$$\begin{aligned} a=\left[ m_0\Vert u_\epsilon \Vert ^2_{D^s}\left( \int _{{\mathbb {R}}^{4s} }|u_\epsilon |^4\text {d}x\right) ^{-1}\right] ^{\frac{1}{2}}, \\ b=\left[ \frac{1}{m_0}\Vert u_\epsilon \Vert ^2_{D^s}\left( \int _{{\mathbb {R}}^{4s} }|u_\epsilon |^4\text {d}x\right) ^{-1}\right] ^{\frac{1}{2}}. \end{aligned}$$

Then, system (5) has a minimum solution $\left( {\tilde{k}},{\tilde{l}}\right) =\left( \sqrt{m_0},\frac{1}{\sqrt{m_0}}\right)$. Since ${\mathcal {N}}_0\subset {\mathcal {N}}^\prime _0$, one has

$$\begin{aligned} A^\prime \le A \le J_0\left( \sqrt{m_0}u_\epsilon ,\frac{1}{\sqrt{m_0}}u_\epsilon \right) =\frac{1}{4}(3{\tilde{k}}^2+{\tilde{l}}^2)S_s^2. \end{aligned}$$

For $\mu \in [0,\frac{\sqrt{3}}{10})$, let $\{(u_n,v_n)\}\subset {\mathcal {N}}^\prime _0$ be a minimizing sequence, which implies $J_0(u_n,v_n)\rightarrow A^\prime$. By proposition 2.2, we assume $u_n\ne 0$ and $v_n\ne 0$. Then. from Lemma 2.1

$$\begin{aligned} \begin{aligned} A^\prime +o_n(1)&=J_0(u_n,v_n)=\frac{1}{4}\big (3\Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s}\big )\\&\ge \frac{1}{4}S_s\left( 3|u_n|^2_{2^*_s}+|v_n|^2_{2^*_s} \right) \\&\ge \frac{1}{4}(3{\tilde{k}}^2+{\tilde{l}}^2)S_s^2. \end{aligned} \end{aligned}$$

Therefore, $A^\prime =\frac{1}{4}(3{\tilde{k}}^2+{\tilde{l}}^2)S_s^2$ and $\left( \sqrt{m_0}u_\epsilon , \frac{1}{\sqrt{m_0}}u_\epsilon \right)$ is a nontrivial ground state solution of (2). $\square$

3 Positive ground states for (1)

In this section, we study the existence of ground state solutions of problem (1) and we will give the proof of Theorem 1.1. Before proving the main result, we will give some Lemmas that will be used throughout this section. Since

$$\begin{aligned} G^\prime (u,v)[u,v]=\left( (2-2^*_s)(\Vert u\Vert ^2_{D^s}-\int _{\Omega }|u|^2\text {d}x), (2-2^*_s)\int _{\Omega }|v|^{2^*_s}\text {d}x \right) \ne (0,0) \end{aligned}$$

for all $(u,v)\in {\mathcal {N}}$, we get that ${\mathcal {N}}$ is a $C^1$-manifold, where ${\mathcal {N}}$ is defined in (2).

Lemma 3.1

Assume that $\lambda \in \left( 0,\lambda _1(\Omega )\right)$ and $\mu >0$ hold, then ${\mathcal {N}}\ne \emptyset$.

Proof

For given $u\in H^s(\Omega )$, $u>0$, we denote $\theta =\frac{\Vert u\Vert ^2_{D^s}}{\Vert u\Vert ^2_{D^s}-\lambda \Vert u\Vert ^2_{D^s}}$, ${\overline{\theta }}:=\frac{\Vert u\Vert ^2_{D^s}-\lambda \Vert u\Vert ^2_{D^s}}{\int _{\Omega }|u|^{2^*_s}\text {d}x}.$ Then, let $m_0$ be a strictly positive solution of

$$\begin{aligned} m^{2^*_s-1}-\theta m^{2^*_s-3}+\mu =0; \end{aligned}$$

there holds

$$\begin{aligned} \left( (m_0{\overline{\theta }})^{\frac{1}{2^*_s-2}}u,(m_0^{3-2^*_s}{\overline{\theta }})^{\frac{1}{2^*_s-2}}u\right) \in {\mathcal {N}}. \end{aligned}$$

$\square$

Denote

$$\begin{aligned} {\mathcal {B}}:=\inf \limits _{w\in \Gamma }\max \limits _{t\in [0,1]}J(w(t)), \end{aligned}$$

where $\Gamma :=\left\{ w\in C\big ( [0,1], H^s(\Omega )\times H^s(\Omega )\big ) : w(0)=(0,0), J(w(1))<0\right\}$.

Lemma 3.2

Assume that $\lambda >0$ and $\mu > 0$ hold, then ${\mathcal {B}}<A$.

Proof

To prove ${\mathcal {B}}<A$, we may assume that $0\in \Omega$ without loss of generality. Then, there exists $r>0$, such that $\overline{B_r}(0)\subset \Omega$. Let $\phi \in C^1_0(\Omega )$ be a non-negative function with $\phi \equiv 1$ on $\overline{B_r}(0)$. For any $\epsilon >0$, define $U_\epsilon :=\phi U_{\epsilon ,0}$. By [14], we obtain that

$$\begin{aligned} \Vert U_\epsilon \Vert ^2_{D^s}=S_s^{N/{(2s)}}+O(\epsilon ^{N-2s}), |U_\epsilon |^{2^*_s}_{2^*_s}=S_s^{N/{(2s)}}+O(\epsilon ^N) \end{aligned}$$

and

$$\begin{aligned} \Vert U_\epsilon \Vert ^2_{D^s}\ge C\psi _N(\epsilon ) \end{aligned}$$

for some $C>0$, where

$$\begin{aligned} \psi _N(\epsilon )= {\left\{ \begin{array}{ll} \epsilon ^{2s}+O(\epsilon ^{N-2s})\ &{}\text {if} \ N>4s,\\ \epsilon ^{2s}|\log \epsilon |+O(\epsilon ^{2s})\ &{}\text {if}\ N=4s. \end{array}\right. } \end{aligned}$$

Define $(u_\epsilon ,v_\epsilon ):=(kU_\epsilon ,lU_\epsilon )$, where $(k,l)\in {\mathbb {R}}^2$, $k,l>0$ and $(kU_{\epsilon ,0},lU_{\epsilon ,0})$ is a ground state solution of the limit problem (2). Then

$$\begin{aligned} \Vert u_\epsilon \Vert ^2_{D^s}=k^2S_s^{N/{(2s)}}+O\left( \epsilon ^{N-2s}\right) , \Vert v_\epsilon \Vert ^2_{D^s}=l^2S_s^{N/{(2s)}}+O \left( \epsilon ^{N-2s}\right) , \\ |u_\epsilon |^{2^*_s}_{2^*_s}=l^{2^*_s}S_s^{N/{(2s)}}+O(\epsilon ^N),\int _{\Omega }u_\epsilon ^{2^*_s-1}v_\epsilon \text {d}x=k^{2^*_s-1}lS_s^{N/{(2s)}}+O \left( \epsilon ^N \right) \end{aligned}$$

and

$$\begin{aligned} \Vert u_\epsilon \Vert ^2_{D^s}\ge C\psi _N(\epsilon )+O \left( \epsilon ^{N-2s}\right) . \end{aligned}$$

It is clear that

$$\begin{aligned} (2^*_s-1)k^2+l^2=2^*_sk^{2^*_s-1}l+\mu l^{2^*_s}; \end{aligned}$$

we have

$$\begin{aligned} \begin{aligned} J(tu_\epsilon ,tv_\epsilon )=&\frac{2^*_s-1}{2}\Vert tu_\epsilon \Vert ^2_{D^s}-\frac{2^*_s-1}{2}\lambda \int _{\Omega }|tu_\epsilon |^2\text {d}x+\frac{1}{2}\Vert tv_\epsilon \Vert ^2_{D^s}\\&-\frac{1}{2^*_s}\mu \int _{\Omega }|tv_\epsilon |^{2^*_s}\text {d}x-\int _{\Omega }|tu_\epsilon |^{2^*_s-1}tv_\epsilon \text {d}x\\ \le&\frac{1}{2}t^2\left[ \big ( (2^*_s-1)k^2+l^2\big ) S_s^{N/{(2s)}}-\lambda C\psi _N(\epsilon )+O(\epsilon ^{N-2s})\right] \\&-\frac{1}{2^*_s}t^{2^*_s}\left[ \big ( \left( 2^*_s-1 \right) k^2+l^2\big ) S_s^{N/{(2s)}}+O \left( \epsilon ^N \right) \right] \\ =&\frac{1}{2}t^2\left( \frac{NA}{s}-\lambda C\psi (\epsilon )+O \left( \epsilon ^{N-2}\right) \right) -\frac{1}{2^*_s}t^{2^*_s}\left( \frac{NA}{s}+O \left( \epsilon ^N \right) \right) . \end{aligned} \end{aligned}$$

Consider

$$\begin{aligned} h(t):=\frac{t^2}{2}a_\epsilon -\frac{t^{2^*_s}}{2^*_s}b_\epsilon , \end{aligned}$$

where

$$\begin{aligned} a_\epsilon =\frac{NA}{s}-\lambda C\psi (\epsilon )+O \left( \epsilon ^2 \right) ,\ b_\epsilon =\frac{NA}{s}+O \left( \epsilon ^N \right) . \end{aligned}$$

Obviously, for $\epsilon >0$ and small enough

$$\begin{aligned} \max \limits _{t>0}h(t)=\frac{s}{N}\left( \frac{a_\epsilon }{b_\epsilon ^{(N-2s)/N}}\right) ^\frac{N}{2s}<A, \end{aligned}$$

thus

$$\begin{aligned} {\mathcal {B}}\le \max \limits _{t>0}J(tu_\epsilon ,tv_\epsilon )<A. \end{aligned}$$

$\square$

Now, we define some notions which will be useful in the paper.

$$\begin{aligned} {\mathcal {N}}^\prime =\,\,\bigg \lbrace (u,v)\in \big (H^s(\Omega )\times H^s(\Omega )\big ) \setminus \{(0,0)\}:H(u,v)=0\bigg \rbrace , \end{aligned}$$

where

$$\begin{aligned} \begin{aligned} H(u,v)=&(2^*_s-1)\Vert u\Vert ^2_{D^s}-(2^*_s-1)\lambda \int _{\Omega }|u|^2\text {d}x+\Vert v\Vert ^2_{D^s}\\ {}&-\mu \int _{\Omega }|v|^{2^*_s}\text {d}x-2^*_s\int _{\Omega }|u|^{2^*_s-2}uv\text {d}x \end{aligned} \end{aligned}$$

and

$$\begin{aligned} {\mathcal {A}}:=\left\{ (u,v)\in \big ( H^s(\Omega )\times H^s(\Omega )\big ) :\mu \int _{\Omega }|v|^{2^*_s}\text {d}x+2^*_s\int _{\Omega }|u|^{2^*_s-2}uv\text {d}x>0\right\} \end{aligned}$$

the set of admissible pairs. Moreover, if $\lambda \in (0,\lambda _1(\Omega ))$, for all $(u,v)\in {\mathcal {N}}^\prime$, we have that ${\mathcal {N}}^\prime$ is a $C^1$-manifold being. Notice that ${\mathcal {N}}\subset {\mathcal {N}}^\prime \subset {\mathcal {A}}$, and for some constant $C>0$, we have

$$\begin{aligned} H(u,v)\ge \Vert (u,v)\Vert ^2-C\Vert (u,v)\Vert ^{2^*_s}, \end{aligned}$$

(6)

where

$$\begin{aligned} \Vert (u,v)\Vert ^2:=\Vert u\Vert ^2_{D^s}-\lambda \int _{\Omega }|u|^2\text {d}x+\frac{1}{2^*_s-1}\Vert v\Vert ^2_{D^s}. \end{aligned}$$

Proposition 3.3

Assume $\lambda \in (0,\lambda _1(\Omega ))$ and $\mu > 0$ hold, then

$$\begin{aligned} \inf \limits _{(u,v)\in {\mathcal {N}}^\prime }J(u,v)=\inf \limits _{(u,v)\in {\mathcal {A}}}\max \limits _{t\ge 0}J(tu,tv)={\mathcal {B}}>0. \end{aligned}$$

Proof

Taking $(u,v)\in {\mathcal {A}}$, if $H({\tilde{t}}u,{\tilde{t}}v)=0$, then

$$\begin{aligned} \begin{aligned} H({\tilde{t}}u,{\tilde{t}}v)=\,\,&(2^*_s-1){\tilde{t}}^2\Vert u\Vert ^2_{D^s}-(2^*_s-1)\lambda {\tilde{t}}^2\int _{\Omega }|u|^2\text {d}x+{\tilde{t}}^2\Vert v\Vert ^2_{D^s}\\&-\mu {\tilde{t}}^{2^*_s}\int _{\Omega }|v|^{2^*_s}\text {d}x-2^*_s{\tilde{t}}^{2^*_s}\int _{\Omega }|u|^{2^*_s-1}v\text {d}x\\ =\,\,&{\tilde{t}}^2\left[ (2^*_s-1)\Vert u\Vert ^2_{D^s}-(2^*_s-1)\lambda \int _{\Omega }|u|^2\text {d}x+\Vert v\Vert ^2_{D^s}\right. \\&\left. -\mu {\tilde{t}}^{2^*_s-2}\int _{\Omega }|v|^{2^*_s}\text {d}x-2^*_s{\tilde{t}}^{2^*_s-2}\int _{\Omega }|u|^{2^*_s-2}uv\text {d}x\right] , \end{aligned} \end{aligned}$$

that is

$$\begin{aligned} \begin{aligned}&(2^*_s-1)\Vert u\Vert ^2_{D^s}-(2^*_s-1)\lambda \int _{\Omega }|u|^2\text {d}x+\Vert v\Vert ^2_{D^s}\\&\quad ={\tilde{t}}^{2^*_s-2}\left( \mu \int _{\Omega }|v|^{2^*_s}\text {d}x+2^*_s\int _{\Omega }|u|^{2^*_s-1}v\text {d}x\right) ; \end{aligned} \end{aligned}$$

thus

$$\begin{aligned} \begin{aligned} {\tilde{t}}=&\left[ \bigg ( (2^*_s-1)\Vert u\Vert ^2_{D^s}-(2^*_s-1)\lambda \int _{\Omega }|u|^2\text {d}x+\Vert v\Vert ^2_{D^s}\bigg )\right. \\&\left. \left( \mu \int _{\Omega }|v|^{2^*_s}\text {d}x+2^*_s\int _{\Omega }|u|^{2^*_s-1}v\text {d}x\right) ^{-1}\right] ^{\frac{1}{2^*_s-2}}; \end{aligned} \end{aligned}$$

we derive $\left( {\tilde{t}}u,{\tilde{t}}v\right) \in {\mathcal {N}}^\prime$ and $J\left( {\tilde{t}}u,{\tilde{t}}v\right) \ge \inf \limits _{(u,v)\in {\mathcal {N}}^\prime }J(u,v)$. For any $(u,v)\in {\mathcal {A}}$, there exists $t>0$, such that $J(tu,tv)<0$, therefore

$$\begin{aligned} \inf \limits _{(u,v)\in {\mathcal {A}}}\max \limits _{t\ge 0}J(tu,tv)\ge {\mathcal {B}}. \end{aligned}$$

(7)

On the other hand, for any $(u,v)\in {\mathcal {N}}^\prime$, we have ${\tilde{t}}=1$ and

$$\begin{aligned} \inf \limits _{(u,v)\in {\mathcal {N}}^\prime }J(u,v)\ge \inf \limits _{(u,v)\in {\mathcal {A}}}\max \limits _{t\ge 0}J(tu,tv). \end{aligned}$$

(8)

Taking $w=(w_1,w_2)\in \Gamma$, then for a small t, $H(w(t))>0$ and

$$\begin{aligned} H(w(1))=2J(w(1))-\frac{2s}{N}\left[ \mu \int _{\Omega }|w_2(1)|^{2^*_s}\text {d}x+2^*_s\int _{\Omega }|w_1(1)|^{2^*_s-1}w_2(1)\text {d}x\right] <0, \end{aligned}$$

which means that there exists $t^\prime >0$, such that $H(w(t^\prime ))=0$, i.e., $w(t^\prime )\in {\mathcal {N}}^\prime$. Thereby

$$\begin{aligned} {\mathcal {B}}\ge \inf \limits _{(u,v)\in {\mathcal {N}}^\prime }J(u,v). \end{aligned}$$

(9)

Combining (7)–(9), there holds

$$\begin{aligned} \inf \limits _{(u,v)\in {\mathcal {N}}^\prime }J(u,v)=\inf \limits _{(u,v)\in {\mathcal {A}}}\max \limits _{t\ge 0}J(tu,tv)={\mathcal {B}}. \end{aligned}$$

We prove ${\mathcal {B}}>0$. If $J(u_n,v_n)\rightarrow 0$ and $(u_n,v_n)\in {\mathcal {N}}^\prime$, then $\Vert (u_n,v_n)\Vert \rightarrow 0$ which is a contradiction with the inequality (6). Therefore, we have

$$\begin{aligned} \inf \limits _{(u,v)\in {\mathcal {N}}^\prime }J(u,v)={\mathcal {B}}>0 \end{aligned}$$

. $\square$

Now, we show a preliminary property before we prove the main result of this section.

Proposition 3.4

Assume $\lambda \in (0,\lambda _1(\Omega ))$ and $\mu > 0$ hold. Then, every ground state solution of (1) is nontrivial.

Proof

Assume $(u,v)\in {\mathcal {N}}$, such that $J(u,v)=\inf \limits _{(u,v)\in {\mathcal {N}}}J$. If $v=0$, then $\langle J^\prime (u,0),(u,0)\rangle =0$ implies $u=0$. Now, suppose that $u=0$. If $\mu =0$, then $v=0$. So, let $\mu >0$ and v is a nontrivial solution to

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^s v=\mu |v|^{2^*_s-2}v, &{}x\in \ \Omega ,\\ v=0,&{}x\in {\mathbb {R}}^N\setminus \Omega . \end{array}\right. } \end{aligned}$$

Observe that

$$\begin{aligned} \begin{aligned}&\inf \left\{ J(0,w):w\in H^s(\Omega )\setminus \{0\}, \Vert w\Vert ^2_{D^s}=\mu \int _{\Omega }|w|^{2^*_s}\text {d}x\right\} \le J(0,v)=\inf \limits _{{\mathcal {N}}}J\\ \le&\inf \left\{ J(0,w):w\in H^s(\Omega )\setminus \{0\},\Vert w\Vert ^2_{D^s}=\mu \int _{\Omega }|w|^{2^*_s}\text {d}x\right\} \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&\inf \left\{ J(0,w):w\in H^s(\Omega )\setminus \{0\},\ \Vert w\Vert ^2_{D^s}=\mu \int _{\Omega }|w|^{2^*_s}\text {d}x\right\} \\&\quad =\frac{s}{N}\inf \left\{ \Vert w\Vert ^2_{D^s}:w\in H^s(\Omega )\setminus \{0\},\Vert w\Vert ^2_{D^s}=\mu \int _{\Omega }|w|^{2^*_s}\text {d}x\right\} \\&\quad =\frac{s}{N\mu ^{(N-2s)/(2s)}}\inf \left\{ \Vert w\Vert ^N_{D^s}:w\in H^s(\Omega ), |w|_{2^*_s}=1\right\} . \end{aligned} \end{aligned}$$

Then, ${\tilde{v}}=\left( \frac{\mu }{\Vert v\Vert ^2_{D^s}}\right) ^{\frac{1}{2^*_s}}v$ satisfies $\Vert {\tilde{v}}\Vert _{2^*_s}=1$ and

$$\begin{aligned} \Vert v\Vert ^N_{D^s}=\frac{N}{s\mu ^{(N-2s)/(2s)}}J(0,v)=\inf \{\Vert w\Vert ^N_{D^s}:w\in H^s(\Omega ), |w|_{2^*_s}=1\}, \end{aligned}$$

which is a contradiction. Therefore, the ground state solutions of (1) are nontrivial. $\square$

Theorem 3.5

Assume $\lambda \in (0,\lambda _1(\Omega ))$ and $\mu > 0$ hold, then there exists a ground state (u, v) of $J(u,v)=\inf \limits _{{\mathcal {N}}}J=\inf \limits _{{\mathcal {N}}^\prime }J={\mathcal {B}}$.

Proof

By the Sobolev and Poincar$\acute{e}$ inequalities, we know that

$$\begin{aligned} J(u,v)\ge C \left( \Vert u\Vert ^2_{D^s}+\Vert v\Vert ^2_{D^s}-\Vert v\Vert ^{2^*_s}_{D^s}-\Vert u\Vert ^{2^*_s-1}_{D^s}\Vert v\Vert _{D^s}\right) \ge d \end{aligned}$$

for some $d>0$ and $\rho =\sqrt{\Vert u\Vert ^2_{D^s}+\Vert v\Vert ^2_{D^s}}$ sufficiently small.

For any $(u,v)\in H^s(\Omega )\times H^s(\Omega )$ satisfying

$$\begin{aligned} \mu \int _{\Omega }|v|^{2^*_s}\text {d}x+2^*_s\int _{\Omega }|u|^{2^*_s-1}v\text {d}x>0, \end{aligned}$$

we obtain that

$$\begin{aligned} \begin{aligned} J(tu,tv)=\,\,&\frac{t^2}{2}\left[ (2^*_s-1)\Vert u\Vert ^2_{D^s}-(2^*_s-1)\lambda \int _{\Omega }|u|^2\text {d}x+\Vert v\Vert ^2_{D^s}\right] \\&-t^{2^*_s}\left( \frac{\mu }{2^*_s}\int _{\Omega }|v|^{2^*_s}\text {d}x-\int _{\Omega }|u|^{2^*_s-1}v\text {d}x\right) \rightarrow -\infty ,\ \text {as}\ t\rightarrow +\infty . \end{aligned} \end{aligned}$$

Therefore, there exists a $(PS)_{{\mathcal {B}}}$-sequence $\{(u_n,v_n)\}\in H^s(\Omega )\times H^s(\Omega )$ for J at level ${\mathcal {B}}$, namely, a sequence, such that $J(u_n,v_n)\rightarrow {\mathcal {B}}$ and $J^\prime (u_n,v_n)\rightarrow 0$. There holds

$$\begin{aligned} \begin{aligned} C \left( \Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s}\right) \le&J(u_n,v_n)-\frac{1}{2^*_s}\langle J^\prime (u_n,v_n),(u_n,v_n)\rangle \\ \le&\left( {\mathcal {B}} +1 \right) +\sqrt{\Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s}} \end{aligned} \end{aligned}$$

for some constant $C>0$, which implies the sequence $\{(u_n,v_n)\}$ is bounded. Thus, consider a weakly convergent subsequence, it follows from Sobolev embedding theorem that there exists $(u,v)\in H^s(\Omega )\times H^s(\Omega )$, such that:

$$\begin{aligned} \begin{aligned}&u_n\rightharpoonup u\ \text {in}\ H^s(\Omega ),\qquad u_n\rightarrow u\ \text {in}\ L^{2}(\Omega ),\qquad u_n\rightarrow u \ a.e.\ \text {on}\ \Omega ,\\&v_n\rightharpoonup v\ \text {in}\ H^s(\Omega ),\qquad v_n\rightarrow v\ a.e.\ \text {on}\ \Omega ,\\&|u_n|^{2^*_s-1}\rightharpoonup |u|^{2^*_s-1} \qquad \qquad \text {in} \ L^{2^*_s/(2^*_s-1)}(\Omega ),\\&|v_n|^{2^*_s-1}\rightharpoonup |v|^{2^*_s-1}\ \qquad \qquad \text {in}\ L^{2^*_s/(2^*_s-1)}(\Omega ), \\&|u_n|^{2^*_s-3}u_nv_n\rightharpoonup |u|^{2^*_s-3}uv\quad \text {in} \ L^{2^*_s/(2^*_s-1)}(\Omega ). \end{aligned} \end{aligned}$$

In fact, for any $(\xi ,\eta )\in H^s(\Omega )\times H^s(\Omega )$, we have

$$\begin{aligned} \begin{aligned} |\langle J^\prime&(u_n,v_n),(\xi ,\eta )\rangle -\langle J^\prime (u,v),(\xi ,\eta )\rangle |\\ =&\bigg | (2^*_s-1)( \Vert u_n\Vert _{D^s}-\Vert u\Vert _{D^s})\cdot \Vert \xi \Vert _{D^s}-(2^*_s-1)\lambda \int (u_n-u)\xi \text {d}x\\&-(2^*_s-1)\int (|u_n|^{2^*_s-2}v_n-|u|^{2^*_s-2}v)\eta \text {d}x+(\Vert v_n\Vert _{D^s}-\Vert v\Vert _{D^s})\cdot \Vert \eta \Vert _{D^s}\\&-\mu \int (|v_n|^{2^*_s-1}-|v|^{2^*_s-1})\eta \text {d}x-\int (|u_n|^{2^*_s-1}-|u|^{2^*_s-1}) \eta \text {d}x \bigg | \rightarrow 0. \end{aligned} \end{aligned}$$

Therefore, $J^\prime (u,v)=0$. We claim that $(u,v)\ne (0,0)$. Otherwise

$$\begin{aligned} u_n\rightarrow 0\ \text {in}\ L^{2}(\Omega ). \end{aligned}$$

(10)

Since J is continuous and $J(u_n,v_n)\rightarrow {\mathcal {B}}>0$, then $(u_n,v_n)$ cannot converge to (0, 0) in $H^s(\Omega )\times H^s(\Omega )$. Thus, up to a subsequence, we may assume that $(u_n,v_n)\ne (0,0)$ and $\Vert (u_n,v_n)\Vert \ge C>0$, $(u_n,v_n)\in {\mathcal {A}}$ for all $n\in {\mathbb {N}}$. Taking a subsequence $\{(u_{n_k},v_{n_k})\}$ of $\{(u_n,v_n)\}$ in $(H^s(\Omega )\times H^s(\Omega ))\cap {\mathcal {A}}^c$, then

$$\begin{aligned} \langle J^\prime (u_{n_k},v_{n_k}),(u_{n_k}, v_{n_k})\rangle \ge \Vert (u_{n_k},v_{n_k})\Vert ^2. \end{aligned}$$

Thus, there exists a contradiction with

$$\begin{aligned} \langle J^\prime (u_{n_k},v_{n_k}),(u_{n_k},v_{n_k})\rangle \rightarrow 0 \ \text {as}\ k\rightarrow +\infty . \end{aligned}$$

Choose

$$\begin{aligned} t_n=\left[ \bigg ((2^*_s-1)\Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s}\bigg )\left( \mu \int _{\Omega }|v_n|^{2^*_s}\text {d}x+2^*_s\int _{\Omega }|u_n|^{2^*_s-1}v_n\text {d}x\right) ^{-1}\right] ^{\frac{1}{2^*_s-2}}, \end{aligned}$$

and we denote in the same way the functions in $H^s(\Omega )$ and their extensions in ${\mathbb {R}}^N$ putting the function equal to zero in ${\mathbb {R}}^N\setminus \Omega$, we have $(t_nu_n,t_nv_n)\in {\mathcal {N}}^\prime _0$, and so

$$\begin{aligned} \langle J^\prime _0(t_nu_n,t_nv_n),\ (t_nu_n,t_nv_n)\rangle =0. \end{aligned}$$

(11)

Recalling (10), we conclude that

$$\begin{aligned} \langle J^\prime _0(u_n,v_n),\ (u_n,v_n)\rangle =\langle J^\prime (u_n,v_n),\ (u_n,v_n)\rangle +o(1)=o(1). \end{aligned}$$

(12)

Thus, from (11) and (12), we get $t_n\rightarrow 1$. By Lemma 3.2 and Theorem 1.2 or Theorem 1.3, we obtain

$$\begin{aligned} {\mathcal {B}} <A=A^\prime \le \lim \limits _{n}J(t_nu_n,t_nv_n)={\mathcal {B}} \end{aligned}$$

which is a contradiction. Thus, $(u,v)\ne (0,0)$ and $(u,v)\in {\mathcal {N}}\subset {\mathcal {N}}^\prime$. Likewise, we find $t_n\rightarrow 1$, such that $(t_nu_n,t_nv_n)\in {\mathcal {N}}^\prime$, and according to Proposition 3.3, we have

$$\begin{aligned} \inf \limits _{{\mathcal {N}}}J\le J(u,v)\le \lim \limits _{n\rightarrow \infty }J(t_nu_n,t_nv_n)={\mathcal {B}}=\inf \limits _{{\mathcal {N}}^\prime }J\le \inf \limits _{{\mathcal {N}}}J; \end{aligned}$$

thus, we completed the proof of Theorem 3.3. $\square$

To prove that the solutions of problem (1) are positive, we denote $u=u_++u_-$, where $u_+$ and $u_-$ are, respectively, the positive and the negative part of u. We can write

$$\begin{aligned} \begin{aligned} J_+(u,v)=&\frac{2^*_s-1}{2}\Vert u\Vert ^2_{D^s}-\frac{2^*_s-1}{2}\lambda \int _{\Omega }|u|^2\text {d}x+\frac{1}{2}\Vert v\Vert ^2_{D^s}\\&-\frac{1}{2^*_s}\mu \int _{\Omega }|v_+|^{2^*_s}\text {d}x-\int _{\Omega }|u_+|^{2^*_s-1}v\text {d}x, \end{aligned} \end{aligned}$$

which is of $C^1$ class on $H^s(\Omega )\times H^s(\Omega )$. Furthermore, one easily checks that

$$\begin{aligned} \begin{aligned} \langle J^\prime _+(u,v),(\xi ,\eta )\rangle =(&2^*_s-1)\Vert u\Vert _{D^s}\cdot \Vert \xi \Vert _{D^s}-(2^*_s-1)\lambda \int _{\Omega }u\xi \text {d}x\\ -&(2^*_s-1)\int _{\Omega }|u|^{2^*_s-2}v\xi \text {d}x +\Vert v\Vert _{D^s}\cdot \Vert \eta \Vert _{D^s} \\ -&\mu \int _{\Omega }|v_+|^{2^*_s-1}\eta \text {d}x-\int _{\Omega }|u_+|^{2^*_s-1}\eta \text {d}x. \end{aligned} \end{aligned}$$

Lemma 3.6

Assume that for $q>0$, we have $\{(u_n,v_n)\}$ is a $(PS)_c$-sequence for $J_+$. One gets that $\{(u_n,v_n)\}$ is bounded and $\{\big ((u_n)_+,(v_n)_+\big )\}$ is a $(PS)_c$-sequence for $J_+$.

Proof

Taking $\{(u_n,v_n)\}$ a $(PS)_c$-sequence for $J_+$, there exists $C>0$, such that

$$\begin{aligned} \begin{aligned} C(\Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s})\le&J_+(u_n,v_n)-\frac{1}{2^*_s}\langle J^\prime _+(u_n,v_n),(u_n,v_n)\rangle \\ \le&q+1+\sqrt{\Vert u_n\Vert ^2_{D^s}+\Vert v_n\Vert ^2_{D^s}}, \end{aligned} \end{aligned}$$

and so, $\{(u_n,v_n)\}$ is bounded. We observe that

$$\begin{aligned} \begin{aligned} o(1)&=\langle J^\prime _+(u_n,v_n),\ ((u_n)_-,(v_n)_-)\rangle \\&=(2^*_s-1)\Vert u_n\Vert _{D^s}\Vert (u_n)_-\Vert _{D^s}-(2^*_s-1)\lambda \int _{\Omega }|(u_n)_-|^2\text {d}x\\&\ \ \ +\Vert v_n\Vert _{D^s}\Vert (v_n)_-\Vert _{D^s}-\int _{\Omega }(u_n)_+^{2^*_s-1}(v_n)_-\text {d}x\\&\ge C\left( \Vert u_n\Vert _{D^s} \Vert (u_n)_-\Vert _{D^s}+\Vert v_n\Vert _{D^s}\Vert (v_n)_-\Vert _{D^s}\right) , \end{aligned} \end{aligned}$$

and then, $((u_n)_-,(v_n)_-)\rightarrow (0,0) \ \text {in}\ H^s(\Omega )\times H^s(\Omega )$ and $\int _{\Omega }(u_n)_+^{2^*_s-1}(v_n)_-\text {d}x\rightarrow 0$. Therefore

$$\begin{aligned} \begin{aligned}&J_+(u_n,v_n)-J_+((u_n)_+,(v_n)_+)\\&\quad =\frac{2^*_s-1}{2}(\Vert u_n\Vert _{D^s}\Vert (u_n)_-\Vert _{D^s}-\lambda |(u_n)_-|^2)\\&\qquad +\frac{1}{2}\Vert v_n\Vert _{D^s}\Vert (v_n)_-\Vert _{D^s}-\int _{\Omega }(u_n)_+^{2^*_s-1}(v_n)_-\text {d}x \rightarrow 0. \end{aligned} \end{aligned}$$

It is clear that $\{(u_n)_+\}$ is bounded and $\big ((u_n)_-,(v_n)_-\big )\rightarrow (0,0)$ in $H^s(\Omega )$. For any $(\xi ,\eta )\in H^s(\Omega )\times H^s(\Omega )$, we derive that

$$\begin{aligned} \begin{aligned}&\big |\langle J^\prime _+(u_n,v_n)-J^\prime _+((u_n)_+,(v_n)_+),(\xi ,\eta )\rangle \big |\\&\quad =(2^*_s-1)\left[ \Vert (u_n)_-\Vert _{D^s}\Vert \xi \Vert _{D^s}-\lambda \int _{\Omega }(u_n)_-\xi \text {d}x-\int _{\Omega }(u_n)_+^{2^*_s-2}(v_n)_-\xi \text {d}x\right] \\&\qquad +\Vert (v_n)_-\Vert _{D^s}\Vert \eta \Vert _{D^s}\\&\quad \le C\Big (\Vert (u_n)_-\Vert _{D^s}\Vert \xi \Vert _{D^s}+\Vert (v_n)_-\Vert _{D^s}\Vert \eta \Vert _{D^s}+\Vert (u_n)_+\Vert ^{2^*_s-2}_{D^s}\Vert (v_n)_-\Vert _{D^s}\Vert \xi \Vert _{D^s}\Big )\\&\quad \le C\Big (\Vert (u_n)_-\Vert _{D^s}+\left( 1+\Vert (u_n)_+\Vert ^{2^*_s-2}_{D^s}\right) \Vert (v_n)_-\Vert _{D^s}\Big )\big (\Vert \xi \Vert ^2_{D^s}+\Vert \eta \Vert ^2_{D^s}\big )^{\frac{1}{2}}, \end{aligned} \end{aligned}$$

and then, $\Vert J^\prime _+(u_n,v_n)-J^\prime _+((u_n)_+,(v_n)_+)\Vert \rightarrow 0$, that is, $\{(u_n,v_n)\}$ is a $(PS)_c$-sequence for $J_+$. $\square$

Proof of Theorem 1.1

According to the proof of Theorem 3.5, we can know that the functional $J_+$ satisfies the geometrical assumptions of the Mountain Pass Theorem. For the functional $J_+$, there exists a $(PS)_{\mathcal {B}}$-sequence $\{(u_n,v_n)\} \in H^s(\Omega )\times H^s(\Omega )$. By Lemma 3.6, we can assume that $u_n=(u_n)_+$, $v_n=(v_n)_+$ and $\{(u_n,v_n)\}$ is bounded. Since $J(u_n,v_n)=J_+(u_n,v_n)$, we can obtain the same conclusion as Theorem 3.5. Therefore, we get a ground state solution (u, v) of J with $u,v\ge 0$. Then, using the strong maximum principle, we obtain that $u,v>0$. The proof is completed. $\square$