Multiple Positive Solutions for a Fractional Laplacian System with Critical Nonlinearities

Abstract

In this paper, we study the following nonlinear fractional Laplacian system with critical exponent

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s}u=\lambda |u|^{p-2}u+\frac{2\alpha }{\alpha +\beta }|u|^{\alpha -2}u|v|^{\beta }, &{}\quad \hbox {in} \;\ \Omega ,\\ (-\Delta )^{s}v=\mu |v|^{p-2}v+\frac{2\beta }{\alpha +\beta }|u|^{\alpha }|v|^{\beta -2}v, &{} \quad \hbox {in} \;\ \Omega ,\\ u=v=0, &{} \quad \hbox {in} \;\ {\mathbb {R}}^{N}\backslash \Omega , \end{array}\right. \end{aligned}$$

where $\Omega \subset {\mathbb {R}}^{N}$ is a bounded domain with smooth boundary, $0<s<1, 1<p<2, \alpha , \beta >1$ satisfy $\alpha +\beta =2_{s}^{*}, 2_{s}^{*}=\frac{2N}{N-2s}$ is the critical Sobolev exponent, and $N>4s, \lambda , \mu >0$ are parameters. Using the ${\mathcal {N}}$ ehari manifold, fibering maps and the Lusternik–Schnirelmann category, we prove that the problem has at least $\hbox {cat}(\Omega )+1$ distinct positive solutions, where $\hbox {cat} (\Omega )$ denotes the Lusternik–Schnirelmann category of $\Omega$ in itself.

1 Introduction

In the present paper, we consider the existence and multiplicity of positive solutions for the following nonlinear system with fractional Laplacian

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s}u=\lambda |u|^{p-2}u+\frac{2\alpha }{\alpha +\beta }|u|^{\alpha -2}u|v|^{\beta }, &{} \quad \hbox {in} \;\ \Omega ,\\ (-\Delta )^{s}v=\mu |v|^{p-2}v+\frac{2\beta }{\alpha +\beta }|u|^{\alpha }|v|^{\beta -2}v, &{}\quad \hbox {in} \;\ \Omega ,\\ u=v=0, &{} \quad \hbox {in} \;\ {\mathbb {R}}^{N}\backslash \Omega , \end{array}\right. \qquad (H_{\lambda ,\mu }) \end{aligned}$$

where $\Omega \subset {\mathbb {R}}^{N}$ is a bounded domain with smooth boundary, $0<s<1, 1<p<2, \alpha , \beta >1$ satisfy $\alpha +\beta =2_{s}^{*}, 2_{s}^{*}=\frac{2N}{N-2s}$ is the critical Sobolev exponent, and $N>4s, \lambda , \mu >0$ are parameters.

Let ${\mathscr {T}}$ be the Schwartz space of rapidly decaying $C^{\infty }$ functions in ${\mathbb {R}}^{N}$, for any $u\in {\mathscr {T}}$, we have

$$\begin{aligned} (-\Delta )^{s}u(x)= & {} \kappa _{N,s} P.V. \int _{{\mathbb {R}}^{N}}\frac{u(x)-u(y)}{|x-y|^{N+2s}}dy \nonumber \\= & {} \kappa _{N,s}\underset{\varepsilon \rightarrow 0}{\lim } \int _{{\mathscr {C}}B_{\varepsilon }(x)}\frac{u(x)-u(y)}{|x-y|^{N+2s}}dy. \end{aligned}$$

(1.1)

Here P.V. stands for the Cauchy principle value, and $\kappa _{N,s}$ is a dimensional constant that depends only on N and s, precisely given by

$$\begin{aligned} \kappa _{N,s}=\big (\int _{{\mathbb {R}}^{N}}\frac{1-\cos \zeta _{1}}{|\zeta |^{N+2s}}d\zeta \big )^{-1}. \end{aligned}$$

The fractional Laplacian appears in diverse areas including physics, biological modeling and mathematical finances, and partial differential equations involving the fractional Laplacian have attracted the attention of many researches. An important feature of the fraction Laplacian is its nonlocal property, which makes it difficult to handle.

Recently, there are plenty of works on the fractional Laplacian equations. For example, Silvestre [1] established the regularity of solutions for elliptic problems modeling the American putting options with the fractional Laplacian operator. Cabre, Sol$\grave{a}$-Morales and Sire [2, 3] studied layer solutions (solutions which are monotone with respect to one variable) of

$$\begin{aligned} (-\Delta )^{s}u=f(u)\;\;\;\ \hbox {in}\;\ {\mathbb {R}}^{N}, \end{aligned}$$

(1.2)

where $0<s<1, f$ is of balanced bistable type. The conformal geometry involving a fractional Laplacian was studied by Chang and Gonzalez in [4].

At the same time, the elliptic boundary problem driven by the fractional Laplacian operator

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s}u=f(x,u), &{} \quad \hbox {in} \;\ \Omega ,\\ u=0, &{} \quad \hbox {in} \;\ {\mathbb {R}}^{N}\backslash \Omega \end{array}\right. \end{aligned}$$

(1.3)

was also widely studied recently. For example, Servadei and Valdinoci [5] proved (1.3) admits a Mountain Pass type solution which is not identically zero. Wei and Su [6] established the existence of multiple nontrivial solutions for problem (1.3) and give an $L^{\infty }$ regularity result. Moreover, they also studied the case of concave–convex nonlinearities and proved that (1.3) possesses at least six solutions. By exploiting a suitable Trudinger–Moser inequality for fractional Sobolev spaces, Iannizzotto and Squassina [7] showed that (1.3) has infinitely many solutions for the case $N=1$ and $s=\frac{1}{2}$. When $f(x,u)=\lambda u^{q}+u^{2_{s}^{*}-1}$, Barrios etc. [8] obtained the existence and multiplicity of solutions for problem (1.3) with different values of $\lambda$. Additionally, they considered both the concave power case ($0<q<1$) and the convex power case $(1<q<2_{s}^{*}-1)$. For more results, we refer the readers to [9–21] and the references therein.

However, as far as we know, there are few works on problem ($H_{\lambda ,\mu }$) with $0<s<1$. For the case $s=1$ and $u=0$ on $\partial \Omega$, various studies concerning the existence and multiplicity of solutions have been presented in [22–29]. Motivated by the above works, in this paper, we study the multiplicity of solutions for problem ($H_{\lambda ,\mu }$) with $0<s<1$. Note that a direct extension of those methods to the case $0<s<1$ is faced with serious difficulties. For example, one typical feature of the fractional Laplacian operator is nonlocality, in the sense that the value of $(-\Delta )^{s}u(x)$ at any point $x\in \Omega$ depends not only on the value of u on the whole $\Omega$, but actually on the whole ${\mathbb {R}}^{N}$, which makes some discussions and calculations difficult. And the most difficult problem we need to deal with is how to find a critical value in the interval where the $(PS)_{c}$ condition hold. To overcome this difficulty, we apply the idea of [30, Lemma 3.8] and get some useful estimates.

In the following, we state our main result.

Theorem 1.1

There exists $\Lambda ^{*}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$, $(H_{\lambda ,\mu })$ admits at least $\hbox {cat}(\Omega )$+1 distinct positive solutions.

This paper is organized as follows: In Sect. 2, we introduce the environment we will work in and prove some preliminary results. In Sect. 3, we give some technical lemmas which will be useful to exhibit the necessary homotopies. In Sect. 4, we prove Theorem 1.1.

2 Notations and Preliminaries

For any $s\in (0,1)$, we define the fractional Sobolev space $H^{s}({\mathbb {R}}^{N})$ via the Fourier transform

$$\begin{aligned} H^{s}\left( {\mathbb {R}}^{N}\right) =\left\{ u\in L^{2}\left( {\mathbb {R}}^{N}\right) : \int _{{\mathbb {R}}^{N}}\left( 1+|\xi |^{2s}\right) |{\mathscr {F}}(u)|^{2}d\xi <+\infty \right\} , \end{aligned}$$

endowed with the norm

$$\begin{aligned} \Vert u\Vert _{H^{s}({\mathbb {R}}^{N})}=\big (\Vert u\Vert _{L^{2}({\mathbb {R}}^{N})}^{2}+\int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{u}(\xi )|^{2}d\xi \big )^{\frac{1}{2}}, \end{aligned}$$

where $\hat{u}\equiv {\mathscr {F}}(u)$ denotes the Fourier transform of u.

Denote the Gagliardo semi-norm of u by

$$\begin{aligned}{}[u]_{H^{s}({\mathbb {R}}^{N})}=\left( \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2s}}dxdy\right) ^{\frac{1}{2}}. \end{aligned}$$

Then it follows directly from [31, Proposition 3.4] that

$$\begin{aligned} 2\kappa _{N,s}^{-1}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{u}(\xi )|^{2}d\xi =[u]_{H^{s}\left( {\mathbb {R}^{N}}\right) }. \end{aligned}$$

(2.1)

Now, we introduce the closed line subspace

$$\begin{aligned} X(\Omega )=\left\{ u\in H^{s}\left( {\mathbb {R}}^{N}\right) : u(x)=0\;\ \hbox {a.e.}\;\ \hbox {in}\;\ {\mathbb {R}}^{N}\backslash \Omega \right\} , \end{aligned}$$

which can be equivalently renormed by setting

$$\begin{aligned} \Vert u\Vert _{X(\Omega )}=\left( \int _{\mathbb {R}^{N}}|\xi |^{2s}|\hat{u}(\xi )|^{2}d\xi \right) ^{\frac{1}{2}}. \end{aligned}$$

Clearly, $(X(\Omega ),\Vert \cdot \Vert _{X(\Omega )})$ is a uniformly convex Banach space and we have the following embedding result.

Lemma 2.1

[31, Theorem 6.7] Let $0<s<1$ be such that $2s<N$. Then the embedding $X(\Omega )\hookrightarrow L^{s}(\Omega )$ is continuous for any $\varsigma \in [1,2_{s}^{*}]$, and is compact whenever $\varsigma \in [1,2_{s}^{*})$.

In the present paper, we propose to study $(H_{\lambda ,\mu })$ in the framework of the fractional Sobolev space $H=X(\Omega )\times X(\Omega )$ using the standard norm

$$\begin{aligned} \Vert (u,v)\Vert _{H}=(\Vert u\Vert _{X(\Omega )}^{2}+\Vert v\Vert _{X(\Omega )}^{2})^{\frac{1}{2}} =\left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \right) ^{\frac{1}{2}}. \end{aligned}$$

Denote

$$\begin{aligned} S_{\alpha ,\beta }=\underset{u,v\in X(\Omega )\backslash \{0\}}{\inf }\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{2}+|v(x)-v(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|u|^{\alpha }|v|^{\beta }dx\right) ^{\frac{2}{\alpha +\beta }}} \end{aligned}$$

(2.2)

and

$$\begin{aligned} S=\underset{u\in X(\Omega )\backslash \{0\}}{\inf }\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|u|^{2_{s}^{*}}dx\right) ^{\frac{2}{2_{s}^{*}}}}. \end{aligned}$$

(2.3)

Then we have the following result.

Lemma 2.2

$S_{\alpha ,\beta }=\left[ (\frac{\alpha }{\beta })^{\frac{\beta }{\alpha +\beta }}+(\frac{\beta }{\alpha })^{\frac{\alpha }{\alpha +\beta }}\right] S.$

Proof

Suppose $\{w_{n}\}$ is a minimizing sequence for S and let $u_{n}=\sigma _{1}w_{n}$, $v_{n}=\sigma _{2}w_{n}$, $\sigma _{1},$ $\sigma _{2}>0$ will be chosen later. Then, we infer from (2.2) that

$$\begin{aligned} S_{\alpha ,\beta }\le & {} \frac{\left( \sigma _{1}^{2}+\sigma _{2}^{2}\right) \displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|w_{n}(x)-w_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \sigma _{1}^{\alpha } \sigma _{2}^{\beta }\right) ^{\frac{2}{\alpha +\beta }}\left( \displaystyle \int _{\Omega }|w_{n}|^{2_{s}^{*}}dx\right) ^{\frac{2}{2_{s}^{*}}}}\nonumber \\= & {} \left[ \left( \frac{\sigma _{1}}{\sigma _{2}}\right) ^{\frac{2\beta }{\alpha +\beta }} +\left( \frac{\sigma _{2}}{\sigma _{1}}\right) ^{\frac{2\alpha }{\alpha +\beta }}\right] \frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|w_{n}(x)-w_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|w_{n}|^{2_{s}^{*}}dx\right) ^{\frac{2}{2_{s}^{*}}}}.\;\;\;\; \end{aligned}$$

(2.4)

Define the function

$$\begin{aligned} h(x)=x^{\frac{2\beta }{\alpha +\beta }}+x^{-\frac{2\alpha }{\alpha +\beta }},\quad x>0. \end{aligned}$$

By a direct computation, the minimum of the function h is achieved at the point $x_{0}=(\frac{\alpha }{\beta })^{\frac{1}{2}}$ with the minimum value

$$\begin{aligned} h(x_{0})=\left( \frac{\alpha }{\beta }\right) ^{\frac{\beta }{\alpha +\beta }}+\left( \frac{\beta }{\alpha }\right) ^{\frac{\alpha }{\alpha +\beta }}. \end{aligned}$$

Then, choosing $\sigma _{1}, \sigma _{2}>0$ in (2.4) such that $\frac{\sigma _{1}}{\sigma _{2}}=(\frac{\alpha }{\beta })^{\frac{1}{2}}$, we obtain

$$\begin{aligned} S_{\alpha ,\beta }\le \left[ \left( \frac{\alpha }{\beta }\right) ^{\frac{\beta }{\alpha +\beta }}+\left( \frac{\beta }{\alpha }\right) ^{\frac{\alpha }{\alpha +\beta }}\right] S. \end{aligned}$$

To complete the proof, we let $z_{n}=(\overline{u}_{n},\overline{v}_{n})$ be a minimizing sequence for $S_{\alpha ,\beta }$. Define $\varpi _{n}=\sigma _{n}\overline{v}_{n}$ for some $\sigma _{n}>0$ such that

$$\begin{aligned} \int _{\Omega }|\overline{u}_{n}|^{\alpha +\beta }dx=\int _{\Omega }|\varpi _{n}|^{\alpha +\beta }dx. \end{aligned}$$

Then we have

$$\begin{aligned} \int _{\Omega }|\overline{u}_{n}|^{\alpha }|\varpi _{n}|^{\beta }dx\le & {} \frac{\alpha }{\alpha +\beta }\int _{\Omega }|\overline{u}_{n}|^{\alpha +\beta }dx+ \frac{\beta }{\alpha +\beta }\int _{\Omega }|\varpi _{n}|^{\alpha +\beta }dx\\= & {} \int _{\Omega }|\overline{u}_{n}|^{\alpha +\beta }dx=\int _{\Omega }|\varpi _{n}|^{\alpha +\beta }dx. \end{aligned}$$

Therefore, we deduce from the above inequality that

$$\begin{aligned}&\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|\overline{u}_{n}(x)-\overline{u}_{n}(y)|^{2}+|\overline{v}_{n}(x)-\overline{v}_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|\overline{u}_{n}|^{\alpha }|\overline{v}_{n}|^{\beta }dx\right) ^{\frac{2}{\alpha +\beta }}}\\&\quad = \frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|\overline{u}_{n}(x) -\overline{u}_{n}(y)|^{2}+\sigma _{n}^{-2}|\varpi _{n}(x)-\varpi _{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\sigma _{n}^ {-\frac{2\beta }{\alpha +\beta }}\left( \displaystyle \int _{\Omega }|\overline{u}_{n}|^{\alpha }|\varpi _{n}|^{\beta }dx\right) ^{\frac{2}{\alpha +\beta }}}\\&\quad = \sigma _{n}^{\frac{2\beta }{\alpha +\beta }}\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|\overline{u}_{n}(x) -\overline{u}_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|\overline{u}_{n}|^{\alpha }|\varpi _{n}|^{\beta }dx\right) ^{\frac{2}{\alpha +\beta }}}+ \sigma _{n}^{\frac{-2\alpha }{\alpha +\beta }}\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|\varpi _{n}(x) -\varpi _{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|\overline{u}_{n}|^{\alpha }|\varpi _{n}|^{\beta }dx\right) ^{\frac{2}{\alpha +\beta }}}\\&\quad \ge \sigma _{n}^{\frac{2\beta }{\alpha +\beta }}\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|\overline{u}_{n}(x) -\overline{u}_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|\overline{u}_{n}|^{2_{s}^{*}}dx\right) ^{\frac{2}{2_{s}^{*}}}}+ \sigma _{n}^{\frac{-2\alpha }{\alpha +\beta }}\frac{\displaystyle \int _{{\mathbb {R}}^{2N}}\frac{|\varpi _{n}(x) -\varpi _{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \displaystyle \int _{\Omega }|\varpi _{n}|^{2_{s}^{*}}dx\right) ^{\frac{2}{2_{s}^{*}}}}\\&\quad \ge \sigma _{n}^{\frac{2\beta }{\alpha +\beta }}S+\sigma _{n}^{-\frac{2\alpha }{\alpha +\beta }}S=h(\sigma _{n})S\ge h(x_{0})S. \end{aligned}$$

Passing to the limit in the above inequality, we obtain

$$\begin{aligned} S_{\alpha ,\beta }\ge \left[ \left( \frac{\alpha }{\beta }\right) ^{\frac{\beta }{\alpha +\beta }}+ \left( \frac{\beta }{\alpha }\right) ^{\frac{\alpha }{\alpha +\beta }}\right] S. \end{aligned}$$

This completes the proof. $\square$

We will show the multiplicity of positive solutions for $(H_{\lambda ,\mu })$ by looking for critical points of the associated functional

$$\begin{aligned} I_{\lambda ,\mu }(u,v)= & {} \frac{1}{2}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi -\frac{1}{p}\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx\\&-\frac{2}{\alpha +\beta } \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx, \end{aligned}$$

where $u_{+}=\max \{u,0\}$ and $v_{+}=\max \{v,0\}$.

The nontrivial critical points of the functional $I_{\lambda , \mu }$ are in fact positive weak solutions of $(H_{\lambda ,\mu })$. By a weak solution (u, v) for $(H_{\lambda ,\mu })$, we mean that $(u,v)\in H$ satisfying

$$\begin{aligned}&\int _{\mathbb {R}^{N}}|\xi |^{2s}\big (\hat{u}(\xi )\hat{\varphi }_{1}(\xi )+\hat{v}(\xi )\hat{\varphi }_{2}(\xi )\big )d\xi -\int _{\Omega }\left( \lambda u_{+}^{p-1}\varphi _{1}+\mu v_{+}^{p-1}\varphi _{2}\right) dx\\&\quad -\frac{2\alpha }{\alpha +\beta }\int _{\Omega }u_{+}^{\alpha -1}v_{+}^{\beta }\varphi _{1}dx-\frac{2\beta }{\alpha +\beta }\int _{\Omega }u_{+}^{\alpha } v_{+}^{\beta -1}\varphi _{2}dx=0, \end{aligned}$$

for all $(\varphi _{1},\varphi _{2})\in H$.

As the energy functional $I_{\lambda , \mu }$ is not bounded on H, it is useful to consider the functional on the ${\mathcal {N}}$ ehari manifold

$$\begin{aligned} {\mathcal {N}}_{\lambda ,\mu }:=\left\{ (u,v)\in H\backslash \{(0,0)\} : \;\ \langle I'_{\lambda ,\mu }(u,v),(u,v)\rangle =0\right\} . \end{aligned}$$

Then, $(u,v)\in {\mathcal {N}}_{\lambda ,\mu }$ if and only if

$$\begin{aligned} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi -\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx-2\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx=0. \end{aligned}$$

Note that ${\mathcal {N}}_{\lambda ,\mu }$ contains all positive weak solutions for $(H_{\lambda ,\mu })$.

It is easy to see that if $(u,v)\in {\mathcal {N}}_{\lambda ,\mu }$, then

$$\begin{aligned} I_{\lambda ,\mu }(u,v)= & {} \left( \frac{1}{2}-\frac{1}{p}\right) \int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx+2\left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\= & {} \left( \frac{1}{2}-\frac{1}{p}\right) \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi +2\left( \frac{1}{p}-\frac{1}{2_{s}^{*}}\right) \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\= & {} \left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&+\left( \frac{1}{2_{s}^{*}}-\frac{1}{p}\right) \int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx. \end{aligned}$$

By a direct computation, we infer that the functional $I_{\lambda ,\mu }$ is coercive and bounded below on ${\mathcal {N}}_{\lambda ,\mu }$. Moreover, we have the following result.

Lemma 2.3

For $(u,v)\in {\mathcal {N}}_{\lambda ,\mu }$, there exists a positive constant $C_{0}$ (depending on $s, p, N, S, |\Omega |$) such that $I_{\lambda ,\mu }(u,v)\ge - C_{0}(\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}).$

Proof

For any $(u,v)\in \mathcal {N}_{\lambda ,\mu }$, we deduce from (2.1), (2.3), the H$\ddot{o}$lder inequality and the Young inequality that

$$\begin{aligned} I_{\lambda ,\mu }(u,v)= & {} \left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&+\left( \frac{1}{2_{s}^{*}}-\frac{1}{p}\right) \int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx\\\ge & {} \frac{s}{N}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&-\frac{2_{s}^{*}-p}{2_{s}^{*}p}|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}\left[ \lambda (\int _{\Omega }u_{+}^{2_{s}^{*}}dx)^{\frac{p}{2_{s}^{*}}}+\mu \left( \int _{\Omega }v_{+}^{2_{s}^{*}}dx\right) ^{\frac{p}{2_{s}^{*}}}\right] \\\ge & {} \frac{s}{N}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi -\frac{2_{s}^{*}-p}{2_{s}^{*}p}|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}} S^{-\frac{p}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{p}{2}}\\&\left[ \lambda \left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{u}(\xi )|^{2}d\xi \right) ^{\frac{p}{2}}+\mu \left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{v}(\xi )|^{2}d\xi \right) ^{\frac{p}{2}}\right] \\\ge & {} \frac{s}{N}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi -\frac{2_{s}^{*}-p}{2_{s}^{*}p}|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}} S^{-\frac{p}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{p}{2}}\\&\left[ \frac{2-p}{2}\left( \lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\right) +\frac{p}{2}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \right] \\\ge & {} \frac{s}{N}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&-\frac{s}{N}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi -C_{0}\left( \lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\right) \\= & {} -C_{0}\left( \lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\right) , \end{aligned}$$

where $C_{0}$ is a positive constant depending on $s, p, N, S, \kappa _{N,s}$, and $|\Omega |$.

For $t>0$, we define the fibering maps

$$\begin{aligned} \Phi _{u,v}(t)= & {} \frac{t^{2}}{2}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi \\&-\frac{t^{p}}{p}\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx-\frac{2t^{2_{s}^{*}}}{2_{s}^{*}} \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx. \end{aligned}$$

Then

$$\begin{aligned} \Phi '_{u,v}(t)= & {} t\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi -t^{p-1}\int _{\Omega }( \lambda u_{+}^{p}+\mu v_{+}^{p})dx\\&-2t^{2_{s}^{*}-1} \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\= & {} t^{2_{s}^{*}-1}\left[ t^{2-2_{s}^{*}}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi -t^{p-2_{s}^{*}}\int _{\Omega }( \lambda u_{+}^{p}+\mu v_{+}^{p})dx\right. \\&-\left. 2\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\right] \end{aligned}$$

and

$$\begin{aligned} \Phi ''_{u,v}(t)= & {} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi -(p-1)t^{p-2}\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx\\&-2(2_{s}^{*}-1)t^{2_{s}^{*}-2} \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx. \end{aligned}$$

It is easy to see that $(u,v)\in {\mathcal {N}}_{\lambda ,\mu }$ if and only if $\Phi '_{u,v}(1)=0$, and more generally, $(tu,tv)\in {\mathcal {N}}_{\lambda ,\mu }$ if and only if $\Phi '_{u,v}(t)=0$, that is, the elements in ${\mathcal {N}}_{\lambda ,\mu }$ correspond to stationary points of fibering maps $\Phi _{u,v}(t)$. Thus, for $(u,v)\in {\mathcal {N}}_{\lambda ,\mu }$, we have

$$\begin{aligned} \Phi ''_{u,v}(1)= & {} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi -(p-1)\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx\\&-2(2_{s}^{*}-1) \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\= & {} (2-p)\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi +2(p-2_{s}^{*})\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\= & {} (2-2_{s}^{*})\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi -(p-2_{s}^{*})\int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx. \end{aligned}$$

Therefore, we can split the ${\mathcal {N}}$ ehari manifold ${\mathcal {N}}_{\lambda ,\mu }$ into three parts, that is,

$$\begin{aligned} {\mathcal {N}}_{\lambda ,\mu }^{+}:= & {} \{(u,v)\in {\mathcal {N}}_{\lambda ,\mu } : \;\ \Phi ''_{u,v}(1)>0 \},\\ {\mathcal {N}}_{\lambda ,\mu }^{-}:= & {} \{(u,v)\in {\mathcal {N}}_{\lambda ,\mu } : \;\ \Phi ''_{u,v}(1)<0 \}, \\ {\mathcal {N}}_{\lambda ,\mu }^{0}:= & {} \{(u,v)\in \mathcal {N}_{\lambda ,\mu } :\;\ \Phi ''_{u,v}(1)=0 \}, \end{aligned}$$

corresponding to the local minima, the local maxima, and the points of inflection.

In the sequel, we denote weak convergence by $\rightharpoonup$, and strong convergence by $\rightarrow$, also we use $\Lambda ^{*}$ to denote different small parameters. Then, we have the following lemma. $\square$

Lemma 2.4

There exists $\Lambda ^{*}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$, ${\mathcal {N}}^{0}_{\lambda ,\mu }=\emptyset$.

Proof

Let

$$\begin{aligned} \Lambda ^{*}=\left[ \frac{(2-p)S_{\alpha ,\beta }^{\frac{2_{s}^{*}}{2}}\kappa _{N,s}^{\frac{2_{s}^{*}}{2}}}{2^{\frac{2N-2s}{N-2s}}(2_{s}^{*}-p)}\right] ^{\frac{N-2s}{2s}} \left[ \frac{(2_{s}^{*}-2)S^{\frac{p}{2}}\kappa _{N,s}^{\frac{p}{2}}}{2^{\frac{2+p}{2}}(2_{s}^{*}-p)|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}}\right] ^{\frac{2}{2-p}}. \end{aligned}$$

Arguing by way of contradiction, we suppose that there exists $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$ such that ${\mathcal {N}}^{0}_{\lambda ,\mu }\ne \emptyset$. Then for any $(u,v)\in {\mathcal {N}}^{0}_{\lambda ,\mu }$, we have

$$\begin{aligned} (2-p)\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi =2(2_{s}^{*}-p)\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx \end{aligned}$$

and

$$\begin{aligned} (2_{s}^{*}-2)\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi =(2_{s}^{*}-p)\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx. \end{aligned}$$

By (2.1) and (2.3) and the H $\ddot{o}$ lder inequality, we have

$$\begin{aligned}&2(2_{s}^{*}-p)\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\&\quad \le 2(2_{s}^{*}-p)S_{\alpha ,\beta }^{-\frac{2_{s}^{*}}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{2_{s}^{*}}{2}}\left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+ |\hat{v}(\xi )|^{2})d\xi \right) ^{\frac{2_{s}^{*}}{2}} \end{aligned}$$

and

$$\begin{aligned}&(2_{s}^{*}-p)\int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx\\&\quad \le (2_{s}^{*}-p)|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}S^{-\frac{p}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{p}{2}}(\lambda +\mu )\left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+ |\hat{v}(\xi )|^{2})d\xi \right) ^{\frac{p}{2}}\\&\quad \le 2(2_{s}^{*}-p)|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}S^{-\frac{p}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{p}{2}}(\lambda ^{\frac{2}{2-p}}\\&\qquad +\,\mu ^{\frac{2}{2-p}})^{\frac{2-p}{2}}\left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+ |\hat{v}(\xi )|^{2})d\xi \right) ^{\frac{p}{2}}. \end{aligned}$$

Then

$$\begin{aligned} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \ge \left[ \frac{(2-p)S_{\alpha ,\beta }^{\frac{2_{s}^{*}}{2}}\kappa _{N,s}^{\frac{2_{s}^{*}}{2}}}{2^{\frac{2N-2s}{N-2s}}(2_{s}^{*}-p)}\right] ^{\frac{N-2s}{2s}} \end{aligned}$$

and

$$\begin{aligned} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \le \left[ \frac{2^{\frac{2+p}{2}}(2_{s}^{*}-p)|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}}{(2_{s}^{*}-2) S^{\frac{p}{2}}\kappa _{N,s}^{\frac{p}{2}}}\right] ^{\frac{2}{2-p}}(\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}). \end{aligned}$$

This implies

$$\begin{aligned} \lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\ge \left[ \frac{(2-p)S_{\alpha ,\beta }^{\frac{2_{s}^{*}}{2}}\kappa _{N,s}^{\frac{2_{s}^{*}}{2}}}{2^{\frac{2N-2s}{N-2s}}(2_{s}^{*}-p)}\right] ^{\frac{N-2s}{2s}} \left[ \frac{(2_{s}^{*}-2) S^{\frac{p}{2}}\kappa _{N,s}^{\frac{p}{2}}}{2^{\frac{2+p}{2}}(2_{s}^{*}-p)|\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}} }\right] ^{\frac{2}{2-p}}. \end{aligned}$$

This contradiction shows that there exists a constant $\Lambda ^{*}>0$ such that ${\mathcal {N}}^{0}_{\lambda ,\mu }=\emptyset$ for $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$. Thus, we finish the proof. $\square$

By Lemma 2.4, for $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*}), {\mathcal {N}}_{\lambda ,\mu }={\mathcal {N}}^{+}_{\lambda ,\mu }\cup {\mathcal {N}}^{-}_{\lambda ,\mu }$, and we can define

$$\begin{aligned} \theta ^{+}_{\lambda ,\mu }:=\underset{(u,v)\in {\mathcal {N}}^{+}_{\lambda ,\mu }}{\inf }{I_{\lambda ,\mu }(u,v)}, \;\;\ \theta ^{-}_{\lambda ,\mu }:=\underset{(u,v)\in {\mathcal {N}}^{-}_{\lambda ,\mu }}{\inf }{I_{\lambda ,\mu }(u,v)}. \end{aligned}$$

Subsequently, we have the following results.

Lemma 2.5

Suppose that (u, v) is a local minimizer for $I_{\lambda ,\mu }$ on ${\mathcal {N}}_{\lambda ,\mu }$. Then, if $(u,v)\not \in {\mathcal {N}}_{\lambda ,\mu }^{0}, (u,v)$ is a critical point of $I_{\lambda ,\mu }$ on H.

Proof

If $(u,v)\in {\mathcal {N}}_{\lambda ,\mu }$, then $\langle I'_{\lambda ,\mu }(u,v),(u,v)\rangle =0.$ On the other hand,

$$\begin{aligned} I'_{\lambda ,\mu }(u,v)=\vartheta \Psi '_{\lambda ,\mu }(u,v), \;\ \hbox {for}\;\ \hbox {some}\;\ \vartheta \in {\mathbb {R}}, \end{aligned}$$

(2.5)

where

$$\begin{aligned} \Psi _{\lambda ,\mu }(u,v)= & {} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi \nonumber \\&-\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx-2 \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx. \end{aligned}$$

(2.6)

Note that for any $(u,v)\not \in {\mathcal {N}}^{0}_{\lambda ,\mu }$, there holds

$$\begin{aligned} \langle \Psi '_{\lambda ,\mu }(u,v),(u,v)\rangle \ne 0. \end{aligned}$$

Thus, we deduce from (2.5) that

$$\begin{aligned} 0=\vartheta \langle \Psi '_{\lambda ,\mu }(u,v),(u,v)\rangle , \end{aligned}$$

which implies that $\vartheta =0$, and so $I'_{\lambda ,\mu }(u,v)=0.$ $\square$

Lemma 2.6

There exists $\Lambda ^{*}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$, then

• (i) $\theta ^{+}_{\lambda ,\mu }<0$;

• (ii) $\theta ^{-}_{\lambda ,\mu }\ge d_{0}$ for some $d_{0}>0$.

Proof

• (i) For any $(u,v)\in \mathcal {N}^{+}_{\lambda ,\mu }\subset {\mathcal {N}}_{\lambda ,\mu }$, we have

  $$\begin{aligned} (2-p)\int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi >2(2_{s}^{*}-p)\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx. \end{aligned}$$

  Then

  $$\begin{aligned} I_{\lambda ,\mu }(u,v)= & {} \left( \frac{1}{2}-\frac{1}{p}\right) \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi +2\left( \frac{1}{p}-\frac{1}{2_{s}^{*}}\right) \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx\\<&\left( \frac{1}{2}-\frac{1}{p}\right) \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&+\frac{2-p}{p2_{s}^{*}} \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\= & {} \frac{p-2}{p}\left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\< & {} 0, \end{aligned}$$

  which leads to $\theta ^{+}_{\lambda ,\mu }<0$.

• (ii) For any $(u,v)\in \mathcal {N}^{-}_{\lambda ,\mu }\subset {\mathcal {N}}_{\lambda ,\mu }$, we have

$$\begin{aligned} (2-p)\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi <2\left( 2_{s}^{*}-p\right) \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx. \end{aligned}$$

(2.7)

Combining with (2.1) and (2.2), we deduce

$$\begin{aligned} \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi >\left[ \frac{(2-p)S_{\alpha ,\beta }^{\frac{2_{s}^{*}}{2}} \kappa _{N,s}^{\frac{2_{s}^{*}}{2}}}{2^{\frac{2+2_{s}^{*}}{2}}(2_{s}^{*}-p)}\right] ^{\frac{2}{2_{s}^{*}-2}}, \end{aligned}$$

and then

$$\begin{aligned} I_{\lambda ,\mu }(u,v)= & {} \left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&+\left( \frac{1}{2_{s}^{*}}-\frac{1}{p}\right) \int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx\\\ge & {} \left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \\&+\,2\left( \frac{1}{2_{s}^{*}}-\frac{1}{p}\right) |\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}S^{-\frac{p}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{p}{2}}\left( \lambda ^{\frac{2}{2-p}}\right. \\&+\left. \,\mu ^{\frac{2}{2-p}}\right) ^{\frac{2-p}{2}}\Bigg (\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+ |\hat{v}(\xi )|^{2})d\xi \Bigg )^{\frac{p}{2}} \\\ge & {} \left[ \frac{(2-p)S_{\alpha ,\beta }^{\frac{2_{s}^{*}}{2}} \kappa _{N,s}^{\frac{2_{s}^{*}}{2}}}{2^{\frac{2+2_{s}^{*}}{2}}(2_{s}^{*}-p)}\right] ^{\frac{p}{2_{s}^{*}-2}}\Bigg \{(\frac{1}{2}-\frac{1}{2_{s}^{*}}) \left[ \frac{(2-p)S_{\alpha ,\beta }^{\frac{2_{s}^{*}}{2}} \kappa _{N,s}^{\frac{2_{s}^{*}}{2}}}{2^{\frac{2+2_{s}^{*}}{2}}(2_{s}^{*}-p)}\right] ^{\frac{2-p}{2_{s}^{*}-2}}\\&+\,2\left( \frac{1}{2_{s}^{*}}-\frac{1}{p}\right) |\Omega |^{\frac{2_{s}^{*}-p}{2_{s}^{*}}}S^{-\frac{p}{2}}(2\kappa _{N,s}^{-1})^{\frac{p}{2}}\left( \lambda ^{\frac{2}{2-p}} +\mu ^{\frac{2}{2-p}}\right) ^{\frac{2-p}{2}}\Bigg \}. \end{aligned}$$

Thus, there exists $\Lambda ^{*}>0$ small enough and $d_{0}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$, $\theta ^{-}_{\lambda ,\mu }\ge d_{0}$ for all $(u,v)\in \mathcal {N}^{-}_{\lambda ,\mu }$. This completes the proof. $\square$

In order to get a better understanding of the ${\mathcal {N}}$ ehari manifold and fibering maps, we consider the function $\phi _{u,v}(t): {\mathbb {R}}^{+}\rightarrow {\mathbb {R}}$ defined by

$$\begin{aligned} \phi _{u,v}(t)=t^{2-2_{s}^{*}}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2}\right) d\xi -t^{p-2_{s}^{*}}\int _{\Omega }\left( \lambda u_{+}^{p}+\mu v_{+}^{p}\right) dx. \end{aligned}$$

Then

$$\begin{aligned} \Phi '_{u,v}(t)=t^{2_{s}^{*}-1}\left[ \phi _{u,v}(t)-2\int _{\Omega } u_{+}^{\alpha }v_{+}^{\beta }dx\right] . \end{aligned}$$

Clearly, $\underset{t\rightarrow 0^{+}}{\lim }{\phi _{u,v}(t)}=-\infty , \underset{t\rightarrow +\infty }{\lim }{\phi _{u,v}(t)}=0$, and for each $(u,v)\in H$ with $\int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx>0$, $\phi _{u,v}(t)$ achieves its maximum at

$$\begin{aligned} t_{max}=\bigg [\frac{(2_{s}^{*}-p)\int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx}{(2_{s}^{*}-2)\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi }\bigg ]^{\frac{1}{2-p}}. \end{aligned}$$

Moreover, by a direct computation, we have

$$\begin{aligned} \phi _{u,v}(t_{max})=\frac{2-p}{2_{s}^{*}-p}\left[ \frac{(2_{s}^{*}-2) \big (\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}(\xi )|^{2}+|\hat{v}(\xi )|^{2})d\xi \big )^{\frac{2_{s}^{*}-p}{2_{s}^{*}-2}}}{(2_{s}^{*}-p)\int _{\Omega }(\lambda u_{+}^{p}+\mu v_{+}^{p})dx}\right] ^{\frac{2_{s}^{*}-2}{2-p}}>0, \end{aligned}$$

and $\phi '_{u,v}(t)>0$, for $t\in (0,t_{max}), \phi '_{u,v}(t)<0$, for $t\in (t_{max},+\infty )$.

Then we have the following lemma.

Lemma 2.7

For $(u,v)\in H\backslash \{(0,0)\}$, there exists a unique $0<t^{+}<t_{max}$ such that $(t^{+}u,t^{+}v)\in {\mathcal {N}}^{+}_{\lambda ,\mu }$ and

$$\begin{aligned} I_{\lambda ,\mu }(t^{+}u,t^{+}v)= \underset{t\ge 0}{\inf }{I_{\lambda ,\mu }(tu, tv)}. \end{aligned}$$

Moreover, if $\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx>0$, there are unique $0<t^{+}<t_{max}<t^{-}$ such that $(t^{+}u, t^{+}v)\in \mathcal {N}^{+}_{\lambda ,\mu }, (t^{-}u, t^{-}v)\in \mathcal {N}^{-}_{\lambda ,\mu }$ and

$$\begin{aligned} I_{\lambda ,\mu }(t^{+}u, t^{+}v)=\underset{0\le t\le t_{max}}{\inf }{I_{\lambda ,\mu }(tu, tv)},\;\;\ I_{\lambda ,\mu }(t^{-}u, t^{-}v)=\underset{t\ge 0}{\sup }{I_{\lambda ,\mu }(tu, tv)}. \end{aligned}$$

Proof

The proof is almost the same as that in [22, Lemma 1.5] and therefore we omit it here.

Recall that for $c\in {\mathbb {R}}$, a sequence $\{(u_{n},v_{n})\}\subset H$ is called a $(PS)_{c}$-sequence for the functional $I_{\lambda ,\mu }$ if $I_{\lambda ,\mu }(u_{n},v_{n})=c+o(1)$ and $I'_{\lambda ,\mu }(u_{n},v_{n})=o(1)$ strongly in $H^{-1}$, as $n\rightarrow \infty$. Moreover, if any $(PS)_{c}$-sequence in H for $I_{\lambda ,\mu }$ contains a convergent subsequence, we say that $I_{\lambda ,\mu }$ satisfies the $(PS)_{c}$-condition in H.

By modifying the proof of Lemma 2.7 in [22] or Lemma 2.2 in [26], we can easily get the following result. $\square$

Lemma 2.8

For any $c\in (-\infty ,\frac{2s}{N}(\frac{S_{\alpha ,\beta }}{2})^{\frac{N}{2s}}(\frac{\kappa _{N,s}}{2})^{\frac{N}{2s}}-C_{0}(\lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}}))$, the $(PS)_{c}$-sequence $\{(u_{n},v_{n})\}$ is bounded in H.

Lemma 2.9

$I_{\lambda ,\mu }$ satisfies the $(PS)_{c}$-condition for $c\in (-\infty ,\frac{2s}{N}(\frac{S_{\alpha ,\beta }}{2})^{\frac{N}{2s}} (\frac{\kappa _{N,s}}{2})^{\frac{N}{2s}}-C_{0}(\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}))$.

Proof

Let $\{(u_{n},v_{n})\}\subset H$ be such that

$$\begin{aligned} I_{\lambda ,\mu }(u_{n},v_{n})=c+o(1),\;\;\ I'_{\lambda ,\mu }(u_{n},v_{n})=o(1). \end{aligned}$$

(2.8)

By the boundness of $\{(u_{n},v_{n})\}$ and Lemma 2.1, there exists $(u,v)\in H$ such that

Let $(z_{n}^{1},z_{n}^{2})=(u_{n}-u,v_{n}-v)$. Then by Brezis–Lieb Lemma and the fact $\{(z_{n}^{1},z_{n}^{2})\}\subset H$, we get

$$\begin{aligned} \int _{{\mathbb {R}}^{2N}}\frac{|z_{n}^{1}(x)-z_{n}^{1}(y)|^{2}}{|x-y|^{N+2s}}dxdy= & {} \int _{{\mathbb {R}}^{2N}}\frac{|u_{n}(x)-u_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy\\&-\int _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2s}}dxdy+o(1), \\ \int _{{\mathbb {R}}^{2N}}\frac{|z_{n}^{2}(x)-z_{n}^{2}(y)|^{2}}{|x-y|^{N+2s}}dxdy= & {} \int _{{\mathbb {R}}^{2N}}\frac{|v_{n}(x)-v_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy\\&-\int _{{\mathbb {R}}^{2N}}\frac{|v(x)-v(y)|^{2}}{|x-y|^{N+2s}}dxdy+o(1), \end{aligned}$$

and

$$\begin{aligned} \int _{\Omega }\left( z_{n}^{1}\right) _{+}^{\alpha }\left( z_{n}^{2}\right) _{+}^{\beta }dx=\int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx -\int _{\Omega } u_{+}^{\alpha }v_{+}^{\beta }dx+o(1). \end{aligned}$$

Then for any $(\varphi _{1},\varphi _{2})\in H$, there holds

$$\begin{aligned} \underset{n\rightarrow \infty }{\lim }\langle I'_{\lambda ,\mu }(u_{n},v_{n}),(\varphi _{1},\varphi _{2})\rangle = \langle I'_{\lambda ,\mu }(u,v),(\varphi _{1},\varphi _{2})\rangle =0, \end{aligned}$$

which implies that (u, v) is a critical point of $I_{\lambda ,\mu }$. Moreover, we deduce from (2.8) that

$$\begin{aligned} c-I_{\lambda ,\mu }(u,v)+o(1)= & {} \frac{1}{2}\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{z}_{n}^{1}(\xi )|^{2}+|\hat{z}_{n}^{2}(\xi )|^{2}\right) d\xi \nonumber \\&-\frac{2}{\alpha +\beta }\int _{\Omega }\left( z_{n}^{1}\right) _{+}^{\alpha }\left( z_{n}^{2}\right) _{+}^{\beta }dx \end{aligned}$$

(2.9)

and

$$\begin{aligned} 0= & {} \langle I'_{\lambda ,\mu }(u_{n},v_{n}), (z_{n}^{1},z_{n}^{2})\rangle \\= & {} \langle I'_{\lambda ,\mu }(u_{n},v_{n})-I'_{\lambda ,\mu }(u,v),(z_{n}^{1},z_{n}^{2})\rangle \\= & {} \int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{z}_{n}^{1}(\xi )|^{2}+|\hat{z}_{n}^{2}(\xi )|^{2}\right) d\xi -2\int _{\Omega }\left( z_{n}^{1}\right) _{+}^{\alpha }\left( z_{n}^{2}\right) _{+}^{\beta }dx+o(1). \end{aligned}$$

Without loss of generality, we assume that

$$\begin{aligned}&\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{z}_{n}^{1}(\xi )|^{2}+|\hat{z}_{n}^{2}(\xi )|^{2}\right) d\xi =l+o(1)\;\;\ \hbox {and}\;\;\\&\quad 2\int _{\Omega }\left( z_{n}^{1}\right) _{+}^{\alpha }\left( z_{n}^{2}\right) _{+}^{\beta }dx=l+o(1). \end{aligned}$$

If $l=0$, then we finish the proof. On the contrary, we assume that $l>0$. Then it follows from (2.1) and (2.2) that

$$\begin{aligned} l\le 2S_{\alpha ,\beta }^{-\frac{2_{s}^{*}}{2}}\left( 2\kappa _{N,s}^{-1}\right) ^{\frac{2_{s}^{*}}{2}}l^{\frac{2_{s}^{*}}{2}}, \end{aligned}$$

which leads to

$$\begin{aligned} l\ge 2\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}. \end{aligned}$$

(2.10)

Thus, we obtain from (2.9), (2.10), and Lemma 2.3 that

$$\begin{aligned} c=I_{\lambda ,\mu }(u,v)+\frac{s}{N}l\ge \frac{2s}{N}\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}-C_{0}\left( \lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\right) . \end{aligned}$$

This contradiction shows that $l=0$, that is, $(u_{n},v_{n})\rightarrow (u,v)$ strongly in H.

Then we obtain the existence of a local minimizer for $I_{\lambda ,\mu }$ on ${\mathcal {N}}^{+}_{\lambda ,\mu }$. $\square$

Lemma 2.10

There exists $\Lambda ^{*}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*}), I_{\lambda ,\mu }$ has a minimizer $(u^{+}_{\lambda ,\mu },v^{+}_{\lambda ,\mu })\in {\mathcal {N}}^{+}_{\lambda ,\mu }$ and it satisfies

• (i) $I_{\lambda ,\mu }(u^{+}_{\lambda ,\mu },v^{+}_{\lambda ,\mu })=\theta ^{+}_{\lambda ,\mu }$ and $(u^{+}_{\lambda ,\mu },v^{+}_{\lambda ,\mu })$ is a positive solution of $(H_{\lambda ,\mu })$.

• (ii) $I_{\lambda ,\mu }(u^{+}_{\lambda ,\mu },v^{+}_{\lambda ,\mu })\rightarrow 0$ and $\int _{{\mathbb {R}}^{N}}|\xi |^{2s}(|\hat{u}_{\lambda ,\mu }^{+}(\xi )|^{2}+|\hat{v}_{\lambda ,\mu }^{+}(\xi )|^{2})d\xi \rightarrow 0$ as $\lambda , \mu \rightarrow 0.$

Proof

Using the similar methods to Theorem 3.2 in [24] and Lemma 2.8 in [22], we can easily get the results of this lemma.

It is well known that S given in (2.3) is indeed achieved in the case $\Omega ={\mathbb {R}}^{N}$ and moreover, the function $U_{\epsilon }(x)=\frac{C(N,s)\epsilon ^{\frac{N-2s}{2}}}{(|x|^{2}+\epsilon ^{2})^{\frac{N-2s}{2}}}$ is the only positive radial solution of

$$\begin{aligned} (-\Delta )^{s}u=|u|^{2_{s}^{*}-2}u,\;\;\;\ x\in {\mathbb {R}}^{N}. \end{aligned}$$

Let $0\le \eta (x)\le 1$ be a function in $C_{0}^{\infty }(\Omega )$ defined as

$$\begin{aligned} \eta (x) = \left\{ \begin{array}{lll}1, &{} \quad \hbox {if}\;\ |x|\le \frac{R}{2},\\ 0,&{} \quad \hbox {if}\;\ |x|\ge R, \end{array}\right. \end{aligned}$$

where R is a positive constant satisfying $B(0,R)\subset \Omega$.

For every $\epsilon >0$, we let $u_{\epsilon }(x)=\eta (x)U_{\epsilon }(x)$. Then by standard arguments as [30, Lemma 3.8], we can easily obtain the following estimates. $\square$

Lemma 2.11

Let $s\in (0,1)$ and $N>4s$. Then the following estimates hold true:

$$\begin{aligned} \int _{{\mathbb {R}}^{2N}}\frac{|u_{\epsilon }(x)-u_{\epsilon }(y)|^{2}}{|x-y|^{N+2s}}dxdy\le & {} S^{\frac{N}{2s}}+O\left( \epsilon ^{N-2s}\right) , \\ \int _{\Omega }|u_{\epsilon }|^{2_{s}^{*}}dx= & {} S^{\frac{N}{2s}}+O\left( \epsilon ^{N}\right) ,\\ \int _{\Omega }|u_{\epsilon }|^{p}dx\ge & {} \left\{ \begin{array}{lll}O\left( \epsilon ^{\frac{p(N-2s)}{2}}\right) , &{}1<p<\frac{N}{N-2s},\\ O\left( \epsilon ^{\frac{N}{2}}\hbox {ln}\frac{1}{\epsilon }\right) , &{}p=\frac{N}{N-2s}, \\ O\left( \epsilon ^{N-\frac{p(N-2s)}{2}}\right) , &{}\frac{N}{N-2s}<p<2. \end{array}\right. \end{aligned}$$

Define $v_{\epsilon }(x)=\frac{u_{\epsilon }}{|u_{\epsilon }|_{2_{s}^{*}}}$. Then we have the following result.

Lemma 2.12

There exist $\epsilon ^{*}, \Lambda ^{*}, \varrho (\epsilon )>0$ such that for every $\epsilon \in (0,\epsilon ^{*}),$ $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$ and $\varrho \in (0,\varrho (\epsilon )),$ we have

$$\begin{aligned} \underset{t\ge 0}{\sup } I_{\lambda ,\mu }\left( t\sqrt{\alpha }v_{\epsilon },t\sqrt{\beta }v_{\epsilon }\right) <c_{\lambda ,\mu }-\varrho , \end{aligned}$$

where $c_{\lambda ,\mu }=\frac{2s}{N}\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}-C_{0}\left( \lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}}\right)$.

Proof

By the definition of $v_{\epsilon }$, we consider the function

$$\begin{aligned} g(t)=I_{\lambda ,\mu }(t\sqrt{\alpha }v_{\epsilon },t\sqrt{\beta }v_{\epsilon })= & {} \frac{t^{2}}{2}(\alpha +\beta ) \int _{\mathbb {R}^{N}}|\xi |^{2s}|\hat{v}_{\epsilon }(\xi )|^{2}d\xi \\&-\frac{2t^{2_{s}^{*}}}{2_{s}^{*}}\alpha ^{\frac{\alpha }{2}}\beta ^{\frac{\beta }{2}}-\frac{t^{p}}{p}(\lambda \alpha ^{\frac{p}{2}}+\mu \beta ^{\frac{p}{2}}) \int _{\Omega }|v_{\epsilon }|^{p}dx. \end{aligned}$$

Since $g(0)=0, \underset{t\rightarrow \infty }{\lim }g(t)=-\infty$, there exists $t_{\epsilon }>0$ such that

$$\begin{aligned} \underset{t\ge 0}{\sup }g(t)=g(t_{\epsilon }) \;\;\ \hbox {and}\;\ g'(t_{\epsilon })=0. \end{aligned}$$

That is,

$$\begin{aligned} t_{\epsilon }(\alpha +\beta )\int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{v}_{\epsilon }(\xi )|^{2}d\xi -2t_{\epsilon }^{2_{s}^{*}-1}\alpha ^{\frac{\alpha }{2}} \beta ^{\frac{\beta }{2}}-t_{\epsilon }^{p-1}\left( \lambda \alpha ^{\frac{p}{2}}+\mu \beta ^{\frac{p}{2}}\right) \int _{\Omega }|v_{\epsilon }|^{p}dx=0. \end{aligned}$$

Then

$$\begin{aligned} t_{\epsilon }(\alpha +\beta )\int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{v}_{\epsilon }(\xi )|^{2}d\xi \ge t_{\epsilon }^{p-1}(\lambda \alpha ^{\frac{p}{2}}+\mu \beta ^{\frac{p}{2}})\int _{\Omega }|v_{\epsilon }|^{p}dx, \end{aligned}$$

which leads to

$$\begin{aligned} t_{\epsilon }\ge \bigg [\frac{2\left( \lambda \alpha ^{\frac{p}{2}}+\mu \beta ^{\frac{p}{2}}\right) \int _{\Omega }|v_{\epsilon }|^{p}dx}{(\alpha +\beta )\kappa _{N,s} \int _{{\mathbb {R}}^{2N}}\frac{|v_{\epsilon }(x)-v_{\epsilon }(y)|^{2}}{|x-y|^{N+2s}}dxdy}\bigg ]^{\frac{1}{2-p}}. \end{aligned}$$

(2.11)

By Lemma 2.11 and after direct computation, we obtain

$$\begin{aligned} \int _{{\mathbb {R}}^{2N}}\frac{|v_{\epsilon }(x)-v_{\epsilon }(y)|^{2}}{|x-y|^{N+2s}}dxdy\le S+O\left( \epsilon ^{N-2s}\right) , \end{aligned}$$

(2.12)

$$\begin{aligned} \int _{\Omega }|v_{\epsilon }|^{p}dx\ge \left\{ \begin{array}{lll}O(\epsilon ^{\frac{p(N-2s)}{2}}), &{} \quad 1<p<\frac{N}{N-2s},\\ O(\epsilon ^{\frac{N}{2}}\hbox {ln}\frac{1}{\epsilon }), &{} \quad p=\frac{N}{N-2s}, \\ O(\epsilon ^{N-\frac{p(N-2s)}{2}}), &{} \quad \frac{N}{N-2s}<p<2. \end{array}\right. \end{aligned}$$

Thus, combining with (2.11), there exist constants $T>0$, $\epsilon _{0}>0$ such that for any $\epsilon \in (0,\epsilon _{0})$, we have $t_{\epsilon }\ge T$.

Consider

$$\begin{aligned} m(t)=\frac{t^{2}}{2}(\alpha +\beta )\int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{v}_{\epsilon }(\xi )|^{2}d\xi -\frac{2t^{2_{s}^{*}}}{2_{s}^{*}}\alpha ^{\frac{\alpha }{2}}\beta ^{\frac{\beta }{2}}. \end{aligned}$$

Then we deduce from (2.1), (2.12), and Lemma 2.2 that

$$\begin{aligned} \underset{t\ge 0}{\sup } m(t)= & {} \frac{s}{N}\left[ \frac{(\alpha +\beta )\displaystyle \int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{v}_{\epsilon }(\xi )|^{2}d\xi }{\left( 2\alpha ^{\frac{\alpha }{2}}\beta ^{\frac{\beta }{2}}\right) ^{\frac{2}{2_{s}^{*}}}}\right] ^{\frac{N}{2s}}\\= & {} \frac{s}{2^{\frac{N-2s}{2s}}N}\left[ \left( \frac{\alpha }{\beta }\right) ^{\frac{\beta }{\alpha +\beta }} +\left( \frac{\beta }{\alpha }\right) ^{\frac{\alpha }{\alpha +\beta }}\right] ^{\frac{N}{2s}}\left( \int _{{\mathbb {R}}^{N}}|\xi |^{2s}|\hat{v}_{\epsilon }(\xi )|^{2}d\xi \right) ^{\frac{N}{2s}}\\= & {} \frac{s}{2^{\frac{N-2s}{2s}}N}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}\left[ \left( \frac{\alpha }{\beta }\right) ^{\frac{\beta }{\alpha +\beta }} +\left( \frac{\beta }{\alpha }\right) ^{\frac{\alpha }{\alpha +\beta }}\right] ^{\frac{N}{2s}}\left( \int _{{\mathbb {R}}^{2N}}\frac{|v_{\epsilon }(x) -v_{\epsilon }(y)|^{2}}{|x-y|^{N+2s}}dxdy\right) ^{\frac{N}{2s}}\\\le & {} \frac{s}{2^{\frac{N-2s}{2s}}N}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}\left[ \left( \frac{\alpha }{\beta }\right) ^{\frac{\beta }{\alpha +\beta }} +\left( \frac{\beta }{\alpha }\right) ^{\frac{\alpha }{\alpha +\beta }}\right] ^{\frac{N}{2s}}\left( S+O(\epsilon ^{N-2s})\right) ^{\frac{N}{2s}}\\\le & {} \frac{s}{2^{\frac{N-2s}{2s}}N}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}S_{\alpha ,\beta }^{\frac{N}{2s}}+O\left( \epsilon ^{N-2s}\right) , \end{aligned}$$

where we have used the fact:

$$\begin{aligned} \underset{t\ge 0}{\sup }\left( \frac{t^{2}}{2}A-\frac{t^{2_{s}^{*}}}{2_{s}^{*}}B\right) =\frac{s}{N}\left( \frac{A}{B}\right) ^{\frac{2_{s}^{*}}{2_{s}^{*}-2}}B=\frac{s}{N}\left( \frac{A}{B^{\frac{2}{2_{s}^{*}}}}\right) ^{\frac{N}{2s}},\;\ A, B>0. \end{aligned}$$

Now, we consider the following three cases:

Case I. $1<p<\frac{N}{N-2s}$. For $\epsilon \in (0,\epsilon _{0})$, we have

$$\begin{aligned} g(t_{\epsilon })= & {} m(t_{\epsilon })-\frac{t_{\epsilon }^{p}}{p}(\lambda \alpha ^{\frac{p}{2}}+\mu \beta ^{\frac{p}{2}}) \int _{\Omega }|v_{\epsilon }|^{p}dx\\\le & {} \frac{s}{2^{\frac{N-2s}{2s}}N}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}S_{\alpha ,\beta }^{\frac{N}{2s}}+O(\epsilon ^{N-2s})-\frac{T^{p}}{p}(\lambda \alpha ^{\frac{p}{2}}+\mu \beta ^{\frac{p}{2}}) \int _{\Omega }|v_{\epsilon }|^{p}dx\\\le & {} \frac{2s}{N}\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}+O(\epsilon ^{N-2s})-O\left( \epsilon ^{\frac{p(N-2s)}{2}}\right) . \end{aligned}$$

Since $N-2s>\frac{p(N-2s)}{2}$, we can choose $\epsilon _{1}\in (0,\epsilon _{0})$ small enough, $\Lambda ^{*}$, $\varrho (\epsilon )>0$ such that for $\epsilon \in (0,\epsilon _{1}),$ $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$ and $\varrho \in (0,\varrho (\epsilon )),$ there holds

$$\begin{aligned} O\left( \epsilon ^{N-2s}\right) -O\left( \epsilon ^{\frac{p(N-2s)}{2}}\right) <-\varrho -C_{0}\left( \lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}}\right) , \end{aligned}$$

which leads to conclusion.

Case II. $p=\frac{N}{N-2s}$. Similarly, for $\epsilon \in (0,\epsilon _{0})$, we infer

$$\begin{aligned} g(t_{\epsilon })\le \frac{2s}{N}\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{\frac{N}{2s}}+ O\left( \epsilon ^{N-2s}\right) -O\left( \epsilon ^{\frac{N}{2}}\hbox {ln}\frac{1}{\epsilon }\right) . \end{aligned}$$

Since $N>4s$, we can choose $\epsilon _{2}\in (0,\epsilon _{0})$ small enough, $\Lambda ^{*}$, $\varrho (\epsilon )>0$ such that for $\epsilon \in (0,\epsilon _{2}),$ $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$ and $\varrho \in (0,\varrho (\epsilon )),$ there holds

$$\begin{aligned} O(\epsilon ^{N-2s})-O\left( \epsilon ^{\frac{N}{2}}\hbox {ln}\frac{1}{\epsilon }\right) <-\varrho -C_{0}\left( \lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}}\right) . \end{aligned}$$

Case III. $\frac{N}{N-2s}<p<2$. Since $N>4s$, we have $N-\frac{p(N-2s)}{2}<N-2s$, and so choosing $\epsilon _{3}\in (0,\epsilon _{0})$ small enough, $\Lambda ^{*}$, $\varrho (\epsilon )>0$ such that for $\epsilon \in (0,\epsilon _{3}),$ $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$ and $\varrho \in (0,\varrho (\epsilon )),$ we have

$$\begin{aligned} O\left( \epsilon ^{N-2s}\right) -O\left( \epsilon ^{N-\frac{p(N-2s)}{2}}\right) <-\varrho -C_{0}\left( \lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}}\right) . \end{aligned}$$

Hence, let $\epsilon ^{*}=\min \{\epsilon _{1},\epsilon _{2},\epsilon _{3}\}$ and we complete the proof. $\square$

3 Technical Lemmas

In this section, we prove some technical lemmas that we will need to exhibit the necessary homotopies.

Firstly, we consider the filtration of the manifold ${\mathcal {N}}^{-}_{\lambda ,\mu }$ as follows:

$$\begin{aligned} {\mathcal {N}}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }):=\left\{ (u,v)\in {\mathcal {N}}^{-}_{\lambda ,\mu } : \;\ I_{\lambda ,\mu }(u,v)\le c_{\lambda ,\mu }\right\} . \end{aligned}$$

In Sect. 4, we shall prove that $(H_{\lambda ,\mu })$ possesses at least $\hbox {cat}(\Omega )$ solutions in this set.

Now, we recall some related definitions of the Lusternik–Schnirelmann category.

Definition 3.1

• (i) For a topological space H, we say that a nonempty, closed subset $X\subset H$ is contractible to a point in H if and only if there exists a continuous map

  $$\begin{aligned} \zeta :[0,1]\times X\rightarrow H \end{aligned}$$

such that for some $x_{0}\in H$, there hold

$$\begin{aligned} \zeta (0,x)=x,\;\ \hbox {for} \;\ \hbox {all} \;\ x\in X \end{aligned}$$

and

$$\begin{aligned} \zeta (1,x)=x_{0},\;\ \hbox {for} \;\ \hbox {all} \;\ x\in X. \end{aligned}$$

• (ii) If X is a closed subset of a topological space H, $\hbox {cat}_{H}(X)$ denotes Lusternik–Schnirelmann category of X in H, i.e., the least number of closed and contractible sets in H which cover X.

Lemma 3.2

[32, Theorem 2.3] Suppose that H is a Hilbert manifold and $I\in C^{1}(H,P{\mathbb {R}})$. Assume that for $c_{0}\in {\mathbb {R}}$ and $k\in {\mathbb {N}}$:

• (i) I satisfies the $(PS)_{c}$ condition for $c\le c_{0}$;

• (ii) $\hbox {cat}(\{z\in H : \;\ I(z)\le c_{0}\})\ge k$.

Then I has at least k critical points in $\{z\in H : \;\ I(z)\le c_{0}\}$.

By adapting some arguments found in [33], we can easily get the following standard lemma.

Lemma 3.3

Let $\{(u_{n},v_{n})\}\subset H$ be a nonnegative function sequence with

$$\begin{aligned} \int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx=1\; \hbox {and}\; \int _{{\mathbb {R}}^{2N}}\frac{|u_{n}(x)-u_{n}(y)|^{2}+|v_{n}(x)-v_{n}(y)|^{2}}{|x-y|^{N+2s}}dxdy\rightarrow S_{\alpha ,\beta }. \end{aligned}$$

Then there exists a sequence $\{(y_{n}, \tau _{n})\}\subset {\mathbb {R}}^{N}\times {\mathbb {R}}^{+}$ such that

$$\begin{aligned} \omega _{n}(x)=\left( \omega _{n}^{1}(x),\omega _{n}^{2}(x)\right) =\tau _{n}^{\frac{N-2s}{2}}\left( u_{n}(\tau _{n}x+y_{n}),v_{n}(\tau _{n}x+y_{n})\right) \end{aligned}$$

contains a convergent subsequence denoted again by $\{\omega _{n}\}$ such that

$$\begin{aligned} \omega _{n}\rightarrow \omega \;\;\ \text{ in }\;\ H^{s}\left( {\mathbb {R}}^{N}\right) \times H^{s}\left( {\mathbb {R}}^{N}\right) , \end{aligned}$$

where $\omega =(\omega ^{1},\omega ^{2})>0$ in ${\mathbb {R}}^{N}$. Moreover, we have $\tau _{n}\rightarrow 0$ and $y_{n}\rightarrow y\in \overline{\Omega }$ as $n\rightarrow \infty$.

Since $\Omega$ is a smooth bounded domain of ${\mathbb {R}}^{N}$, we choose $\delta >0$ small enough so that

$$\begin{aligned} \Omega _{\delta }^{+}:=\left\{ x\in {\mathbb {R}}^{N} : \;\ \hbox {dist}(x,\Omega )<\delta \right\} \end{aligned}$$

and

$$\begin{aligned} \Omega _{\delta }^{-}:=\left\{ x\in \Omega : \;\ \hbox {dist}(x,\partial \Omega )>\delta \right\} \end{aligned}$$

are homotopically equivalent to $\Omega$. Moreover, without loss of generality, we assume $B_{\delta }(0):=B(0,\delta )\subset \Omega$.

Note that for any $(u,v)\in {\mathcal {N}}^{-}_{\lambda ,\mu }$, $\int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx>0$. So we can define the continuous map $\psi : {\mathcal {N}}^{-}_{\lambda ,\mu }\rightarrow {\mathbb {R}}^{N}$ by setting

$$\begin{aligned} \psi (u,v):=\frac{\displaystyle \int _{\Omega }xu_{+}^{\alpha }v_{+}^{\beta }dx}{\displaystyle \int _{\Omega }u_{+}^{\alpha }v_{+}^{\beta }dx}. \end{aligned}$$

Then we have the following result.

Lemma 3.4

There exists $\Lambda ^{*}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$ and $(u,v)\in {\mathcal {N}}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu })$, then $\psi (u,v)\in \Omega _{\delta }^{+}$.

Proof

Suppose by contradiction that there exist $\lambda _{n}$, $\mu _{n}\rightarrow 0$, and $\{(u_{n},v_{n})\}\subset {\mathcal {N}}^{-}_{\lambda _{n},\mu _{n}}(c_{\lambda _{n},\mu _{n}})$ such that $\psi (u_{n},v_{n}) \not \in \Omega _{\delta }^{+}$. Using the same method as [22, Lemma 2.7] or [26, Lemma 2.2], we can easily obtain $\{(u_{n},v_{n})\}$ is bounded in H and then

$$\begin{aligned} \int _{\Omega }\left( \lambda _{n}(u_{n})_{+}^{p}+\mu _{n}(v_{n})_{+}^{p}\right) dx\rightarrow 0,\;\;\ \hbox {as}\;\ n\rightarrow \infty . \end{aligned}$$

Therefore, we have

$$\begin{aligned} I_{\lambda _{n},\mu _{n}}(u_{n},v_{n})=\left( \frac{1}{2}-\frac{1}{2_{s}^{*}}\right) \int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}{+}|\hat{v}_{n}(\xi )|^{2}\right) d\xi +o(1)\le c_{\lambda _{n},\mu _{n}}+o(1), \end{aligned}$$

which implies that

$$\begin{aligned}&\int _{{\mathbb {R}}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2}\right) d\xi \le \frac{N}{s}c_{\lambda _{n},\mu _{n}}+o(1)\nonumber \\&\quad \le 2\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{^{\frac{N}{2s}}}+o(1). \end{aligned}$$

(3.1)

On the other hand, we deduce from the fact that $\{(u_{n},v_{n})\}\subset \mathcal {N}^{-}_{\lambda _{n},\mu _{n}}(c_{\lambda _{n},\mu _{n}})\subset \mathcal {N}_{\lambda _{n},\mu _{n}}$ that

$$\begin{aligned} \int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2}\right) d\xi =2\int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx+o(1). \end{aligned}$$

(3.2)

By (2.1), (3.1), (3.2), and the definition of $S_{\alpha ,\beta }$, we have

$$\begin{aligned} S_{\alpha ,\beta }\le & {} \frac{2\kappa _{N,s}^{-1}\displaystyle \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi }{\left( \displaystyle \int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx \right) ^{\frac{2}{\alpha +\beta }}}\\\le & {} 2^{1+\frac{2}{2_{s}^{*}}}\kappa _{N,s}^{-1}\left( \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi \right) ^{\frac{2s}{N}}\\\le & {} 2^{1+\frac{2}{2_{s}^{*}}}\kappa _{N,s}^{-1}\left[ 2\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{^{\frac{N}{2s}}}\right] ^{\frac{2s}{N}}+o(1)\\= & {} S_{\alpha ,\beta }+o(1), \end{aligned}$$

which leads to

$$\begin{aligned} \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi \rightarrow 2\left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{^{\frac{N}{2s}}} \end{aligned}$$

and

$$\begin{aligned} \int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx\rightarrow \left( \frac{S_{\alpha ,\beta }}{2}\right) ^{\frac{N}{2s}}\left( \frac{\kappa _{N,s}}{2}\right) ^{^{\frac{N}{2s}}}. \end{aligned}$$

Now, set $\eta _{n}^{1}=u_{n}\big (\int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx\big )^{\frac{-1}{\alpha +\beta }},$ $\eta _{n}^{2}=v_{n}\big (\int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx\big )^{\frac{-1}{\alpha +\beta }}.$ Then it is easy to see that $\eta _{n}^{1}$, $\eta _{n}^{2}$ satisfy

$$\begin{aligned} \int _{\Omega }\left( \eta _{n}^{1}\right) _{+}^{\alpha }\left( \eta _{n}^{2}\right) _{+}^{\beta }dx=1,\;\ \int _{\mathbb {R}^{2N}}\frac{|\eta _{n}^{1}(x)-\eta _{n}^{1}(y)|^{2}+|\eta _{n}^{2}(x)-\eta _{n}^{2}(y)|^{2}}{|x-y|^{N+2s}}dxdy\rightarrow S_{\alpha ,\beta }, \end{aligned}$$

as $n\rightarrow \infty .$

Then the function $\big (\widetilde{\eta }_{n}^{1},\widetilde{\eta }_{n}^{2}\big )=\big ((\eta _{n}^{1})_{+},(\eta _{n}^{2})_{+}\big )$ satisfies

$$\begin{aligned} \int _{\Omega }\left( \widetilde{\eta }_{n}^{1}\right) ^{\alpha }\left( \widetilde{\eta }_{n}^{2}\right) ^{\beta }dx=1,\;\ \int _{\mathbb {R}^{2N}}\frac{|\widetilde{\eta }_{n}^{1}(x)-\widetilde{\eta }_{n}^{1}(y)|^{2} +|\widetilde{\eta }_{n}^{2}(x)-\widetilde{\eta }_{n}^{2}(y)|^{2}}{|x-y|^{N+2s}}dxdy\rightarrow S_{\alpha ,\beta }, \end{aligned}$$

as $n\rightarrow \infty .$

Using Lemma 3.3, there exists a sequence $\{(y_{n}, \tau _{n})\}\subset \mathbb {R}^{N}\times \mathbb {R}^{+}$ such that

$$\begin{aligned} \left( \omega ^{1}_{n}(x),\omega ^{2}_{n}(x)\right) =\tau _{n}^{\frac{N-2s}{2}}\left( \widetilde{\eta }_{n}^{1}(\tau _{n}x+y_{n}),\widetilde{\eta }_{n}^{2}(\tau _{n}x+y_{n})\right) \end{aligned}$$

converges strongly to $(\omega _{1},\omega _{2})\in H^{s}(\mathbb {R}^{N})\times H^{s}(\mathbb {R}^{N})$.

Considering $\chi \in C_{0}^{\infty }(\mathbb {R}^{N})$ such that $\chi (x)=x$ in $\Omega$, we infer

$$\begin{aligned} \psi (u_{n},v_{n})= & {} \frac{\int _{\Omega }\chi (x)(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx}{\int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx}\\= & {} \int _{\Omega } \chi (x)\left( \widetilde{\eta }_{n}^{1}\right) ^{\alpha }\left( \widetilde{\eta }_{n}^{2}\right) ^{\beta }dx\\= & {} \int _{\Omega }\chi (\tau _{n}x+y_{n})\left( \omega _{n}^{1}(x)\right) ^{\alpha }\left( \omega _{n}^{2}(x)\right) ^{\beta }dx. \end{aligned}$$

Then it follows from the Lebesgue dominated convergence theorem and the fact $\tau _{n}\rightarrow 0$, $y_{n}\rightarrow y\in \overline{\Omega }$ that

$$\begin{aligned} \int _{\Omega }\chi (\tau _{n}x+y_{n})\left( \omega _{n}^{1}(x)\right) ^{\alpha }\left( \omega _{n}^{2}(x)\right) ^{\beta }dx\rightarrow y\in \overline{\Omega }, \;\;\ \hbox {as}\;\ n\rightarrow \infty . \end{aligned}$$

That is, $\psi (u_{n},v_{n})\rightarrow y\in \overline{\Omega }$, as $n\rightarrow \infty$, which is a contradiction. Thus, we finish the proof.

By Lemmas 2.6, 2.10 and the definition of $\Omega _{\delta }^{-},$ we infer $\underset{M_{\delta }}{\inf } u_{\lambda ,\mu }^{+}>0$ and $\underset{M_{\delta }}{\inf } v_{\lambda ,\mu }^{+}>0$, where $M_{\delta }:=\{x\in \Omega :\;\ \hbox {dist}(x,\Omega _{\delta }^{-})\le \frac{\delta }{2}\}$. Note that $\Omega _{\delta }^{-}$ is compact, then by Lemma 2.12 and using the idea of [25, Lemma 3.4], [27, Lemma 3.3], we can easily deduce that there exists ${\widetilde{t}}^{-}>0$ such that

$$\begin{aligned} \left( \widetilde{t}^{-}\sqrt{\alpha }v_{\epsilon }(x-y), \widetilde{t}^{-}\sqrt{\beta }v_{\epsilon }(x-y)\right) \in \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho ) \end{aligned}$$

uniformly in $y\in \Omega _{\delta }^{-}$. Moreover, it follows from Lemma 3.4 that $\psi (\widetilde{t}^{-}\sqrt{\alpha }v_{\epsilon }(x-y), \widetilde{t}^{-}\sqrt{\beta }v_{\epsilon }(x-y))\in \Omega _{\delta }^{+}$.

Applying the same idea as that in [22], we define the map $\gamma : \Omega _{\delta }^{-}\rightarrow \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )$ by

$$\begin{aligned} \gamma (y) = \left\{ \begin{array}{lll}(\widetilde{t}^{-}\sqrt{\alpha }v_{\epsilon }(x-y), \widetilde{t}^{-}\sqrt{\beta }v_{\epsilon }(x-y)), &{} \quad \hbox {if}\;\ x\in B_{\delta }(y),\\ 0, &{} \quad \hbox {otherwise}. \end{array}\right. \end{aligned}$$

Below we denote by $\psi _{\lambda ,\mu }$ the restriction of $\psi$ on $\mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho ).$

Note that $v_{\epsilon }$ is radial, then for each $y\in \Omega _{\delta }^{-}$, we have

$$\begin{aligned} (\psi _{\lambda ,\mu }\circ \gamma )(y)= \frac{\int _{\Omega }x \left( \widetilde{t}^{-}\sqrt{\alpha }v_{\epsilon }(x-y)\right) ^{\alpha }\left( \widetilde{t}^{-}\sqrt{\beta }v_{\epsilon }(x-y)\right) ^{\beta }dx}{\int _{\Omega }\left( \widetilde{t}^{-}\sqrt{\alpha }v_{\epsilon }(x-y)\right) ^{\alpha }(\widetilde{t}^{-}\sqrt{\beta }v_{\epsilon }(x-y))^{\beta }dx} = y. \end{aligned}$$

Next, we define the map ${\mathcal {H}}_{\lambda ,\mu }: [0,1]\times \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )\rightarrow \mathbb {R}^{N}$ by

$$\begin{aligned} \mathcal {H}_{\lambda ,\mu }(t,z):=t \psi _{\lambda ,\mu }(z)+(1-t)\psi _{\lambda ,\mu }(z). \end{aligned}$$

Then we have the following lemma. $\square$

Lemma 3.5

For each $\epsilon \in (0,\epsilon ^{*})$, there exists $\Lambda ^{*}>0$ such that if $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$, $\mathcal {H}_{\lambda ,\mu }([0,1]\times \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho ))\subset \Omega _{\delta }^{+}.$

Proof

We argue by contradiction and suppose that there exists $t_{n}\in [0,1]$, $\lambda _{n}, \mu _{n}\rightarrow 0$ and $z_{n}=(u_{n},v_{n})\in \mathcal {N}^{-}_{\lambda _{n},\mu _{n}}(c_{\lambda _{n},\mu _{n}}-\varrho )$ such that

$$\begin{aligned} \mathcal {H}_{\lambda _{n},\mu _{n}}(t_{n},z_{n})\not \in \Omega _{\delta }^{+}\;\;\ \hbox {for}\;\ \hbox {all}\;\ n\in \mathbb {N}. \end{aligned}$$

Moreover, we can assume that, up to a subsequence, $t_{n}\rightarrow t_{0}\in [0,1]$. By Lemma 2.10 (ii) and the same argument as that in the proof of Lemma 3.4, we have

$$\begin{aligned} \mathcal {H}_{\lambda _{n},\mu _{n}}(t_{n},z_{n})\rightarrow y\in \overline{\Omega },\;\;\ \hbox {as}\;\ n\rightarrow \infty , \end{aligned}$$

which is a contradiction. $\square$

4 Proof of Theorem 1.1

To prove Theorem 1.1, we need the following lemmas.

Lemma 4.1

If (u, v) is a critical point of $I_{\lambda ,\mu }$ on $\mathcal {N}^{-}_{\lambda ,\mu }$, then it is a critical point of $I_{\lambda ,\mu }$ on H.

Proof

We refer the readers to Lemma 2.5 for similar proofs.

Below we denote by $I_{\mathcal {N}^{-}_{\lambda ,\mu }}$ the restriction of $I_{\lambda ,\mu }$ on $\mathcal {N}^{-}_{\lambda ,\mu }$. $\square$

Lemma 4.2

There exists $\Lambda ^{*}>0$ such that any sequence $\{(u_{n},v_{n})\}\subset \mathcal {N}^{-}_{\lambda ,\mu }$ with $I_{\mathcal {N}^{-}_{\lambda ,\mu }}(u_{n},v_{n})\rightarrow c\in (-\infty ,c_{\lambda ,\mu })$ and $I'_{\mathcal {N}^{-}_{\lambda ,\mu }}(u_{n},v_{n})\rightarrow 0$ contains a convergent subsequence for all $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$.

Proof

By hypothesis, there exists a sequence $\{\varsigma _{n}\}\subset \mathbb {R}$ such that

$$\begin{aligned} \Vert I'_{\lambda ,\mu }(u_{n},v_{n})-\varsigma _{n}\Psi '_{\lambda ,\mu }(u_{n},v_{n})\Vert \rightarrow 0,\;\;\ \hbox {as}\;\ n\rightarrow \infty , \end{aligned}$$

where $\Psi _{\lambda ,\mu }$ is given by (2.6).

Then

$$\begin{aligned} I'_{\lambda ,\mu }(u_{n},v_{n})=\varsigma _{n}\Psi '_{\lambda ,\mu }(u_{n},v_{n})+o(1). \end{aligned}$$

Since $(u_{n},v_{n})\in \mathcal {N}^{-}_{\lambda ,\mu }\subset \mathcal {N}_{\lambda ,\mu }$, by a simple computation, we get

$$\begin{aligned} \langle \Psi '_{\lambda ,\mu }(u_{n},v_{n}), (u_{n},v_{n}) \rangle <0. \end{aligned}$$

If $\langle \Psi '_{\lambda ,\mu }(u_{n},v_{n}), (u_{n},v_{n})\rangle \rightarrow 0$, we have

$$\begin{aligned}&(2_{s}^{*}-2)\int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2}\right) d\xi \\&\quad =(2_{s}^{*}-p)\int _{\Omega }\left( \lambda (u_{n})_{+}^{p}+\mu (v_{n})_{+}^{p}\right) dx+o(1) \end{aligned}$$

and

$$\begin{aligned} (2-p)\int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2}\right) d\xi =2(2_{s}^{*}-p)\int _{\Omega }(u_{n})_{+}^{\alpha }(v_{n})_{+}^{\beta }dx+o(1). \end{aligned}$$

Using (2.1) and (2.3) and the H$\ddot{o}$lder inequality, there exist constants $C_{1}$, $C_{2}>0$ such that

$$\begin{aligned}&\int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi \\&\quad \le C_{1}(\lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}})^{\frac{2-p}{2}}\left( \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi \right) ^{\frac{p}{2}}+o(1) \end{aligned}$$

and

$$\begin{aligned} \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi \le C_{2}\left( \int _{\mathbb {R}^{N}}|\xi |^{2s}(|\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2})d\xi \right) ^{\frac{2_{s}^{*}}{2}}+o(1). \end{aligned}$$

Then we have

$$\begin{aligned} \int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2}\right) d\xi \le C_{1}^{\frac{2}{2-p}}\left( \lambda ^{\frac{2}{2-p}}+ \mu ^{\frac{2}{2-p}}\right) +o(1) \end{aligned}$$

and

$$\begin{aligned} \int _{\mathbb {R}^{N}}|\xi |^{2s}\left( |\hat{u}_{n}(\xi )|^{2}+|\hat{v}_{n}(\xi )|^{2}\right) d\xi \ge C_{2}^{-\frac{2}{2_{s}^{*}-2}}+o(1). \end{aligned}$$

If we choose $\Lambda ^{*}$ sufficiently small, this is impossible. Thus we may assume that

$$\begin{aligned} \Psi '_{\lambda ,\mu }(u_{n},v_{n}), (u_{n},v_{n}) \rangle \rightarrow L<0 \;\ \hbox {as}\;\ n\rightarrow \infty . \end{aligned}$$

Since $\langle I'_{\lambda ,\mu }(u_{n},v_{n}), (u_{n},v_{n})\rangle =0$, we conclude that $\varsigma _{n}\rightarrow 0$ and the sequence $\{(u_{n},v_{n})\}$ is a $(PS)_{c}$-sequence for $I_{\lambda ,\mu }$ with $c\in (-\infty ,c_{\lambda ,\mu })$. Then the proof follows by Lemmas 2.8 and 2.9. $\square$

Lemma 4.3

If $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$, then we have

$$\begin{aligned} \hbox {cat}\left( \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )\right) \ge \hbox {cat}(\Omega ). \end{aligned}$$

Proof

Assume $\hbox {cat}(\mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho ))=n$. Then we may suppose

$$\begin{aligned} \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )=A_{1}\cup \cdots \cup A_{n}, \end{aligned}$$

where $A_{j}, j=1,2,\ldots ,n,$ are closed and contractible in $\mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )$, i.e., there exists $h_{j}\in C([0,1]\times A_{j}, \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho ))$ such that

$$\begin{aligned} h_{j}(0,z)=z,\;\;\ h_{j}(1,z)=\nu \;\;\ \hbox {for}\;\;\ \hbox {all}\;\ z\in A_{j}, \end{aligned}$$

where $\nu \in A_{j}$ is fixed. Consider $B_{j}:=\gamma ^{-1}(A_{j})$, $j=1,2,\ldots ,n.$ Then the sets $B_{j}$ are closed and

$$\begin{aligned} \Omega _{\delta }^{-}=B_{1}\cup \cdots \cup B_{n}. \end{aligned}$$

We define the deformation $g_{j}:[0,1]\times B_{j}\rightarrow \Omega _{\delta }^{+}$ by setting

$$\begin{aligned} g_{j}(t,y):=\mathcal {H}_{\lambda ,\mu }\left( t, h_{j}(t,\gamma (y))\right) \end{aligned}$$

for $\lambda ^{\frac{2}{2-p}}+\mu ^{\frac{2}{2-p}}\in (0,\Lambda ^{*})$. Note that

$$\begin{aligned} g_{j}(0,y)=\mathcal {H}_{\lambda ,\mu }\left( 0,h_{j}(0,\gamma (y))\right) =\left( \psi _{\lambda ,\mu }\circ \gamma \right) (y)=y, \;\ \hbox {for}\;\ \hbox {all}\;\ y\in B_{j} \end{aligned}$$

and

$$\begin{aligned} g_{j}(1,y)=\mathcal {H}_{\lambda ,\mu }\left( 1,h_{j}(1,\gamma (y))\right) =\psi _{\lambda ,\mu }(\nu )\in \Omega _{\delta }^{+}, \;\ \hbox {for}\;\ \hbox {all}\;\ y\in B_{j}. \end{aligned}$$

This implies the sets $B_{j} (j=1,2,\ldots , n)$ are contractible in $\Omega _{\delta }^{+}$. Thus

$$\begin{aligned} \hbox {cat}(\Omega )=\hbox {cat}_{\Omega _{\delta }^{+}}(\Omega _{\delta }^{-})\le n=\hbox {cat}\left( \mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )\right) . \end{aligned}$$

$\square$

Proof of Theorem 1.1

By Lemmas 2.9 and 4.2, $I_{\mathcal {N}^{-}_{\lambda ,\mu }}$ satisfies $(PS)_{c}$ condition for $c\in (-\infty ,c_{\lambda ,\mu })$. Then, by Lemmas 3.2 and 4.3, we deduce that $I_{\mathcal {N}^{-}_{\lambda ,\mu }}$ admits at least $\hbox {cat}(\Omega )$ critical points in $\mathcal {N}^{-}_{\lambda ,\mu }(c_{\lambda ,\mu }-\varrho )$. By Lemma 4.1, we have $I_{\lambda ,\mu }$ has at least $\hbox {cat} (\Omega )$ critical points in $\mathcal {N}^{-}_{\lambda ,\mu }$. Moreover, since $\mathcal {N}^{+}_{\lambda ,\mu }\cap \mathcal {N}^{-}_{\lambda ,\mu }=\emptyset$, the proof is completed. $\square$