Positive Ground States for the Fractional Klein–Gordon–Maxwell System with Critical Exponents

Abstract

In this paper, we consider a fractional Klein–Gordon–Maxwell system when the non-linearity exhibits critical growth. Under some appropriate assumptions on potential V, we prove the existence of positive ground state solutions using the Nehari method.

1 Introduction

In this article, we study the following fractional Klein–Gordon–Maxwell (KGM) system with critical exponents:

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s}u+V(x)u-(2\omega +\phi )\phi u=\lambda |u|^{\alpha -2}u+|u|^{2_{s}^{*}-2}u&{} \quad in\ {\mathbb {R}}^{N},\\ (-\Delta )^{s}\phi +\phi u^{2}=-\omega u^{2}&{}\quad in\ {\mathbb {R}}^{N}, \end{array} \right. \end{aligned}$$

(1.1)

where $\lambda >0$, $\omega >0$, $N>2s$ with $s\in (0,1)$, $2_{s}^{*}=2N/(N-2s)$ is the fractional Sobolev exponent, $V(x)\in C ({\mathbb {R}}^{N},{\mathbb {R}})$ is a function, $2<\alpha <2_{s}^{*}$, $\phi \in D^{s}({\mathbb {R}}^{N},{\mathbb {R}})$, and $u\in H^{s}({\mathbb {R}}^{N},{\mathbb {R}})$ are functions, and $(-\Delta )^{s}$ is a fractional Laplacian operator. Up to normalization constants, $(-\Delta )^{s}$ can be defined as

$$\begin{aligned} (-\Delta )^{s}u(x)=\lim _{\varepsilon \rightarrow 0}\int _{B_{\varepsilon }^ {c}(x)}\dfrac{u(x)-u(y)}{|x-y|^{N+2s}}\text {d}y,\quad \ x,y\in {\mathbb {R}}^{N}, \end{aligned}$$

where $B_{\varepsilon }(x)$ denotes a ball with radius $\varepsilon >0$ centered at $x\in {\mathbb {R}}^{N}$. It can be regarded as the infinitesimal generator of Lévy stable diffusion process. In recent years, non-linear equations or systems involving fractional operators have received extensive attention due to their important roles in numerous fields, such as flame propagation, water waves, phase transitions, combustion, and other physical environments (see [1] and its references for more details).

KGM system was proposed by Benci and Fortunato as a model describing solitary waves for the non-linear stationary Klein–Gordon equation coupled with Maxwell equation in the three-dimensional space interacting with the electrostatic field. In [3], they proved that when $\omega \in (0,m_{0})$ and $q\in (4,6)$, there are infinitely many radially symmetric solutions to the following system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+[m_{0}^{2}-(\omega +\phi )^{2}]u=|u|^{q-2}u&{} \quad in\ {\mathbb {R}}^{3},\\ \Delta \phi =(\omega +\phi ) u^{2}&{}\quad in\ {\mathbb {R}}^{3}. \end{array} \right. \end{aligned}$$

(1.2)

Later, D’Aprile and Mugnai [7] extended the research to$\omega \in (0,\sqrt{(q-2)/2} m_{0})$ and $q\in (2,4]$. Cassani [5] investigated the existence of solutions for the (KGM) system in the critical case. D’Aprile and Mugnai [6] studied some non-existence results for the Klein–Gordon equation coupled with the electrostatic field. In [2], Azzollini and Pomponio studied the existence of the ground state solution of (1.2) when one of the following conditions holds:

$\mathrm {(i)}$:

$4\le q<6$ and $0<\omega <m_{0}$;

$\mathrm {(ii)}$:

$2<q<4$ and $0<\omega <\sqrt{(q-2)/(6-q)}m_{0}$.

Carri${\tilde{a}}$o et al. [4] considered the existence of positive ground state solutions for the (KGM) system with a periodic potential V. Miyagaki et al. [10] studied the existence of positive ground state solutions for the fractional Klein–Gordon–Maxwell system.

Inspired by the above literature, in this article, we are going to discuss the existence of positive ground state solutions for the system (1.1), which involves the fractional Laplacian operator and critical exponents. To state our result, we list the following conditions on $V(x)\in C({\mathbb {R}}^{N},{\mathbb {R}})$:

$(V_{1})$:

V is periodic in $x_{i}(i=1,\cdots ,N)$.

$(V_{2})$:

There exists $V_{*}>0$, such that $V(x)\ge V_{*}$.

In what follows, we can state our main result.

Theorem 1.1

Suppose that $(V_{1})$ and $(V_{2})$ are hold. Then, system (1.1) has a positive ground states if one of the following conditions is met:

• (i) $4\le \alpha <2_{s}^{*}$ and $V_{*}>0;$

• (ii) $2<\alpha <4$ and $\frac{V_{*}}{\omega ^{2}}>\frac{(\alpha -4)^{2}}{4(\alpha -2)}$.

2 Preliminaries

In this section, we first begin to give some definitions and results that are useful for the proof. Let $L^{2}({\mathbb {R}}^{N})$ be the Lebesgue space with the scalar product

$$\begin{aligned} \langle u,v\rangle _{L^{2}}=\int _{{\mathbb {R}}^{N}}uv\text {d}x \end{aligned}$$

for any $u,v\in L^{2}({\mathbb {R}}^{N})$, and we denote by $|\cdot |_{q}$ the norm of $L^{q}({\mathbb {R}}^{N})$. The following fractional critical Sobolev space:

$$\begin{aligned} D^s({\mathbb {R}}^N)=\{u\in L^{2^*_s}({\mathbb {R}}^N):(-\Delta )^{\frac{s}{2}}u\in L^{2}({\mathbb {R}}^{N})\}, \end{aligned}$$

with the norm

$$\begin{aligned} \Vert u\Vert ^{2}_{D^{s}}=\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u|^{2}\text {d}x, \end{aligned}$$

and the scalar product

$$\begin{aligned} \langle u,v\rangle ^{2}_{D^{s}}=\int _{{\mathbb {R}}^{N}}(-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v\text {d}x \end{aligned}$$

for any $u,v\in D^s({\mathbb {R}}^N)$. The fractional Hilbert space

$$\begin{aligned} H^{s}({\mathbb {R}}^{N})=\{u\in L^{2}({\mathbb {R}}^{N}):(-\Delta )^{\frac{s}{2}}u\in L^{2}({\mathbb {R}}^{N})\}, \end{aligned}$$

equipped with the scalar product

$$\begin{aligned} \langle u,v\rangle _{s}=\langle u,v\rangle ^{2}_{D^{s}}+\langle u,v\rangle _{L^{2}}^{2} \end{aligned}$$

and the norm

$$\begin{aligned} \Arrowvert u\Arrowvert ^{2}_{s}= \Vert u\Vert ^{2}_{D^{s}}+|u|_{2}^{2}. \end{aligned}$$

Set

$$\begin{aligned} {\mathcal {H}}=\bigg \{u\in H^{s}({\mathbb {R}}^{N}):\int _{{\mathbb {R}}^{N}}V(x)u^{2}\text {d}x<\infty \bigg \}, \end{aligned}$$

the corresponding norm

$$\begin{aligned} \Arrowvert u\Arrowvert ^{2}=\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u|^{2}+V(x)u^{2}\text {d}x, \end{aligned}$$

which is equivalent to the norm $\Vert u\Vert _{s}^{2}$. Moreover, ${\mathcal {H}}$ is continuously embedded into $L^{q}({\mathbb {R}}^{N})$ for $q\in [2, 2_{s}^{*}]$.

The weak solution $(u,\phi )\in {\mathcal {H}}\times D^{s}({\mathbb {R}}^{N})$ of the system (1.1) is critical point of the following energy functional:

$$\begin{aligned} {\mathcal {J}}(u,\phi ){} & {} =\dfrac{1}{2}\bigg (\Vert u\Vert ^{2}_{D^{s}}-\Vert \phi \Vert ^{2}_{D^{s}} +\int _{{\mathbb {R}}^{N}}(V(x)u^{2}-(2\omega +\phi )\phi u^{2})\text {d}x\bigg )\nonumber \\{} & {} \quad \ -\dfrac{\lambda }{\alpha }\int _{{\mathbb {R}}^{N}}|u|^{\alpha }\text {d}x-\dfrac{1}{2_{s}^{*}} \int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x. \end{aligned}$$

(2.1)

Lemma 2.1

For any $u\in {\mathcal {H}}$, there exists a unique $\phi =\phi _{u}\in D^{s}({\mathbb {R}}^{N})$ which solves

$$\begin{aligned} (-\Delta )^{s}\phi +\phi u^{2}=-\omega u^{2}. \end{aligned}$$

(2.2)

Moreover, on the set $\{x\in {\mathbb {R}}^{N}|u(x)\ne 0\}$, we have $-\omega \le \phi _{u}\le 0$ for $\omega >0$.

Proof

The proof of existence and uniqueness is similar to [10, Lemma 2.1], so we omit it. For any fixed $u\in {\mathcal {H}}$ and $\omega >0$. We multiply (2.2) by $(\omega +\phi _{u})^{-}=\min \{\omega +\phi _{u},0\}$, and integrating on $\{x|\omega +\phi _{u}<0\}$, we have

$$\begin{aligned} \int _{\{x|\omega +\phi _{u}<0\}}|(-\Delta )^{\frac{s}{2}}\phi _{u}|^{2}\text {d}x +\int _{\{x|\omega +\phi _{u}<0\}}(\omega +\phi _{u})^{2}u^{2}\text {d}x=0, \end{aligned}$$

which means that $\phi _{u}+\omega \ge 0$, that is, $\phi _{u}\ge -\omega$, where $u\ne 0$.

In what follows, we use $(\phi _{u})^{+}=\max \{\phi _{u},0\}$ as a test function in (2.2) to get:

$$\begin{aligned} \int _{\{x|\phi _{u}>0\}}|(-\Delta )^{\frac{s}{2}}\phi _{u}|^{2}\text {d}x +\int _{\{x|\phi _{u}>0\}}(\omega +\phi _{u})\phi _{u}u^{2}\text {d}x=0. \end{aligned}$$

According to $\omega +\phi _{u}\ge 0$, we have $\phi _{u}\le 0$. $\square$

By Lemma 2.1, define the map $\varUpsilon :u\in {\mathcal {H}}\rightarrow \phi _{u}\in D^{s}({\mathbb {R}}^{N})$. According to [3, Lemma 3.4], we know that the map $\varUpsilon$ is $C^{1}$ by standard arguments, and each $u\in {\mathcal {H}}$ is mapped into the unique solution of (2.2). Then, we can get that ${\mathcal {J}}'_{\phi }(u,\phi _{u})=0$.

Now, we consider the functional

$$\begin{aligned} \begin{aligned} {\mathcal {I}}(u)&=\dfrac{1}{2}\bigg (\Vert u\Vert ^{2}-\int _{{\mathbb {R}}^{N}}\omega \phi _{u} u^{2}\text {d}x\bigg ) -\dfrac{\lambda }{\alpha }\int _{{\mathbb {R}}^{N}}|u|^{\alpha }\text {d}x -\dfrac{1}{2_{s}^{*}}\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x, \end{aligned} \end{aligned}$$

(2.3)

and

$$\begin{aligned} \begin{aligned} \langle {\mathcal {I}}'(u),v\rangle&=\int _{{\mathbb {R}}^{N}}\bigg [(-\Delta )^ {\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v+V(x)uv-(2\omega +\phi _{u})\phi _{u} uv\\&\quad \ -\lambda |u|^{\alpha -2}uv-|u|^{2_{s}^{*}-2}uv\bigg ]\text {d}x \end{aligned} \end{aligned}$$

for any $u,v\in {\mathcal {H}}$. Therefore, $(u,\phi )\in {\mathcal {H}}\times D^{s}({\mathbb {R}}^{N})$ is a solution of (1.1) if and only if u is a critical point of ${\mathcal {I}}(u)={\mathcal {J}}(u,\phi _{u})$ and $\phi =\phi _{u}$.

Lemma 2.2

Let $u\in {\mathcal {H}}$ and $2\varphi _{u}=\langle \gamma '(u),\gamma (u)\rangle$. Then, $\varphi _{u}$ is a solution of equation

$$\begin{aligned} \int _{{\mathbb {R}}^{N}}\omega \varphi _{u}u^{2}\text {d}x= \int _{{\mathbb {R}}^{N}}(\omega +\phi _{u})\phi _{u}u^{2}\text {d}x. \end{aligned}$$

(2.4)

Moreover, on the set $\{x\in {\mathbb {R}}^{N}:u(x)\ne 0\}$, we have

$$\begin{aligned} \max \{-\omega -\phi _{u},\phi _{u}\}\le \varphi _{u}\le 0. \end{aligned}$$

Proof

Set the map $\Sigma :{\mathcal {H}}\times D^{s}\rightarrow D^{s}$ as follows:

$$\begin{aligned} \Sigma (u,\gamma (u))=-((-\Delta )^{s})^{-1}[(\omega +\phi _{u})u^{2}]-\phi _{u}. \end{aligned}$$

We can find that $(u,\phi _{u})$ solves (2.2) if and only if $\Sigma (u,\gamma (u))=0$.

For any $(u,\gamma (u))\in {\mathcal {H}}\times D^{s}$, we get

$$\begin{aligned} \dfrac{\partial \Sigma (u,\gamma (u))}{\partial \gamma (u)}:\phi _{u} \longmapsto -((-\Delta )^{s})^{-1}[\phi _{u}u^{2}]-\phi _{u} \end{aligned}$$

and

$$\begin{aligned} \dfrac{\partial \Sigma (u,\gamma (u))}{\partial u}:u\longmapsto -2((-\Delta )^{s})^{-1}[(\omega +\phi _{u})u^{2}]. \end{aligned}$$

Obviously, $\frac{\partial \Sigma (u,\gamma (u))}{\partial \gamma (u)}$ is reversible for each $(u,\gamma (u))\in {\mathcal {H}}\times D^{s}$. For any $u\in {\mathcal {H}}$, since $(u,\gamma (u))$ is the solution of (2.2), we have

$$\begin{aligned} \begin{aligned} 2\varphi _{u}+((-\Delta )^{s})^{-1}[2\varphi _{u}u^{2}]&=-2((-\Delta )^{s})^{-1}[(\omega +\phi _{u})u^{2}]. \end{aligned} \end{aligned}$$

Then

$$\begin{aligned} \begin{aligned} ((-\Delta )^{s}+u^{2})\varphi _{u}&=-(\omega +\phi _{u})u^{2}, \end{aligned} \end{aligned}$$

that is

$$\begin{aligned} \varphi _{u}=\dfrac{\langle \gamma '(u),\gamma (u)\rangle }{2} =-((-\Delta )^{s}+u^{2})^{-1}[(\omega +\phi _{u})u^{2}]. \end{aligned}$$

Then, we have

$$\begin{aligned} (-\Delta )^{s}\varphi _{u}+\varphi _{u}u^{2}=-(\omega +\phi _{u})u^{2}. \end{aligned}$$

(2.5)

Furthermore, by (2.2), we obtain

$$\begin{aligned} \begin{aligned} \int _{{\mathbb {R}}^{N}}\omega \varphi _{u} u^{2}\text {d}x&=\int _{{\mathbb {R}}^{N}}-((-\Delta )^{s}+u^{2})^{-1}[(\omega +\phi _{u})u^{2}]\omega u^{2}\text {d}x\\&=\int _{{\mathbb {R}}^{N}}-((-\Delta )^{s}+u^{2})^{-1}[\omega u^{2}](\omega +\phi _{u})u^{2}\text {d}x\\&=\int _{{\mathbb {R}}^{N}}(\omega +\phi _{u})\phi _{u}u^{2}\text {d}x. \end{aligned} \end{aligned}$$

For each fixed $u\in {\mathcal {H}}$, we set

$$\begin{aligned} {\mathcal {L}}_{1}=\{x\in {\mathbb {R}}^{N}:\varphi _{u}+\omega +\phi _{u}\le 0\} \end{aligned}$$

and

$$\begin{aligned} (\varphi _{u}+\omega +\phi _{u})^{-}=\min \{\varphi _{u}+\omega +\phi _{u},0\}. \end{aligned}$$

Then, multiplying (2.2) and (2.5) by $\varphi _{u}$ and integrating on ${\mathcal {L}}_{1}$, respectively, we have

$$\begin{aligned} \int _{{\mathcal {L}}_{1}}\varphi _{u}(-\Delta )^{s}\phi _{u}\text {d}x= -\int _{{\mathcal {L}}_{1}}\varphi _{u}(\phi _{u}+\omega )u^{2}\text {d}x \end{aligned}$$

(2.6)

and

$$\begin{aligned} \int _{{\mathcal {L}}_{1}}\varphi _{u}(-\Delta )^{s}\varphi _{u}\text {d}x=- \int _{{\mathcal {L}}_{1}}\varphi _{u}(\varphi _{u}+\omega +\phi _{u})u^{2}\text {d}x. \end{aligned}$$

(2.7)

Then, multiplying (2.5) by $(\varphi _{u}+\omega +\phi _{u})^{-}$ and integrating on ${\mathcal {L}}_{1}$, we have

$$\begin{aligned} \begin{aligned} -\int _{{\mathcal {L}}_{1}}(\varphi _{u}+\omega +\phi _{u})^{2}u^{2}\text {d}x&=\int _{{\mathcal {L}}_{1}}(-\Delta )^{s}\varphi _{u}(\varphi _{u}+\omega +\phi _{u})\text {d}x\\&=\int _{{\mathcal {L}}_{1}}\varphi _{u}(-\Delta )^{s}(\varphi _{u}+\omega +\phi _{u})\text {d}x\\&=\int _{{\mathcal {L}}_{1}}\varphi _{u}(-\Delta )^{s}\varphi _{u}\text {d}x +\int _{{\mathcal {L}}_{1}}\varphi _{u}(-\Delta )^{s}\phi _{u}\text {d}x. \end{aligned} \end{aligned}$$

According to (2.6) and (2.7), we get that

$$\begin{aligned} \int _{{\mathcal {L}}_{1}}(\varphi _{u}+\omega +\phi _{u})^{2}u^{2}\text {d}x =\int _{{\mathcal {L}}_{1}}\varphi _{u}(\varphi _{u}+\omega +\phi _{u})u^{2}\text {d}x +\int _{{\mathcal {L}}_{1}}\varphi _{u}(\phi _{u}+\omega )u^{2}\text {d}x. \end{aligned}$$

Then

$$\begin{aligned} \int _{{\mathcal {L}}_{1}}(\varphi _{u}+\omega +\phi _{u})(\omega +\phi _{u}) u^{2}-\varphi _{u}(\phi _{u}+\omega )u^{2}\text {d}x=0. \end{aligned}$$

Thus, we obtain

$$\begin{aligned} \int _{{\mathcal {L}}_{1}}(\omega +\phi _{u})^{2}u^{2}\text {d}x=0. \end{aligned}$$

This means that $\varphi _{u}\ge -\omega -\phi _{u}$.

Using the same method, we can get that $\varphi _{u}\ge \phi _{u}$. Finally, by (2.4), we can easily know that $\varphi _{u}\le 0$. $\square$

Define

$$\begin{aligned} {\mathcal {M}}:=\left\{ u\in {\mathcal {H}}\backslash \{0\}:\langle {\mathcal {I}}'(u),u\rangle =0\right\} . \end{aligned}$$

Then, for any solution u of (1.1), we have $u\in {\mathcal {M}}$.

Lemma 2.3

For every $u\in {\mathcal {M}}$ and $2<\alpha <2_{s}^{*}$, there exists a constant $\delta >0$, such that $\Vert u\Vert >\delta$.

Proof

For any $u\in {\mathcal {M}}$, according to Sobolev inequality and Lemma 2.1, we have

$$\begin{aligned} \begin{aligned} \langle {\mathcal {I}}'(u),u\rangle&=\Arrowvert u\Arrowvert ^{2}-\int _{{\mathbb {R}}^{N}}(2\omega +\phi _{u})\phi _{u}u^{2}\text {d}x -\lambda \int _{{\mathbb {R}}^{N}}|u|^{\alpha }\text {d}x-\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x\\&\ge \Arrowvert u\Arrowvert ^{2}-\lambda C_{1}\Arrowvert u\Arrowvert ^{\alpha }-C_{2}\Arrowvert u\Arrowvert ^{2_{s}^{*}}. \end{aligned} \end{aligned}$$

This shows that there exists $\delta >0$, such that $\Vert u\Vert >\delta$. $\square$

Lemma 2.4

${\mathcal {M}}$ is a $C^{1}$ manifold.

Proof

Set

$$\begin{aligned} {\mathcal {G}}(u):=\langle {\mathcal {I}}'(u),u\rangle . \end{aligned}$$

Moreover, we can find that

$$\begin{aligned} {\mathcal {G}}(u)={} & {} 2{\mathcal {I}}(u)-\int _{{\mathbb {R}}^{N}}(\omega +\phi _{u}) \phi _{u}u^{2}\text {d}x\\{} & {} +\dfrac{\lambda (2-\alpha )}{\alpha }\int _{{\mathbb {R}} ^{N}}|u|^{\alpha }\text {d}x-\dfrac{2_{s}^{*}-2}{2_{s}^{*}}\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x. \end{aligned}$$

Thus, for any $u\in {\mathcal {M}}$, by Lemma 2.1 and Lemma 2.2, we have

$$\begin{aligned} \begin{aligned} \langle {\mathcal {G}}'(u),u\rangle&=2\langle {\mathcal {I}}'(u),u\rangle -2 \int _{{\mathbb {R}}^{N}}(\omega +\phi _{u})\phi _{u}u^{2} \text {d}x-(2_{s}^{*}-2)\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x\\&\quad \ -4\int _{{\mathbb {R}}^{N}}\phi _{u}\varphi _{u}u^{2}\text {d}x +\lambda (2-\alpha )\int _{{\mathbb {R}}^{N}}|u|^{\alpha }\text {d}x-2 \int _{{\mathbb {R}}^{N}}\omega \varphi _{u}u^{2}\text {d}x\\&=\lambda (2-\alpha )\int _{{\mathbb {R}}^{N}}|u|^{\alpha }\text {d}x-4 \int _{{\mathbb {R}}^{N}}(\omega +\phi _{u}+\varphi _{u})\phi _{u}u^{2}\text {d}x \\&\quad \ -(2_{s}^{*}-2)\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x. \end{aligned} \end{aligned}$$

Since $\langle {\mathcal {I}}'(u),u\rangle =0$, that is

$$\begin{aligned} \begin{aligned} \lambda (2-\alpha )\int _{{\mathbb {R}}^{N}}|u|^{\alpha }\text {d}x&=(2-\alpha )\bigg (\int _{{\mathbb {R}}^{N}}|(-\Delta )^ {\frac{s}{2}}u|^{2}\text {d}x+\int _{{\mathbb {R}}^{N}}V(x)u^{2}\text {d}x\\&\quad \ -\int _{{\mathbb {R}}^{N}}(2\omega +\phi _{u})\phi _{u}u^ {2}\text {d}x-\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x\bigg ). \end{aligned} \end{aligned}$$

Therefore

$$\begin{aligned} \begin{aligned} \langle {\mathcal {G}}'(u),u\rangle&\le (2-\alpha )\int _{{\mathbb {R}}^{N}} \bigg (|(-\Delta )^{\frac{s}{2}}u|^{2}+V_{*}u^{2}\bigg ) \text {d}x-4\int _{{\mathbb {R}}^{N}}\varphi _{u}\phi _{u}u^{2}\text {d}x\\&\quad \ -2(4-\alpha )\int _{{\mathbb {R}}^{N}}\omega \phi _{u}u^ {2}\text {d}x+(\alpha -6)\int _{{\mathbb {R}}^{N}}\phi _{u}^{2}u^{2}\text {d}x\\&=(2-\alpha )\int _{{\mathbb {R}}^{N}}\bigg (|(-\Delta )^ {\frac{s}{2}}u|^{2}+V_{*}u^{2}\bigg )\text {d}x-4\int _{{\mathbb {R}}^{N}} \varphi _{u}\phi _{u}u^{2}\text {d}x\\&\quad \ +(\alpha -2)\int _{{\mathbb {R}}^{N}}(\omega \varphi _{u} -\phi _{u}^{2})u^{2}\text {d}x+(\alpha -6)\int _{{\mathbb {R}}^{N}}\omega \varphi _{u}u^{2}\text {d}x\\&\le (2-\alpha )\int _{{\mathbb {R}}^{N}}\bigg (|(-\Delta )^{\frac{s}{2}}u| ^{2}+V_{*}u^{2}\bigg )\text {d}x-4\int _{{\mathbb {R}}^{N}}\varphi _{u}^{2}u^{2}\text {d}x\\&\quad \ +2(\alpha -4)\int _{{\mathbb {R}}^{N}}\omega \varphi _{u}u^{2}\text {d}x\\&\le (2-\alpha )\int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u| ^{2}\text {d}x-\int _{{\mathbb {R}}^{N}}K_{u}u^{2}\text {d}x, \end{aligned} \end{aligned}$$

where $K_{u}:=4\varphi _{u}^{2}-2(\alpha -4)\omega \varphi _{u}+(\alpha -2)V_{*}$.

Set

$$\begin{aligned} \begin{aligned} \Psi (t)&=4t^{2}-2(\alpha -4)\omega t+(\alpha -2)V_{*}\\&=4\bigg [t-\dfrac{\omega (\alpha -4)}{4}\bigg ]^{2} -\dfrac{\omega ^{2}(\alpha -4)^{2}}{4}+(\alpha -2)V_{*} \end{aligned} \end{aligned}$$

for $t\in [-\omega ,0]$. We first consider the case of $4\le \alpha <2_{s}^{*}$ and $V_{*}>0$. By $\omega >0$, we have

$$\begin{aligned} \begin{aligned} \Psi '(t)=8t-2(\alpha -4)\omega \le 0,\quad \ t\in [-\omega ,0]. \end{aligned} \end{aligned}$$

Thus, $\Psi$ is monotonically decreasing as $t\in [-\omega ,0]$, that is

$$\begin{aligned} \Psi (t)\ge \Psi (0)=(\alpha -2)V_{*}. \end{aligned}$$

The other case is that $2<\alpha <4$ and $\frac{V_{*}}{\omega ^{2}}>\frac{(\alpha -4)^{2}}{4(\alpha -2)}$. Then

$$\begin{aligned} -\frac{\omega }{2}<\frac{\omega (\alpha -4)}{4}<0. \end{aligned}$$

Therefore, we have

$$\begin{aligned} \Psi (t)\ge \Psi \bigg (\dfrac{\omega (\alpha -4)}{4}\bigg ) =(\alpha -2)V_{*}-\dfrac{\omega ^{2}(\alpha -4)^{2}}{4}>0,\quad \ t\in [-\omega ,0]. \end{aligned}$$

Therefore, as mentioned above, for $t\in [-\omega ,0]$, we have

$$\begin{aligned} \langle {\mathcal {G}}'(u),u\rangle \le (2-\alpha )\int _{{\mathbb {R}}^{N}}| (-\Delta )^{\frac{s}{2}}u|^{2}\text {d}x-K_{\star }\int _{{\mathbb {R}}^{N}}u^{2}\text {d}x\le C, \end{aligned}$$

where $C<0$ is a constant and

$$\begin{aligned} \begin{aligned} K_{\star }= \left\{ \begin{array}{ll} (\alpha -2)V_{*},&{} \quad 4\le \alpha<2_{s}^{*},\\ (\alpha -2)V_{*}-\dfrac{\omega ^{2}(\alpha -4)^{2}}{4},&{}\quad 2<\alpha <4. \end{array} \right. \end{aligned} \end{aligned}$$

Thus, ${\mathcal {G}}'(u)\ne 0$ for any $u\in {\mathcal {M}}$, and by the implicit function Theorem, ${\mathcal {M}}$ is a $C^{1}$ manifold. $\square$

Lemma 2.5

There exists a positive constant $C_{0}>0$, such that ${\mathcal {I}}(u)\ge C_{0}$ for any $u\in {\mathcal {M}}$.

Proof

For any $u\in {\mathcal {M}}$, we have

$$\begin{aligned} \begin{aligned} {\mathcal {I}}(u)&={\mathcal {I}}(u)-\dfrac{1}{\alpha }\langle {\mathcal {I}}'(u),u\rangle \\&=\bigg (\dfrac{1}{2}-\dfrac{1}{\alpha }\bigg )\int _{{\mathbb {R}}^{N}}| (-\Delta )^{\frac{s}{2}}u|^{2}+V(x)u^{2}\text {d}x-\bigg (\dfrac{1}{2} -\dfrac{2}{\alpha }\bigg )\int _{{\mathbb {R}}^{N}}\omega \phi _{u}u^{2}\text {d}x\\&\quad \ +\dfrac{1}{\alpha }\int _{{\mathbb {R}}^{N}}\phi _{u}^{2}u^{2}\text {d}x -\bigg (\dfrac{1}{2_{s}^{*}}-\dfrac{1}{\alpha }\bigg )\int _{{\mathbb {R}}^{N}}|u|^{2_{s}^{*}}\text {d}x\\&\ge \bigg (\dfrac{1}{2}-\dfrac{1}{\alpha }\bigg )\int _{{\mathbb {R}}^{N}}|(-\Delta ) ^{\frac{s}{2}}u|^{2}+V_{*}(x)u^{2}\text {d}x-\bigg (\dfrac{1}{2}-\dfrac{2}{\alpha }\bigg ) \int _{{\mathbb {R}}^{N}}\omega \phi _{u}u^{2}\text {d}x\\&\quad \ +\dfrac{1}{\alpha }\int _{{\mathbb {R}}^{N}}\phi _{u}^{2}u^{2}\text {d}x\\&=\bigg (\dfrac{1}{2}-\dfrac{1}{\alpha }\bigg )\int _{{\mathbb {R}}^{N}}|(-\Delta ) ^{\frac{s}{2}}u|^{2}\text {d}x+\dfrac{1}{2\alpha }\int _{{\mathbb {R}}^{N}}W_{u}u^{2}\text {d}x, \end{aligned} \end{aligned}$$

where

$$\begin{aligned} W_{u}=2\phi _{u}^{2}-(\alpha -4)\omega \phi _{u}+(\alpha -2)V_{*}. \end{aligned}$$

Set

$$\begin{aligned} \begin{aligned} \Lambda (t)&=2t^{2}-(\alpha -4)\omega t+(\alpha -2)V_{*}\\&=2\bigg (t-\dfrac{\omega (\alpha -4)}{4}\bigg )^{2} -\dfrac{\omega ^{2}(\alpha -4)^{2}}{8}+(\alpha -2)V_{*} \end{aligned} \end{aligned}$$

for $t\in [-\omega ,0]$. In what follows, we divided into two cases discussion. Similar to the proof of lemma 2.4, we have

$$\begin{aligned} \begin{aligned} {\mathcal {I}}(u)&\ge \bigg (\dfrac{1}{2}-\dfrac{1}{\alpha }\bigg ) \int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u|^{2}\text {d}x +\dfrac{1}{2\alpha }W_{\star }\int _{{\mathbb {R}}^{N}}u^{2}\text {d}x\\&\ge C\Arrowvert u\Arrowvert ^{2} \ge C_{0}, \end{aligned} \end{aligned}$$

(2.8)

where $C>0$, $C_{0}>0$ are constants and

$$\begin{aligned} \begin{aligned} W_{\star }= \left\{ \begin{array}{ll} (\alpha -2)V_{*},&{} \quad 4\le \alpha<2_{s}^{*},\\ (\alpha -2)V_{*}-\dfrac{\omega ^{2}(\alpha -4)^{2}}{8}>0,&{}\quad 2<\alpha <4. \end{array} \right. \end{aligned} \end{aligned}$$

$\square$

Lemma 2.6

Any bounded sequence $\{u_{n}\}_{n\in {\mathbb {N}}}\subset {\mathcal {M}}$ does not vanish.

Proof

According to concentration–compactness principle of Lions [8, 9], we get the definition of vanish sequence, that is, for any fixed $R >0$, there holds

$$\begin{aligned} \lim _{n\rightarrow \infty }\sup _{y\in {\mathbb {R}}^{N}}\int _{B_{R}(y)}|u_{n}|^{2}\text {d}x=0. \end{aligned}$$

Therefore, for bounded sequence $\{u_{n}\}_{n\in {\mathbb {N}}}\subset {\mathcal {M}}$, we suppose by contradiction that there exists $r>0$ and a sequence $\{y_{n}\}_{n\in {\mathbb {N}}}$, such that

$$\begin{aligned} \lim _{n\rightarrow \infty }\sup _{y_{n}\in {\mathbb {R}}^{N}}\int _{B_{r}(y_{n})}|u_{n}|^{2}\text {d}x=0. \end{aligned}$$

According to [11, Lemma 2.4], we find that $u_{n}\rightarrow 0$ in $L^{q}({\mathbb {R}}^{N})$ for $2<q<2_{s}^{*}$. Then, by Hölder inequality, we have

$$\begin{aligned} -\int _{{\mathbb {R}}^{N}}(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}^{2}\text {d}x\le -2 \int _{{\mathbb {R}}^{N}}\omega \phi _{u_{n}}u_{n}^{2}\text {d}x\le C_{1}|u_{n}|_{12/5}^{2}\rightarrow 0 \end{aligned}$$

(2.9)

and

$$\begin{aligned} \lambda |u_{n}|_{\alpha }^{\alpha }\rightarrow 0 \end{aligned}$$

(2.10)

as $n\rightarrow \infty$, where $C_{1}$ is a constant. Thus

$$\begin{aligned} o_{n}(1)=\langle {\mathcal {I}}'(u_{n}),u_{n}\rangle =\Arrowvert u_{n}\Arrowvert ^{2}- |u_{n}|_{2_{s}^{*}}^{2_{s}^{*}}. \end{aligned}$$

However, by Lemma 2.5, (2.9) and (2.10), we have

$$\begin{aligned} C_{2}\bigg (\Arrowvert u_{n}\Arrowvert ^{2}-|u_{n}|_{2_{s}^{*}}^{2_{s}^{*}}\bigg )\ge {\mathcal {I}}(u_{n})=\dfrac{1}{2}\Arrowvert u_{n}\Arrowvert ^{2}-\dfrac{1}{2_{s}^{*}}|u_{n}|_{2_{s}^{*}}^{2_{s}^{*}}+o_{n}(1)\ge C_{0}>0, \end{aligned}$$

where $C_{0}$ and $C_{2}$ are positive constants. Then, we have

$$\begin{aligned} \bigg (\Arrowvert u_{n}\Arrowvert ^{2}-|u_{n}|_{2_{s}^{*}}^{2_{s}^{*}}\bigg )\ge \dfrac{C_{0}}{C_{2}}>0. \end{aligned}$$

This is contradictory. $\square$

Lemma 2.7

(see [10]) If $u_{n}\rightharpoonup u_{0}$ in ${\mathcal {H}}$, then, up to subsequences, $\phi _{u_{n}}\rightharpoonup \phi _{u_{0}}$ in $D^{s}({\mathbb {R}}^{N})$ as $n\rightarrow \infty$.

3 Proof of Theorem 1.1

In this section, we will devote to the proof of Theorem 1.1.

Proof

Let us assume that $c_{\star }:=\inf \limits _{u\in {\mathcal {M}}}{\mathcal {I}}(u)$, sequence $\{u_{n}\}_{n\in {\mathbb {N}}}\subset {\mathcal {M}}$ and satisfies ${\mathcal {I}}(u_{n})\rightarrow c_{\star }$ as $n\rightarrow \infty$. By Lemma 2.5, we know that $c_{\star }>0$. For sufficiently large n, by virtue of (2.8), we have

$$\begin{aligned} c_{\star }+1\ge {\mathcal {I}}(u_{n})\ge C\Arrowvert u\Arrowvert ^{2}, \end{aligned}$$

which implies that $\{u_{n}\}_{n\in {\mathbb {N}}}$ is bounded in ${\mathcal {H}}$. According to Lemma 2.6, there exist $C>0$, $r>0$ and a sequence $\{y_{n}\}_{n\in {\mathbb {N}}}$, such that

$$\begin{aligned} \int _{B_{r}(y_{n})}u_{n}^{2}\text {d}x\ge C. \end{aligned}$$

Let ${\bar{u}}_{n}(x)=u_{n}(x+y_{n})$. According to the invariance of translations, $\{{\bar{u}}_{n}\}_{n\in {\mathbb {N}}}$ is also bounded, and it satisfies

$$\begin{aligned} \int _{B_{r}}{\bar{u}}_{n}^{2}\text {d}x\ge C. \end{aligned}$$

(3.1)

By virtue of $(V_{1})$ and $\phi _{u_{n}}(x+y_{n})=\phi _{{\bar{u}}_{n}}(x)$, it follows that $\Arrowvert {\bar{u}}_{n}\Arrowvert =\Arrowvert u_{n}\Arrowvert$, ${\mathcal {I}}({\bar{u}}_{n})={\mathcal {I}}(u_{n})$ and ${\mathcal {I}}({\bar{u}}_{n})\rightarrow c_{\star }$ as $n\rightarrow \infty$. In the sense of subsequence, there exists ${\bar{u}}_{0}\in {\mathcal {H}}$, such that

$$\begin{aligned} \begin{aligned}&{\bar{u}}_{n}\rightharpoonup {\bar{u}}_{0}\quad in\ {\mathcal {H}},\\&{\bar{u}}_{n}\rightarrow {\bar{u}}_{0}\quad in\ L^{q}(K)\ for\ 2\le q<2_{s}^{*},\\&{\bar{u}}_{n}\rightarrow {\bar{u}}_{0}\ a.e. \ on\ {\mathbb {R}}^{N}, \end{aligned} \end{aligned}$$

where compact set $K\subset {\mathbb {R}}^{N}$. According to Lemma 2.7, we get that $\phi _{{\bar{u}}_{n}}\rightharpoonup \phi _{{\bar{u}}_{0}}$ in $D^{s}({\mathbb {R}}^{N})$ as $n\rightarrow \infty$. Then, we have

$$\begin{aligned} \begin{aligned}&\phi _{{\bar{u}}_{n}}\rightarrow \phi _{{\bar{u}}_{0}}\quad in\ L^{q}(K)\ for\ 2\le q<2_{s}^{*},\\&\phi _{{\bar{u}}_{n}}\rightarrow \phi _{{\bar{u}}_{0}}\ a.e. \ on\ {\mathbb {R}}^{N}, \end{aligned} \end{aligned}$$

where compact set $K\subset {\mathbb {R}}^{N}$. According to [13], without lost of generality, we can suppose that $\{{\bar{u}}_{n}\}_{n\in {\mathbb {N}}}$ is a Palais–Smale sequence that satisfies

$$\begin{aligned} {\mathcal {I}}({\bar{u}}_{n})\rightarrow c_{\star } \quad and \quad ({\mathcal {I}}|_{{\mathcal {M}}})'({\bar{u}}_{n})\rightarrow 0, \quad as\ n\rightarrow \infty . \end{aligned}$$

(3.2)

Then, for suitable Lagrange multipliers $\mu _{n}$, we have

$$\begin{aligned} o_{n}(1)=\langle ({\mathcal {I}}|_{{\mathcal {M}}})'({\bar{u}}_{n}), {\bar{u}}_{n}\rangle =\langle {\mathcal {I}}'({\bar{u}}_{n}), {\bar{u}}_{n}\rangle +\mu _{n}\langle {\mathcal {G}}'({\bar{u}}_{n}), {\bar{u}}_{n}\rangle =\mu _{n}\langle {\mathcal {G}}'({\bar{u}}_{n}),{\bar{u}}_{n}\rangle . \end{aligned}$$

According to Lemma 2.4, we obtain that $\mu _{n}=o_{n}(1)$, and by (3.2), it follows that $\langle {\mathcal {I}}'({\bar{u}}_{n}),{\bar{u}}_{n}\rangle =o_{n}(1)$. In view of Lemma 2.7 and (3.1), we obtain that ${\mathcal {I}}'({\bar{u}}_{0})=0$ and ${\bar{u}}_{0}\ne 0$. Then, ${\bar{u}}_{0}\in {\mathcal {M}}$. In what follows, we are going to prove that ${\mathcal {I}}({\bar{u}}_{0})=c_{\star }$.

By $\{{\bar{u}}_{n}\}_{n\in {\mathbb {N}}}\subset {\mathcal {M}}$, we get

$$\begin{aligned} \begin{aligned} {\mathcal {I}}({\bar{u}}_{n})&={\mathcal {I}}({\bar{u}}_{n}) -\dfrac{1}{\alpha }\langle {\mathcal {I}}'({\bar{u}}_{n}),{\bar{u}}_{n}\rangle \\&=\bigg (\dfrac{1}{2}-\dfrac{1}{\alpha }\bigg )\Arrowvert {\bar{u}}_{n}\Arrowvert -\bigg (\dfrac{1}{2}-\dfrac{2}{\alpha }\bigg )\int _{{\mathbb {R}}^{N}}\omega \phi _{{\bar{u}}_{n}}{\bar{u}}_{n}^{2}\text {d}x+\dfrac{1}{\alpha } \int _{{\mathbb {R}}^{N}}\phi _{{\bar{u}}_{n}}^{2}{\bar{u}}_{n}^{2}\text {d}x\\&\quad \ -\bigg (\dfrac{1}{2_{s}^{*}}-\dfrac{1}{\alpha }\bigg ) \int _{{\mathbb {R}}^{N}}|{\bar{u}}_{n}|^{2_{s}^{*}}\text {d}x. \end{aligned} \end{aligned}$$

Now, let us discuss the following two cases. We first consider $4\le \alpha <2_{s}^{*}$ and $V_{*}>0$. In view of weakly lower semicontinuity of the norm, Fatou’s lemma, we have

$$\begin{aligned} \begin{aligned} c_{\star }&=\lim _{n\rightarrow \infty }{\mathcal {I}}({\bar{u}}_{n})\\&=\liminf _{n\rightarrow \infty }\bigg (\dfrac{\alpha -2}{2\alpha } \Arrowvert {\bar{u}}_{n}\Arrowvert -\dfrac{\alpha -2}{2\alpha } \int _{{\mathbb {R}}^{N}}\omega \phi _{{\bar{u}}_{n}}{\bar{u}}_{n}^{2} \text {d}x+\dfrac{1}{\alpha }\int _{{\mathbb {R}}^{N}}\phi _{{\bar{u}}_{n}}^{2}{\bar{u}}_{n}^{2}\text {d}x\\&\quad \ -\dfrac{\alpha -2_{s}^{*}}{2_{s}^{*}\alpha } \int _{{\mathbb {R}}^{N}}|{\bar{u}}_{n}|^{2_{s}^{*}}\text {d}x\bigg )\\&\ge \dfrac{\alpha -2}{2\alpha }\Arrowvert {\bar{u}}_{0}\Arrowvert -\dfrac{\alpha -2}{2\alpha }\int _{{\mathbb {R}}^{N}}\omega \phi _{{\bar{u}}_{0}}{\bar{u}}_{0}^{2}\text {d}x+\dfrac{1}{\alpha } \int _{{\mathbb {R}}^{N}}\phi _{{\bar{u}}_{0}}^{2}{\bar{u}}_{0}^{2}\text {d}x\\&\quad \ -\dfrac{\alpha -2_{s}^{*}}{2_{s}^{*}\alpha } \int _{{\mathbb {R}}^{N}}|{\bar{u}}_{0}|^{2_{s}^{*}}\text {d}x\\&={\mathcal {I}}({\bar{u}}_{0})-\dfrac{1}{\alpha }\langle {\mathcal {I}}'({\bar{u}}_{0}) ,{\bar{u}}_{0}\rangle ={\mathcal {I}}({\bar{u}}_{0}). \end{aligned} \end{aligned}$$

The other case is that $2<\alpha <4$ and $\frac{V_{*}}{\omega ^{2}}>\frac{(4-\alpha )^{2}}{4(\alpha -2)}$. By $(V_{2})$, we have

$$\begin{aligned} \begin{aligned} {\mathcal {I}}({\bar{u}}_{n})&=\dfrac{\alpha -2}{2\alpha } \int _{{\mathbb {R}}^{N}}|(-\Delta )^{\frac{s}{2}}u|^{2}\text {d}x -\dfrac{\alpha -2_{s}^{*}}{2_{s}^{*}\alpha }\int _{{\mathbb {R}}^{N}}|{\bar{u}}_{n}|^{2_{s}^{*}}\text {d}x\\&\quad \ +\dfrac{1}{2\alpha }\int _{{\mathbb {R}}^{N}}\big [2\phi _{u}^{2} -(\alpha -4)\omega \phi _{u}+(\alpha -2)V(x)\big ]u^{2}\text {d}x. \end{aligned} \end{aligned}$$

According to Lemma 2.5, we have

$$\begin{aligned} 2\phi _{u}^{2}-(\alpha -4)\omega \phi _{u}+(\alpha -2)V(x)\ge V_{*}(\alpha -2)-\dfrac{\omega ^{2}(\alpha -4)^{2}}{8}>0. \end{aligned}$$

Therefore, same argument as before, we can get that ${\mathcal {I}}({\bar{u}}_{0})\le c_{\star }$.

Since ${\mathcal {I}}({\bar{u}}_{0})\ge \inf \limits _{{\bar{u}} \in {\mathcal {M}}}{\mathcal {I}}({\bar{u}})=c_{\star }$, we obtain that ${\mathcal {I}}({\bar{u}}_{0})=c_{\star }$. As mentioned above, we can get that $({\bar{u}}_{0},\phi _{{\bar{u}}_{0}})$ is the ground state solution of system (1.1). According to the weak maximum principle in [12], we can conclude that the solution ${\bar{u}}_{0}$ is positive. $\square$