Existence of Ground State Solutions for Fractional Schrödinger–Poisson Systems with Doubly Critical Growth

Abstract

This paper considers a class of fractional Schrödinger–Poisson type systems with doubly critical growth

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^su+V(x)u-\phi |u|^{2^*_s-3}u=K(x)|u|^{2^*_s-2}u,&{} \text{ in } {\mathbb {R}}^3,\\ (-\Delta )^s\phi =|u|^{2^*_s-1},&{} \text{ in } {\mathbb {R}}^3, \end{array}\right. \end{aligned}$$

where $s\in (3/4,1)$, $2^*_s=\frac{6}{3-2s}$, $V\in L^{\frac{3}{2s}}({\mathbb {R}}^{3})$, $K\in L^{\infty }({\mathbb {R}}^{3})$. By applying the concentration-compactness principle and variational method, the existence of ground state solutions to the systems is derived.

1 Introduction and Main Results

In this paper, we are concerned with the existence of ground state solutions for the following fractional Schrödinger–Poisson type systems with doubly critical terms

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su+V(x)u-\phi |u|^{2^*_s-3}u=K(x)|u|^{2^*_s-2}u,&{} \text{ in } {\mathbb {R}}^3,\\ (-\Delta )^s\phi =|u|^{2^*_s-1},&{} \text{ in } {\mathbb {R}}^3, \end{array}\right. } \end{aligned}$$

(1.1)

where $s\in (3/4,1)$, $2^*_s=\frac{6}{3-2s}$. The operator $(-\Delta )^s$ stands for the fractional Laplacian of order s and can be defined by $(-\Delta )^su={\mathcal {F}}^{-1}(|\xi |^{2s}{\mathcal {F}}u)$, where ${\mathcal {F}}$ is the usual Fourier transform in ${\mathbb {R}}^3$. The fractional Schrödinger equation is of particular interest in fractional quantum mechanics in the study of particles on stochastic fields modeled by Lévy processes introduced by Laskin in Refs. [10, 11]. In recent years, there have been many works about the existence of solutions for fractional Schrödinger–Poisson type systems with critical nonlinearity term (see e.g. [6, 16, 21] and the references therein):

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^su+V(x)u+K(x)\phi u=f(x,u),&{} \text{ in } {\mathbb {R}}^3,\\ (-\Delta )^t\phi =K(x)|u|^{2},&{} \text{ in } {\mathbb {R}}^3, \end{array}\right. } \end{aligned}$$

(1.2)

where $s\in (3/4,1)$, $t\in (0,1)$ are two fixed constants. Using the Pohožaev-Nehari manifold, monotonic trick and global compactness lemma, Teng [16] obtained the existence of ground state solution for the system (1.2) with $f(x,u)=\mu |u|^{q-1}u+|u|^{2^*_s-2}u,\ K(x)=1$. When $f(x,u)=h(x)|u|^{q-1}u+|u|^{2^*_s-2}u,\ V(x)=1$, Yu et al. [21] proved the existence of a positive and a least energy sign-changing solution for system (1.2) via variational methods. Moreover, they showed that the energy of the sign-changing solution is strictly larger than twice that of the ground state solutions. In Ref. [6], Gu, Tang and Zhang considered the general nonlinearity term and showed that the (1.2) has a positive solution in case of $f(x,u)=K(x)g(u)+|u|^{2_s^*-2}u$ by applying variational method. In particular, by applying Nehari manifold method and Ekeland variational principle, Guo et al. [7] obtained the existence and correlation results of the ground state solutions for a class of equations involving fractional Hardy Schrödinger operators and Hardy Sobolev critical exponents. To overcome the lack of compactness, they considered the subcritical auxiliary problem in bounded region, and the compact embedding of the auxiliary problem was proved.

It is well known that (1.1) can be rewritten as a fractional Choquard type equations

$$\begin{aligned} (-\Delta )^s u+V(x)u=\left( \int _{{\mathbb {R}}^3}\frac{|u(y)|^{q}}{|x-y|^{\mu }}\mathrm{d}y\right) |u|^{q-2}u +g(x,u), \end{aligned}$$

(1.3)

with $q=2^*_s-1$, $\mu =3-2s$ and $g(x,u)=K(x)|u|^{2^*_s-2}u$. When V is positive constant, $\mu \in (0,3)$, $q\in (\frac{6-\mu }{3},\frac{6-\mu }{3-2s})$ and $g(x,u)=0$, d’Avenia, Siciliano and Squassina [2] studied the existence, regularity and asymptotic of the solutions for (1.3). Gao et al. [5] derived the existence of ground state solution of Pohožaev-type to (1.3) with general nonlinearities via variational methods. By employing the variational method, Ma and Zhang [15] investigated the critical case and established the existence and multiplicity of weak solutions for (1.3) with $\mu \in (0,3)$, $q=\frac{6-\mu }{3-2s}$ and $g(x,u)=0$.

When $\phi (x)=0$, systems (1.1) reduce to the following fractional Schrödinger type equation

$$\begin{aligned} (-\Delta )^su+V(x)u=f(x,u),\quad x\in {\mathbb {R}}^3. \end{aligned}$$

(1.4)

In Ref. [17], by virtue of the harmonic extension techniques of Caffarelli and Silvestre, Teng and He considered (1.4) and proved the existence of ground state solution through using the concentration-compactness and methods of Brézis and Nirenberg. In the case of $f(x,u)=|u|^{2^*_s-2}u+\lambda g(x,u)$, Li, Teng and Wu [12] proved that (1.4) has a ground state solution for large $\lambda$ by applying the Nehari method under the assumption that V and g are asymptotically periodic in x. Jin and Liu [9] considered (1.4) and obtained the existence of ground state solutions when the potential is not a constant and not radial.

When $s=1$, $K(x)\equiv 1$, systems (1.1) become the following Schrödinger–Poisson systems

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u+V(x)u-\phi |u|^{3}u=f(x,u),&{} \text{ in } {\mathbb {R}}^3,\\ -\Delta \phi =|u|^{5},&{} \text{ in } {\mathbb {R}}^3. \end{array}\right. } \end{aligned}$$

(1.5)

Systems (1.5) describe quantum (nonrelativistic) particles interacting with the electromagnetic field generated by the motion, see [1]. It is well known that system (1.5) can be transformed into a Schrödinger equation with a nonlocal term. Liu [13] considered the existence of positive solution for (1.5) by using mountain pass theorem and the concentration-compactness principle. Using variational method, Wang, Xie and Guan in Ref. [19] studied the existence of a positive ground state solution for the following Schrödinger–Poisson systems

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u+V(x)u+K(x)\phi u=f(x,u),&{} \text{ in } {\mathbb {R}}^3,\\ -\Delta \phi =K(x)|u|^{2},&{} \text{ in } {\mathbb {R}}^3, \end{array}\right. } \end{aligned}$$

(1.6)

By introducing some new tricks, Tang and Chen [18] studied the existence of a ground state solution of Nehari–Pohožaev type and derived a least energy solution under mild assumptions on V and f for systems (1.6) with $K(x)\equiv 1$. In the case of critical nonlinearity term, Liu and Guo [14] proved that systems (1.6) has at least a positive ground state solution via variational method.

Motivated by the above papers, the main purpose of this paper is to consider the existence of ground state solution for systems (1.1). From the technical point of view, there are two difficulties to prove our result. First, since the problem has two critical terms, it is difficult to estimate the minimum energy level on the Nehari manifold. Second, the critical growth in the system presents an obstacle when showing the convergence of the bounded (PS) sequences. To overcome these difficulties, we employ the concentration-compactness principle and variational method to obtain the existence of ground state solutions to systems (1.1).

In Ref. [8], Guo et al. researched the equation $(-\Delta )^s u=|u|^{2_s^*-2}u, u\in D^{s,2}{\mathbb {R}}^3$ and obtained that, for any $\varepsilon >0$ and $y\in {\mathbb {R}}^3$, $u_{\varepsilon ,y}$ is its positive solution, where

$$\begin{aligned} u_{\varepsilon ,y}(x)=(S_s)^{\frac{1}{2_s^*-2}}{\overline{u}}_{\varepsilon ,y}(x),\quad {\overline{u}}_{\varepsilon ,y}(x)=\frac{{\widetilde{u}}_{\varepsilon ,y}(x)}{\Vert {\widetilde{u}}_{\varepsilon ,y}\Vert _{2_s^*}}, \quad {\widetilde{u}}=\kappa (\varepsilon ^2+|x-y|^2)^{-\frac{3-2s}{2}}, \end{aligned}$$

where $k>0$, $S_s$ is given in (2.1). Let, $\Pi$ consist of all the positive solutions for the equation $(-\Delta )^s u=|u|^{2_s^*-2}u, u\in D^{s,2}{\mathbb {R}}^3$ which satisy (2.2).

Define $V^+(x)= \max \{V(x),0\}$, $V^-(x)=\max \{-V(x),0\}$. On V and K, we assume that

$(f_1)$:

$V\in L^{\frac{3}{2s}}({\mathbb {R}}^3)$, $|V^-|_{\frac{3}{2s}}<S_s$;

$(f_2)$:

$K\in L^{\infty }({\mathbb {R}}^{3})$, and $\inf _{x\in {\mathbb {R}}^{3}}K(x)=K_0>0$;

$(f_3)$:

when $K_0=|K|_\infty$, there exists $u\in \Pi$ such that $\int _{{\mathbb {R}}^{3}}V(x)u^2\mathrm{d}x<0$; when $K_0<|K|_\infty$, there exists $u\in \Pi$ such that $T^{2_s^*-2}\le \frac{-3\int _{{\mathbb {R}}^{3}}V(x)u^2\mathrm{d}x}{(3-2s)\int _{{\mathbb {R}}^{3}}[|K|_\infty -K(x)]u^{2^*}_{s}\mathrm{d}x}$, where

$$\begin{aligned} T^{2_s^*-2}=\sqrt{\frac{(3+2s)^2}{36}K_0^2+\frac{3+2s}{3-2s}\left( 1+|V^+|_{\frac{3}{2s}}S_s^{-1}\right) }-\frac{3+2s}{6}K_0. \end{aligned}$$

Now we give the main result of the paper.

Theorem 1.1

Assume that $(f_1)-(f_3)$ hold. Then, the systems (1.1) possesses a ground state solution.

Remark 1.2

Throughout the paper, we denote by $C_i>0$ various positive constants which may vary from line to line and are not essential to the problem.

The paper is organized as follows: in Sect. 2, some preliminary results are presented. Section 3 is dedicated to the proof of Theorem 1.1.

2 Preliminary

In this section, we will give some notations and Lemmas that will be used throughout this paper. Let $L^p({\mathbb {R}}^3),1\le p \le \infty$ be the usual Lebesgue space with the norm $|u|_p=(\int _{{\mathbb {R}}^3}|u|^p\mathrm{d}x)^{\frac{1}{p}}$. We denote the completion of $C_0^\infty ({\mathbb {R}}^3)$ with respect to the norm

$$\begin{aligned} \Vert u\Vert ^2=\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x=\int _{{\mathbb {R}}^3}|\xi |^{2s}|u(\xi )|^2\mathrm{d}\xi \end{aligned}$$

by $D^{s,2}=D^{s,2}({\mathbb {R}}^3)$. Let $((D^{s,2})^{-1},\Vert \cdot \Vert _{(D^{s,2})^{-1}})$ be the dual space of $(D^{s,2},\Vert \cdot \Vert )$. It is well known that $D^{s,2}$ is continuously embedded into $L^{2_s^*}({\mathbb {R}}^3)$, and for any $s\in (0,1)$, there exists a best constant $S_s>0$ such that

$$\begin{aligned} S_s=\inf _{u\in D^{s,2}\backslash \{0\}}\frac{\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u(x)|^2\mathrm{d}x}{\left( \int _{{\mathbb {R}}^3}|u(x)|^{2_s^*}\mathrm{d}x\right) ^{\frac{2}{2_s^*}}}. \end{aligned}$$

(2.1)

For any $u\in \Pi$,

$$\begin{aligned} \Vert u\Vert ^2=|u|^{2_s^*}_{2_s^*}=S_s^{\frac{3}{2s}}. \end{aligned}$$

(2.2)

We observe that by the Lax–Milgram theorem, for given $u\in L^{2_s^*}({\mathbb {R}}^3)$, there exists a unique solution $\phi =\phi _u\in D^{s,2}$ satisfying $(-\Delta )^s\phi =|u|^{2_s^*-1}$ in a weak sense. The function $\phi _u$ is represented by

$$\begin{aligned} \phi _u(x)=C_s\int _{{\mathbb {R}}^3}\frac{|u|^{2_s^*-1}}{|x-y|^{3-2s}}\mathrm{d}y,x\in {\mathbb {R}}^3, \end{aligned}$$

where $C_s=\frac{\Gamma (\frac{3-2s}{2})}{2^{2s}\pi ^{\frac{3}{2}}\Gamma (s)}$, and it has the following properties:

Lemma 2.1

[4] The following properties hold:

• (i) $\phi _u\ge 0$ for all $u\in D^{s,2}$;

• (ii) $\phi _{tu}=t^{2_s^*-1}\phi _u$ for all $t>0$ and $u\in D^{s,2}$;

• (iii) for any $u\in D^{s,2}$,

  $$\begin{aligned} \int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x\le S_s^{-1}|u|_{2_s^*}^{2(2_s^*-1)}; \end{aligned}$$

  (2.3)

• (iv) if $u_n\rightharpoonup u$ in $D^{s,2}$, then, up to a subsequence, $\phi _{u_n}\rightharpoonup \phi _u$ in $D^{s,2}$.

Moreover, the system (1.1) is variational and its solutions are the critical points of the functional defined in $D^{s,2}$ by

$$\begin{aligned} J(u)= & {} \frac{1}{2}\int _{{\mathbb {R}}^3}(|(-\Delta )^{\frac{s}{2}}u|^2+V(x)u^2)\mathrm{d}x-\frac{1}{2(2_s^*-1)}\int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x\\&-\frac{1}{2^*_s}\int _{{\mathbb {R}}^3}K(x)|u|^{2^*_s}\mathrm{d}x. \end{aligned}$$

Obviously, $J\in C^1(D^{s,2},{\mathbb {R}})$ and for any $u,v\in D^{s,2}$,

$$\begin{aligned} J'(u)v= & {} \int _{{\mathbb {R}}^3} ((-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v+V(x)uv )\mathrm{d}x\\&-\int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-3}uv\mathrm{d}x -\int _{{\mathbb {R}}^3}K(x)|u|^{2^*_s-2}uv\mathrm{d}x. \end{aligned}$$

To obtain the ground state solution, we introduce the Nehari manifold

$$\begin{aligned} {\mathcal {N}}=\{u\in D^{s,2}\backslash \{0\}:I(u)=0\}, \end{aligned}$$

where

$$\begin{aligned} I(u)= & {} \int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x+\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x\\&-\int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x- \int _{{\mathbb {R}}^3}K(x)|u|^{2^*_s}\mathrm{d}x. \end{aligned}$$

3 Proof of Theorem 1.1

To prove that the main theorem, we give some lemmas.

Lemma 3.1

Under the assumptions of Theorem 1.1, for any $u\in D^{s,2}\backslash \{0\}$, there exists a unique $t_u>0$ such that $t{_u}u\in {\mathcal {N}}$. Moreover, $J(t_uu)=\max _{t>0}J(tu)$.

Proof

For any $u\in D^{s,2}\backslash \{0\}$, by the Hölder inequality and (2.1), we have

$$\begin{aligned}&\Vert u\Vert ^2+\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x\ge \Vert u\Vert ^2+\int _{{\mathbb {R}}^3}V^+(x)u^2\mathrm{d}x -\frac{|V^-|_{\frac{3}{2s}}}{S_s}\Vert u\Vert ^2 \nonumber \\&\quad \ge \left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \Vert u\Vert ^2. \end{aligned}$$

(3.1)

Let $u\in D^{s,2}\backslash \{0\}$ be fixed and define the function $f:(0,\infty )\rightarrow {\mathbb {R}}$ by $f(t)=J(tu)$. Note that $f'(t)=\langle J'(tu),u\rangle =0$ if and only if $tu\in {\mathcal {N}}$, by simple calculation, we see that

$$\begin{aligned} f'(t)= & {} t\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x+t\int _{{\mathbb {R}}^3}V(x)|u|^2\mathrm{d}x -t^{2\cdot 2_s^*-3}\\&\times \int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x -t^{2_s^*-1}\int _{{\mathbb {R}}^3}K(x)|u|^{2^*_s}\mathrm{d}x\\= & {} t\left( \int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x+\int _{{\mathbb {R}}^3}V(x)|u|^2\mathrm{d}x -t^{2(2_s^*-2)}\right. \\&\left. \times \int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x -t^{2_s^*-2}\int _{{\mathbb {R}}^3}K(x)|u|^{2^*_s}\mathrm{d}x\right) \\= & {} th(t). \end{aligned}$$

It is obvious that h is a non-increasing function for $t>0$ and from (3.1) $\lim _{t\rightarrow 0^+}h(t)=\Vert u\Vert ^2+\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x>0$, $\lim _{t\rightarrow +\infty }h(t)=-\infty$. Hence, there exists a unique $t_u>0$ such that $f'(t_u)=0$, that is $t_uu\in {\mathcal {N}}$. Furthermore, $J(t_uu)=\max _{t>0}J(tu)$. The proof is completed. $\square$

For any $u\in {\mathcal {N}}$, in view of (2.3), (3.1) and Sobolev embedding theorem, one has

$$\begin{aligned} 0=I(u)= & {} \Vert u\Vert ^2+\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x-\int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x-\int _{{\mathbb {R}}^3}K(x)|u|^{2^*_s}\mathrm{d}x\\\ge & {} \left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \Vert u\Vert ^2-C_1|u^{2}|^{(2_s^*-1)}_{2^*_s}-|K|_\infty |u|^{2_s^*}_{2_s^*}\\\ge & {} \left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \Vert u\Vert ^2-C_2\Vert u^{2}\Vert ^{(2_s^*-1)}-C_3\Vert u\Vert ^{2_s^*}, \end{aligned}$$

which implies that there exists $\alpha >0$ such that

$$\begin{aligned} \Vert u\Vert \ge \alpha . \end{aligned}$$

(3.2)

By virtue of (3.1), we see that for any $u\in {\mathcal {N}}$,

$$\begin{aligned} J(u)= & {} J(u)-\frac{1}{2_s^*}I(u) \nonumber \\= & {} \left( \frac{1}{2}-\frac{1}{2_s^*}\right) \Vert u\Vert ^2+\left( \frac{1}{2}-\frac{1}{2_s^*}\right) \int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x \nonumber \\&-\left( \frac{1}{2(2_s^*-1)}-\frac{1}{2_s^*}\right) \int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x \nonumber \\\ge & {} \frac{s}{3}\left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \Vert u\Vert ^2. \end{aligned}$$

(3.3)

Therefore, ${\mathcal {N}}$ is nonempty and bounded below, we can define $m=\inf _{u\in {\mathcal {N}}}J(u)$, by (3.2) and (3.3), $m>0$.

Lemma 3.2

Under the assumptions of Theorem 1.1, for any $u\in {\mathcal {N}}$, $I'(u)\ne 0$.

Proof

For any $u\in {\mathcal {N}}$, it follows from (3.1) and (3.2) that

$$\begin{aligned} \langle I'(u),u\rangle= & {} \langle I'(u),u\rangle -2_s^*I(u) \nonumber \\= & {} (2-2_s^*)\Vert u\Vert ^2+(2-2_s^*)\int _{{\mathbb {R}}^3}V(x)u^{2}\mathrm{d}x+(2-2_s^*)\int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x \nonumber \\\le & {} -\frac{4s}{3-2s}\left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \Vert u\Vert ^2 \nonumber \\\le & {} -\frac{4s}{3-2s}\left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \alpha ^2<0. \end{aligned}$$

(3.4)

Then for any $u\in {\mathcal {N}}$, $I'(u)\ne 0$. The proof is completed. $\square$

Lemma 3.3

Under the assumptions of Theorem 1.1, there exists a bounded sequence $\{u_n\}\subset {\mathcal {N}}$ satisfying $J(u_n)\rightarrow m$ and $J'(u_n)\rightarrow 0$ in $(D^{s,2})^{-1}$.

Proof

Making use of the Ekeland’s variational principle [3, 20], we get that there exists $\{u_n\}\subset {\mathcal {N}}$ and $\{\lambda _n\}\subset {\mathbb {R}}$ such that $J(u_n)\rightarrow m$ and $J'(u_n)-\lambda _nI'(u_n)\rightarrow 0$ in $(D^{s,2})^{-1}$. Thanks to (3.3), one has

$$\begin{aligned} J(u_n)=J(u_n)-\frac{1}{2_s^*}I(u_n)\ge \frac{s}{3}\left( 1-\frac{|V^-|_{\frac{3}{2s}}}{S_s}\right) \Vert u_n\Vert ^2, \end{aligned}$$

which implies $\{u_n\}$ is bounded in $D^{s,2}$. Thus, $0=\langle J'(u_n),u_n\rangle =\lambda _n\langle I'(u_n),u_n\rangle +o(1)$. Combining this with (3.4), there holds $\lambda _n\rightarrow 0$. We deduce from Hölder inequality and the boundedness of $\{u_n\}$ in $D^{s,2}$ that for any $\varphi \in D^{s,2}$,

$$\begin{aligned} |\langle I'(u_n),\varphi \rangle |= & {} \left| \int _{{\mathbb {R}}^3}((-\Delta )^{\frac{s}{2}}u_n(-\Delta )^{\frac{s}{2}}\varphi +V(x)u_n\varphi )\mathrm{d}x\right. \\&\left. -\int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-3}u_n\varphi \mathrm{d}x-\int _{{\mathbb {R}}^3}K(x)|u_n|^{2^*_s-2}u_n\varphi \mathrm{d}x\right| \\\le & {} \Vert u_n\Vert \Vert \varphi \Vert +|V|_{\frac{3}{2s}}|u_n|_{2_s^*}|\varphi |_{2_s^*} +|\phi _{u_n}|_{2_s^*}|u_n|^{2_s^*-2}_{2_s^*}|\varphi |_{2_s^*}\\&+|K|_\infty |u_n|_{2_s^*}^{2_s^*-1}|\varphi |_{2_s^*}, \end{aligned}$$

which ensures that

$$\begin{aligned} \Vert I'(u_n)\Vert _{(D^{s,2})^{-1}}=\sup _{\Vert \varphi \Vert =1,\varphi \in D^{s,2}}|\langle I'(u_n),\varphi \rangle |\le C. \end{aligned}$$

Therefore, we conclude

$$\begin{aligned} \Vert J'(u_n)\Vert _{(D^{s,2})^{-1}}\le \Vert J'(u_n)-\lambda _nI'(u_n)\Vert _{(D^{s,2})^{-1}} +|\lambda _n|\Vert I'(u_n)\Vert _{(D^{s,2})^{-1}}=o(1). \end{aligned}$$

The proof is completed. $\square$

Lemma 3.4

Assume that the assumptions of Theorem 1.1 hold. Then, we have

$$\begin{aligned} m< & {} \left( \frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}\right) ^{\frac{3-2s}{2s}} \left( \frac{\left( 12+3|K|_\infty ^2-3|K|_\infty \sqrt{|K|_\infty ^2+4}\right) s}{6(3+2s)}\right. \\&\left. +\frac{\left( 2|K|_\infty \sqrt{|K|_\infty ^2+4}-2|K|_\infty ^2\right) s^2}{6(3+2s)}\right) S_s^{\frac{3}{2s}}\\=: & {} \Lambda . \end{aligned}$$

Proof

For $u\in \Pi$ satisfying condition $(f_3)$, define

$$\begin{aligned} g(t)= & {} \frac{t^2}{2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x-\frac{t^{2(2_s^*-1)}}{2(2_s^*-1)}\int _{{\mathbb {R}}^3}\phi _{u}|u|^{2_s^*-1}\mathrm{d}x\\&\quad -\frac{t^{2^*_s}}{2^*_s}\int _{{\mathbb {R}}^3}|K|_\infty |u|^{2^*_s}\mathrm{d}x,\ \ t\ge 0. \end{aligned}$$

Since $(-\Delta )^s\phi _{u}=|u|^{2^*_s-1}$, we find

$$\begin{aligned} \int _{{\mathbb {R}}^3}|u|^{2^*_s}\mathrm{d}x= & {} \int _{{\mathbb {R}}^3}(-\Delta )^{\frac{s}{2}}\phi _{u}(-\Delta )^{\frac{s}{2}}|u|\mathrm{d}x \nonumber \\\le & {} \frac{1}{2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}|u||^2\mathrm{d}x+\frac{1}{2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}\phi _{u}|^2\mathrm{d}x \nonumber \\\le & {} \frac{1}{2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x+\frac{1}{2}\int _{{\mathbb {R}}^3}\phi _{u}|u|^{2_s^*-1}\mathrm{d}x. \end{aligned}$$

(3.5)

Hence, invoking (2.2) and (3.5), there holds

$$\begin{aligned} \int _{{\mathbb {R}}^3}\phi _{u}|u|^{2_s^*-1}\mathrm{d}x\ge & {} 2\int _{{\mathbb {R}}^3}|u|^{2^*_s}\mathrm{d}x-\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u|^2\mathrm{d}x\\= & {} 2|u|^{2^*_s}_{2^*_s}-\Vert u\Vert ^2=S_s^{\frac{3}{2s}}. \end{aligned}$$

Consequently,

$$\begin{aligned} g(t)\le & {} \frac{t^2}{2}S_s^{\frac{3}{2s}}-\frac{t^{2(2_s^*-1)}}{2(2_s^*-1)}S_s^{\frac{3}{2s}}-\frac{t^{2^*_s}}{2^*_s}|K|_\infty S_s^{\frac{3}{2s}}\\= & {} \left( \frac{t^2}{2}-\frac{t^{2(2_s^*-1)}}{2(2_s^*-1)}-\frac{t^{2^*_s}}{2^*_s}|K|_\infty \right) S_s^{\frac{3}{2s}}\\= & {} h(t)S_s^{\frac{3}{2s}}. \end{aligned}$$

Thereby, we obtain that

$$\begin{aligned} t^{2^*_s-2}=\frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2} \end{aligned}$$

is the maximum point of h. Substituting it into h, it follows

$$\begin{aligned}&\sup _{t\ge 0}h(t)\\&\quad =\left( \frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}\right) ^{\frac{2}{2_s^*-2}}\left[ \frac{1}{2} -\frac{1}{2(2_s^*-1)}\left( \frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}\right) ^2\right. \\&\qquad \left. -\frac{1}{2^*_s}\left( \frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}\right) |K|_\infty \right] \\&\quad = \left( \frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}\right) ^{\frac{2}{2_s^*-2}}\left( \frac{\left( 12+3|K|_\infty ^2-3|K|_\infty \sqrt{|K|_\infty ^2+4}\right) s}{6(3+2s)}\right. \\&\qquad \left. +\frac{\left( 2|K|_\infty \sqrt{|K|_\infty ^2+4}-2|K|_\infty ^2\right) s^2}{6(3+2s)}\right) . \end{aligned}$$

Thus one has $g(t)\le \Lambda$, for all $t\ge 0$. When $K_0=|K|_\infty$, there exists a unique $t_u>0$ such that $t_uu\in {\mathcal {N}}$ follows from Lemma 3.1. Then, according to the assumption of Theorem 1.1, $J(t_uu)= g(t_u)+\frac{t_u^2}{2}\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x<\Lambda .$ When $K_0<|K|_\infty$, let

$$\begin{aligned} \xi (t)=\frac{t^2}{2}S_s^{\frac{3}{2s}}+\frac{t^2}{2}|V^+|_{\frac{3}{2s}}S_s^{\frac{3-2s}{2s}}-\frac{t^{2(2_s^*-1)}}{2(2_s^*-1)}S_s^{\frac{3}{2s}}-\frac{t^{2^*_s}}{2^*_s}K_0S_s^{\frac{3}{2s}}, \end{aligned}$$

we have $\xi (T)=0$, where T is given in $(f_3)$. From the Hölder inequality, it is easy to verify that $J(tu)\le \xi (t)\le \xi (T)=0$ for $t\ge T$. Combining with Lemma  3.1, there exists a unique $t_u\in (0,T)$ such that $t_uu\in {\mathcal {N}}$ and

$$\begin{aligned} J(t_uu)= & {} g(t_u)+t_u^2\left\{ \frac{t_u^{2^*_s-2}}{2^*_s}\int _{{\mathbb {R}}^3}[|K|_\infty -K(x)]|u|^{2^*_s}\mathrm{d}x+\frac{1}{2}\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x\right\} \\< & {} \Lambda +t_u^2\left\{ \frac{T^{2^*_s-2}}{2^*_s}\int _{{\mathbb {R}}^3}[|K|_\infty -K(x)]|u|^{2^*_s}\mathrm{d}x+\frac{1}{2}\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x\right\} \\\le & {} \Lambda . \end{aligned}$$

In a word, we obtain $m<\Lambda$. The proof is completed. $\square$

Lemma 3.5

Assume that hypothesis $(f_1)$ holds. Then the functional $F:D^{s,2}({\mathbb {R}}^3)\rightarrow \int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x$ is weakly continuous.

Proof

By the Hölder inequality and Sobolev embedding theorem, we know that the functional F is well defined. Let $\{u_n\}\subset D^{s,2}$ and there exists $u\in D^{s,2}$ such that $u_n\rightharpoonup u$ in $D^{s,2}$, $u_n\rightarrow u$ in $L_{loc}^{q}({\mathbb {R}}^{3})$ for any $q\in [1,2_s^*)$, $u_n(x)\rightarrow u(x), a.e.$ in ${\mathbb {R}}^{3}$. By the boundedness of $\{u_n\}$ in $L^{2^*_s}({\mathbb {R}}^{3})$, we get $\{u_n^2\}$ is bounded in $L^{\frac{3}{3-2s}}({\mathbb {R}}^{3})$, that is $|u_n^2|_{\frac{3}{3-2s}}\le M$. From Fatou’s lemma, it is easy to obtain $|u^2|_{\frac{3}{3-2s}}\le M$. For any $\epsilon >0$, it follows from the integral absolute continuity and Levi theorem that, there exists $\delta >0$ and $R>0$, such that when $e\subset {\mathbb {R}}^3$ and $m(e)<\delta$, we have $\int _{e}|V|^{\frac{3}{2s}}\mathrm{d}x<\epsilon ^{\frac{3}{2s}}$ and $\int _{B_R^c}|V|^{\frac{3}{2s}}\mathrm{d}x<\epsilon ^{\frac{3}{2s}}$. Hence, there exists $A\subset B_R$, $m(B_R\backslash A)<\delta$, such that $\{u_n^2\}$ uniformly converges to $u^2$ in A, hence, there exists $n_0$ such that $n>n_0$, for any $x\in A$, we have $|u_n^2(x)-u^2(x)|<\epsilon |A|^{-\frac{3-2s}{3}}$, furthermore, $\int _A|u_n^2-u^2|^{\frac{3}{3-2s}}\mathrm{d}x<\epsilon ^{\frac{3}{3-2s}}|A|^{-1}|A|=\epsilon ^{\frac{3}{3-2s}}$. Therefore, for any $n>n_0$, we have

$$\begin{aligned}&\left| \int _{{\mathbb {R}}^3}Vu_n^2\mathrm{d}x-\int _{{\mathbb {R}}^3}Vu^2\mathrm{d}x\right| =\left| \int _{{\mathbb {R}}^3}(u_n^2- u^2)V\mathrm{d}x\right| \\&\quad \le \int _{B_R^c}|u_n^2-u^2||V|\mathrm{d}x+\int _{B_R\backslash A}|u_n^2-u^2||V|\mathrm{d}x+\int _A|u_n^2-u^2||V|\mathrm{d}x\\&\quad \le \left( \int _{{\mathbb {R}}^{3}}|u_n^2-u^2|^{\frac{3}{3-2s}}\mathrm{d}x\right) ^{\frac{3-2s}{3}}\left( \int _{B_R^c}|V|^{\frac{3}{2s}}\mathrm{d}x\right) ^{\frac{2s}{3}}\\&\qquad +\left( \int _{{\mathbb {R}}^{3}}|u_n^2-u^2|^{\frac{3}{3-2s}}\mathrm{d}x\right) ^{\frac{3-2s}{3}}\left( \int _{B_R\backslash A}|V|^{\frac{3}{2s}}\mathrm{d}x\right) ^{\frac{2s}{3}}\\&\qquad +\left( \int _{A}|u_n^2-u^2|^{\frac{3}{3-2s}}\mathrm{d}x\right) ^{\frac{3-2s}{3}}\left( \int _A|V|^{\frac{3}{2s}}\mathrm{d}x\right) ^{\frac{2s}{3}}\\&\quad \le 4M\epsilon +\epsilon |V|_{\frac{3}{2s}}. \end{aligned}$$

Hence, $\lim _{n\rightarrow \infty }\int _{{\mathbb {R}}^3}V(x)u_n^2\mathrm{d}x=\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x$. The proof is completed. $\square$

Proof of Theorem 1.1

From Lemmas 3.3 and 3.4, there exists a bounded sequence $\{u_n\}\subset {\mathcal {N}}$ satisfying $J(u_n)\rightarrow m\in (0,\Lambda )$ and $J'(u_n)\rightarrow 0$ in $(D^{1,2})^{-1}$. Up to a subsequence, there exists $u\in D^{s,2}$ such that $u_n\rightharpoonup u$ in $D^{s,2}$, $u_n\rightarrow u$ in $L_{loc}^{q}({\mathbb {R}}^{3})$ for any $q\in [1,2_s^*)$, and $u_n(x)\rightarrow u(x),a.e.$ in ${\mathbb {R}}^{3}$. For any $\varphi \in C_0^\infty ({\mathbb {R}}^3)$, we have

$$\begin{aligned} \langle J'(u_n)\varphi \rangle= & {} \int _{{\mathbb {R}}^3}((-\Delta )^{\frac{s}{2}}u_n(-\Delta )^{\frac{s}{2}}\varphi +V(x)u_n\varphi )\mathrm{d}x-\int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-3}u_n\varphi \mathrm{d}x\\&-\int _{{\mathbb {R}}^3}K(x)|u_n|^{2^*_s-2}u_n\varphi \mathrm{d}x. \end{aligned}$$

It follows from Lemma 2.1 that $\phi _{u_n}\rightharpoonup \phi _{u}$ in $D^{s,2}$, which implies $\phi _{u_n}\rightharpoonup \phi _{u}$ in $L^{2_s^*}({\mathbb {R}}^3)$. Then

$$\begin{aligned} \int _{{\mathbb {R}}^3}(\phi _{u_n}-\phi _u)|u|^{2_s^*-3}u\varphi \mathrm{d}x\rightarrow 0,\quad n\rightarrow \infty . \end{aligned}$$

(3.6)

Since $u_n(x)\rightarrow u(x),a.e.$ in ${\mathbb {R}}^{3}$ and

$$\begin{aligned}&\int _{{\mathbb {R}}^3}|\phi _{u_n}\left( |u_n|^{2_s^*-3}u_n-|u|^{2_s^*-3}u\right) |^{\frac{2_s^*}{2_s^*-1}}\mathrm{d}x\le C\left( |\phi _{u_n}|^{\frac{2_s^*}{2_s^*-1}}_{2_s^*}|u_n|^{\frac{2_s^*(2_s^*-2)}{2_s^*-1}}_{2_s^*}\right. \\&\left. \qquad +\,|\phi _{u_n}|^{\frac{2_s^*}{2_s^*-1}}_{2_s^*}|u|^{\frac{2_s^*(2_s^*-2)}{2_s^*-1}}_{2_s^*}\right) \\&\quad \le C, \end{aligned}$$

we have $\phi _{u_n}(|u_n|^{2_s^*-3}u_n-|u|^{2_s^*-3}u)\rightharpoonup 0$ in $L^{\frac{2_s^*}{2_s^*-1}}({\mathbb {R}}^3)$ and thus

$$\begin{aligned} \int _{{\mathbb {R}}^3}\phi _{u_n}(|u_n|^{2_s^*-3}u_n-|u|^{2_s^*-3}u)\varphi \mathrm{d}x\rightarrow 0,\quad n\rightarrow \infty , \end{aligned}$$

which together with (3.6) implies

$$\begin{aligned} \int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-3}u_n\varphi \mathrm{d}x\rightarrow \int _{{\mathbb {R}}^3}\phi _{u}|u|^{2_s^*-3}u\varphi \mathrm{d}x,\quad n\rightarrow \infty . \end{aligned}$$

(3.7)

By applying $u_n(x)\rightarrow u(x),a.e.$ in ${\mathbb {R}}^{3}$ again and

$$\begin{aligned}&\int _{{\mathbb {R}}^3}|K(x)\left( |u_n|^{2_s^*-2}u_n-|u|^{2_s^*-2}u\right) |^{\frac{2_s^*}{2_s^*-1}}\mathrm{d}x\\&\quad \le C\left( |K|_\infty ^{\frac{2_s^*}{2_s^*-1}}|u_n|^{2_s^*}_{2_s^*}+|K|_\infty ^{\frac{2_s^*}{2_s^*-1}}|u|^{2_s^*}_{2_s^*}\right) \\&\quad \le C, \end{aligned}$$

we get that $K(x)(|u_n|^{2_s^*-2}u_n-|u|^{2_s^*-2}u)\rightharpoonup 0$ in $L^{\frac{2_s^*}{2_s^*-1}}({\mathbb {R}}^3)$, and thus

$$\begin{aligned} \int _{{\mathbb {R}}^3}K(x)(|u_n|^{2_s^*-2}u_n-|u|^{2_s^*-2}u)\varphi \mathrm{d}x\rightarrow 0,\ n\rightarrow \infty . \end{aligned}$$

That is,

$$\begin{aligned} \int _{{\mathbb {R}}^3}K(x)|u_n|^{2_s^*-2}u_n\varphi \mathrm{d}x\rightarrow \int _{{\mathbb {R}}^3}K(x)|u|^{2_s^*-2}u\varphi \mathrm{d}x,\quad n\rightarrow \infty . \end{aligned}$$

(3.8)

It is clear that

$$\begin{aligned} \int _{{\mathbb {R}}^3}V(x)u_n\varphi \mathrm{d}x\rightarrow \int _{{\mathbb {R}}^3}V(x)u\varphi \mathrm{d}x,\quad n\rightarrow \infty . \end{aligned}$$

(3.9)

Combining (3.7), (3.8) with (3.9), we derive

$$\begin{aligned} \langle J'(u),\varphi \rangle =\lim _{n\rightarrow \infty }\langle J'(u_n),\varphi \rangle =0. \end{aligned}$$

Therefore, we derive that u is a critical point of J.

In what follows, we will show that $u\ne 0$. Assume by contradiction that $u=0$. It is clear that

$$\begin{aligned} 0= & {} \langle J'(u_n),u_n\rangle =\Vert u_n\Vert ^2+\int _{{\mathbb {R}}^3}V(x)u_n^2\mathrm{d}x-\int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-1}\mathrm{d}x\!\nonumber \\&\quad -\!\int _{{\mathbb {R}}^3}K(x)|u_n|^{2_s^*}\mathrm{d}x. \end{aligned}$$

(3.10)

For convenience, let $d_n=\Vert u_n\Vert ^2$, $a_n=\int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-1}\mathrm{d}x$, $b_n=\int _{{\mathbb {R}}^3}|u_n|^{2_s^*}\mathrm{d}x$. Without loss of generality, we may assume $d_n\rightarrow d$, $a_n\rightarrow a$ and $b_n\rightarrow b$, as $n\rightarrow \infty$. Notice that

$$\begin{aligned} \int _{{\mathbb {R}}^3}|u_n|^{2^*_s}\mathrm{d}x= & {} \int _{{\mathbb {R}}^3}(-\Delta )^{\frac{s}{2}}\phi _{u_n}(-\Delta )^{\frac{s}{2}}|u_n|\mathrm{d}x\\\le & {} \frac{\varepsilon ^2}{2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}|u_n||^2\mathrm{d}x+\frac{1}{2\varepsilon ^2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}\phi _{u_n}|^2\mathrm{d}x\\\le & {} \frac{\varepsilon ^2}{2}\int _{{\mathbb {R}}^3}|(-\Delta )^{\frac{s}{2}}u_n|^2\mathrm{d}x+\frac{1}{2\varepsilon ^2}\int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-1}\mathrm{d}x, \end{aligned}$$

thus, as $n\rightarrow \infty$, we derive

$$\begin{aligned} b\le \frac{1}{2\varepsilon ^2}a+\frac{\varepsilon ^2}{2}d. \end{aligned}$$

From (3.10) and Lemma 3.5, we have

$$\begin{aligned} a\ge \frac{\varepsilon ^2(2-|K|_\infty \varepsilon ^2)}{2\varepsilon ^2+|K|_\infty }d. \end{aligned}$$

Taking $\varepsilon ^2=\frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}$, we arrive at

$$\begin{aligned} a\ge \frac{2+|K|_\infty ^2-|K|_\infty \sqrt{|K|_\infty ^2+4}}{2}d. \end{aligned}$$

It follows from $J(u_n)\rightarrow m$, (3.10) and Lemma 3.5 that

$$\begin{aligned} m= & {} \frac{s}{3}d+\frac{s(3-2s)}{3(3+2s)}a \nonumber \\\ge & {} \frac{\left( 12+3|K|_\infty ^2-3|K|_\infty \sqrt{|K|_\infty ^2+4}\right) s +\left( 2|K|_\infty \sqrt{|K|_\infty ^2+4}-2|K|_\infty ^2\right) s^2}{6(3+2s)}d.\nonumber \\ \end{aligned}$$

(3.11)

On the other hand, combining (2.3) with (3.10), we obtain

$$\begin{aligned} d\le S_s^{-2_s^*}d^{2_s^*-1}+|K|_\infty S_s^{-\frac{2_s^*}{2}}d^{\frac{2_s^*}{2}}. \end{aligned}$$

Therefore we get either (i) $d=0$ or (ii) $d^{\frac{2s}{3-2s}}=\left( \frac{\sqrt{|K|_\infty ^2+4}-|K|_\infty }{2}\right) S_s^{\frac{2_s^*}{2}}$. By substituting it into (3.11), it will come to a contradiction in each case. Thereby, we have $u\ne 0$.

Thus, we show that $u\in {\mathcal {N}}$. According to the weakly lower semi-continuity of norm and Fatou’s Lemma,

$$\begin{aligned} m\le & {} J(u)=J(u)-\frac{1}{2^*_s}I(u)\\= & {} \frac{s}{3}\Vert u\Vert ^2+\frac{s(3-2s)}{3(3+2s)}\int _{{\mathbb {R}}^3}\phi _u|u|^{2_s^*-1}\mathrm{d}x+\frac{s}{3}\int _{{\mathbb {R}}^3}V(x)u^2\mathrm{d}x\\\le & {} \liminf _{n\rightarrow \infty }\left[ \frac{s}{3}\Vert u_n\Vert ^2+\frac{s(3-2s)}{3(3+2s)}\int _{{\mathbb {R}}^3}\phi _{u_n}|u_n|^{2_s^*-1}\mathrm{d}x +\frac{s}{3}\int _{{\mathbb {R}}^3}V(x)u_n^2\mathrm{d}x\right] \\= & {} \liminf _{n\rightarrow \infty }\left( J(u_n)-\frac{1}{2^*_s}I(u_n)\right) =m, \end{aligned}$$

which yields that $J(u)=m$. Therefore, we conclude that u is a ground state solution of system (1.1). The proof is completed. $\square$