Existence and Concentration of Solutions for the Sublinear Fractional Schrödinger–Poisson System

Abstract

In this paper, we consider a class of nonlinear fractional Schrödinger–Poisson system as follows:

$$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle (-\Delta )^{\alpha } u+\lambda V(x)u+ G(x)\phi u=f(x, u), &{}\quad x\in \mathbb {R}^{3},\\ (-\Delta )^{\alpha } \phi =G(x)K_{\alpha }u^{2}, &{}\quad x\in \mathbb {R}^{3},\\ \end{array} \right. \end{aligned}$$

where $\alpha \in (0,1)$, $K_{\alpha }= \frac{\pi ^{-\alpha }\Gamma (\alpha )}{ \pi ^{ -(3-2\alpha )/2}\Gamma ((3-2\alpha )/2)}$, $V\in C(\mathbb {R}^{3})$ is a potential well, $G\in L^{\infty }(\mathbb {R}^{3})$ is nonnegative and $f\in C(\mathbb {R}^{3}\times \mathbb {R})$ satisfies the sublinear condition in $\mathbb {R}^{3}$. By using variational methods and the genus properties in critical point theory, we study the existence and multiplicity of negative energy solutions for the above system. Moreover, the concentration of solutions is also explored on the set $V^{-}(0)$ as $\lambda \rightarrow \infty$.

1 Introduction

This paper deals with the existence of multiple solutions for the following fractional Schrödinger–Poisson system:

$$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle (-\Delta )^{\alpha } u+\lambda V(x)u+ G(x)\phi u=f(x, u), &{}\quad x\in \mathbb {R}^{3},\\ (-\Delta )^{\alpha } \phi =G(x)K_{\alpha }u^{2}, &{}\quad x\in \mathbb {R}^{3},\\ \end{array} \right. \end{aligned}$$

(1.1)

where $\alpha \in (0,1)$ and $K_{\alpha }= \frac{\pi ^{-\alpha }\Gamma (\alpha )}{ \pi ^{ -(3-2\alpha )/2}\Gamma ((3-2\alpha )/2)}$. Here, the fractional Laplacian $(-\Delta )^{\alpha }$ with $\alpha \in (0,1)$ of a function $\omega$ is defined by:

$$\begin{aligned} \mathcal {F}((-\Delta )^{\alpha }\omega )(\xi )=|\xi |^{2\alpha }\mathcal {F}(\omega )(\xi ),~~\forall ~\alpha \in (0,1), \end{aligned}$$

where $\mathcal {F}$ is the Fourier transform, i.e.,

$$\begin{aligned} \mathcal {F}(\omega )(\xi )=\frac{1}{(2\pi )^{\frac{3}{2}}}\int _{\mathbb {R}^{3}}\exp \{-2\pi i\xi \cdot x\}\,\mathrm{d}x. \end{aligned}$$

If $\omega$ is smooth enough, $(-\Delta )^{\alpha }$ can be computed by the following singular integral:

$$\begin{aligned} (-\Delta )^{\alpha }\omega (x)=c_{3,\alpha }\mathrm{P.V.}\int _{\mathbb {R}^{3}}\frac{w(x)-w(y)}{|x-y|^{2\alpha +3}}\,\mathrm{d}y, \end{aligned}$$

where $\mathrm{P.V.}$ is the principal value and $c_{3,\alpha }$ is a normalization constant that depends on 3 and $\alpha$, precisely given by $c_{3,\alpha }=\big (\int _{\mathbb {R}^{3}}\frac{1-\cos \xi _{1}}{|\xi |^{N+2\alpha }}\,\mathrm{d}\xi \big )^{-1}$.

The fractional Schrödinger equation is a fundamental equation of fractional quantum mechanics, which was discovered by Laskin [14, 15] as a result of extending the Feynman path integral, from the Brownian-like to Lévy-like quantum mechanical paths, where the Feynman path integral leads to the classical Schrödinger equation, and the path integral over Lévy trajectories leads to the fractional Schrödinger equation. A basic motivation for the study of problem (1.1) arises in the study of standing wave solutions of the type $\psi (x,t)=\exp (-ict)u(x)$ for the following time-dependent fractional Schrödinger equation:

$$\begin{aligned} i\frac{\partial \psi }{\partial t}+(-\triangle )^{\alpha }\psi +(V(x)-c)\psi =f(x,\psi )+h(x),~~\quad x\in \mathbb {R}^{3}, \end{aligned}$$

where i is the imaginary unit.

When $\phi (x)=0$, Eq. (1.1) reduces to the following fractional Schrödinger equation:

$$\begin{aligned} (-\Delta )^{\alpha } u+V(x)u=f(x, u), ~~x\in \mathbb {R}^{3}. \end{aligned}$$

(1.2)

In recent years, Eq. (1.2) has been widely studied by a number of authors. For instance, by using the mountain pass theorem, Secchi [20] proved that Eq. (1.2) has at least a nontrivial solution when f has subcritical growth and satisfies the famous Ambrosetti-Rabinowitz condition. In [18], when the nonlinearity f(x, u) is subcritical near infinity and superlinear near zero and satisfies the Berestycki-Lions condition, Liu and Ou proved that Eq. (1.2) possesses at least a positive ground state solution by using the Pohozaev manifold technique and the monotonic trick. Later, when V(x) is a steep potential well, Liu et al. [17] studied some qualitative properties of the curve of Dancer–Fuc̆ik point spectrum for fractional Schrödinger operators $(-\Delta )^{\alpha }+V$ in Eq. (1.2). Furthermore, the existence of nontrivial solutions for a class of more general fractional Schrödinger equations with nonresonant nonlinearity is also established by using the properties of the curve. In [28], by virtue of the harmonic extension techniques of Caffarelli and Silvestre [3], Teng and He proved the existence of ground state solutions for Eq. (1.2) by using the concentration compactness principle and methods of Brezis and Nirenberg. For other related results about fractional Schrödinger equations, we refer the reader to Autuori and Pucci [2], Che and Chen [4] and Che et al. [8] and the references therein.

When $\alpha =1$, we may get the classical Schrödinger–Poisson system like:

$$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle -\Delta u+V(x)u+\phi u=f(x, u), &{}\quad x\in \mathbb {R}^{3},\\ -\Delta \phi =u^{2}, &{}\quad x\in \mathbb {R}^{3}.\\ \end{array} \right. \end{aligned}$$

(1.3)

In quantum mechanics, the Schrödinger–Poisson system can be used to describe the interaction of a charged particle with the electrostatic field. Recently, with the development of critical point theory and variational methods, lots of researchers have studied the existence and multiplicity of solutions for problem (1.3), see [9, 21, 22, 24,25,26, 30,31,32] and the references therein. In [26], by using the penalized functions, Sun and Wu obtained the existence and concentration of nontrivial solutions for problem (1.3) with indefinite steep potential well. Later, when V(x) is replaced by $\lambda V(x)$ with V(x) being a steep potential well, and $f(x,u):=a(x)|u|^{p-2}u+b(x)|u|^{q-2}u,~1<q<2<p<4$, Sun and Wu [25] proved that problem (1.3) possesses two positive solutions. Recently, Wu [30] studied the existence and symmetry of ground state solutions for problem (1.3) with $V(x)\equiv 1$ and $f(x,u)=a(x)|u|^{p-2}u,~2<p<3.$

As far as we are concerned, there are few papers in the available literature that considered problem (1.1). In [19], when $G(x)\equiv 1$ and $f(x,u)=f(u)+|u|^{2_{\alpha }^{*}-2}u,$ Liu and Zhang considered existence and multiplicity of positive solutions for problem (1.1) by using the Ljusternik–Schnirelmann theory. Furthermore, the concentration behavior of positive solutions is also obtained. In [7], Che and Chen obtained the existence of nontrivial solutions for problem (1.1) with sign-changing potential. For other related results about problem (1.1), one can refer to Ambrosio [1], Che et al. [6] and Teng [27] and the references therein.

Inspired by the works mentioned above, in the present paper, we are interested in the existence, multiplicity and concentration of nontrivial solutions for problem (1.1) with a steep potential well and sublinear nonlinearity.

We assume that V(x), G(x) and f(x, u) satisfy the following conditions:

• $(V_{1})$ $V\in C(\mathbb {R}^{3})$ and $V(x)\ge 0$ on $\mathbb {R}^{3}$;

• $(V_{2})$ there exists a constant $b > 0$ such that the set $V_{b}=\{x\in \mathbb {R}^{3}: V(x)\le b\}$ is nonempty and has finite Lebesgue measure;

• $(V_{3})$ $\Omega =V^{-}(0) =int \{x\in \mathbb {R}^{3}, V(x)=0\}$ is nonempty and has smooth boundary with $\bar{\Omega }= \{x\in \mathbb {R}^{3}, V(x)=0\}$;

• (G) $G\in L^{\infty }(\mathbb {R}^{3})$, $0\le G(x)\le G_{\infty }$, $G(x)\not \equiv 0$, and $\lim \nolimits _{x\rightarrow \infty }G(x)=0$;

• $(F_{1})$ $f\in C(\mathbb {R}^{3}\times \mathbb {R})$ and there exist $1<\alpha _{1},\alpha _{2}<2$ and positive functions $c_{1}\in L^{\frac{2}{2-\alpha _{1}}}(\mathbb {R}^{3}), ~~c_{2}\in L^{\frac{2}{2-\alpha _{2}}}(\mathbb {R}^{3})$ such that

  $$\begin{aligned} |f(x,u)|\le \alpha _{1}c_{1}(x)|u|^{\alpha _{1}-1}+\alpha _{2}c_{2}(x)|u|^{\alpha _{2}-1}, \quad \forall ~(x,u)\in \mathbb {R}^{3}\times \mathbb {R}; \end{aligned}$$

• $(F_{2})$ there exist a bounded open set $\Lambda \subset \mathbb {R}^{3}$ and three constants $a_{1}, a_{2}>0$ and $a_{3}\in (1,2)$ such that

  $$\begin{aligned} F(x,u)\ge a_{2}|u|^{a_{3}},~~\forall ~(x,u)\in \Lambda \times [-a_{1},a_{1}], \end{aligned}$$

  where $F(x,u)=\int _{0}^{u}f(x,s)\,\mathrm{d}s;$

• $(F_{3})$ $f(x,u)=-f(x,-u)$, for all $(x,u)\in \mathbb {R}^{3}\times \mathbb {R}.$

Now, we state our main results.

Theorem 1.1

Assume that conditions $(V_{1})-(V_{3})$, (G) and $(F_{1})-(F_{2})$ hold, then problem (1.1) possesses at least one nontrivial solution.

Theorem 1.2

Assume that conditions $(V_{1})-(V_{3})$, (G) and $(F_{1})-(F_{3})$ hold, then problem (1.1) possesses infinitely many solutions $\{u_{k}\}$ such that

$$\begin{aligned} \begin{aligned}&\frac{1}{2}\int _{\mathbb {R}^{3}}\big (|(-\Delta )^{\frac{\alpha }{2}}u_{k}|^{2}+ \lambda \int _{\mathbb {R}^{3}}V(x)u_{k}^{2}\big )\,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\mathbb {R}^{3}} G(x)\phi _{u_{k}} u_{k}^{2}\,\mathrm{d}x\\&\quad -\,\int _{\mathbb {R}^{3}}F(x, u_{k})\,\mathrm{d}x\rightarrow 0^{-}, \quad \mathrm {as}~ k\rightarrow \infty . \end{aligned} \end{aligned}$$

Evidently, the assumption $(F_{2})$ holds if the following conditions holds:

$(F_{2}')$ There exists a bounded open set $J\subset \mathbb {R}^{3}$ and three constants $a_{1}, a_{2}>0$ and $a_{3}\in (1,2)$ such that

$$\begin{aligned} f(x,u)u\ge a_{2}a_{3}|u|^{a_{3}},\quad \forall ~ (x,u)\in J\times [-a_{1},a_{1}]. \end{aligned}$$

Therefore, by Theorems 1.1 and 1.2, we have the following corollary.

Corollary 1.1

Assume that conditions $(V_{1})-(V_{3})$, (G), $(F_{1})$ and $(F_{2}')$ hold, then problem (1.1) possesses at least one nontrivial solution. If additionally, $(F_{3})$ holds, then problem (1.1) possesses infinitely many solutions $(u_{k})$ such that

$$\begin{aligned} \begin{aligned}&\frac{1}{2}\int _{\mathbb {R}^{3}}\big (|(-\Delta )^{\frac{\alpha }{2}}u_{k}|^{2}+ \lambda \int _{\mathbb {R}^{3}}V(x)u_{k}^{2}\big )\,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\mathbb {R}^{3}} G(x)\phi _{u_{k}} u_{k}^{2}\,\mathrm{d}x\\&\quad -\,\int _{\mathbb {R}^{3}}F(x, u_{k})\,\mathrm{d}x\rightarrow 0^{-},\quad \mathrm {as}~ k\rightarrow \infty . \end{aligned} \end{aligned}$$

On the concentration of solutions, we have the following result.

Theorem 1.3

Let $(u_{n}, \phi _{n})$ be a solution of problem (1.1) obtained in Theorem 1.1, then $u_{n}\rightarrow \tilde{u}$ in $H^{\alpha }(\mathbb {R}^{3})$, $\phi _{n}\rightarrow \tilde{\phi }$ in $D^{\alpha }(\mathbb {R}^{3})$ as $\lambda _{n}\rightarrow \infty$, where $\tilde{u}\in H^{\alpha }_{0}(\mathbb {R}^{3})$ is a nontrivial solution of the equation:

$$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle (-\Delta )^{\alpha } u+\frac{1}{4\pi }\big ((G(x)u^{2})*\frac{1}{|x|^{3-2\alpha }}\big )G(x) u=f(x, u), &{}\mathrm { in} ~\Omega ,\\ u=0,&{}\mathrm { on} ~\partial \Omega ,\\ \end{array} \right. \end{aligned}$$

(1.4)

where $\Omega$ is given by the condition $(V_{3})$.

Remark 1.1

There are many functions f(x, u) satisfying all the conditions of Theorem 1.2. For example, let

$$\begin{aligned} f(x,u)=\frac{5\sin ^{2}x_{1}}{4(1+e^{|x|})}|u|^{\frac{-3}{4}}u+\frac{4\cos ^{2}x_{1}}{3(1+e^{|x|})}|u|^{\frac{-2}{3}}u, \end{aligned}$$

where $x=\{x_{1},x_{2},x_{3}\}$. Then,

$$\begin{aligned} |f(x,u)|\le \frac{5\sin ^{2}x_{1}}{4(1+e^{|x|})}|u|^{\frac{1}{4}}+\frac{4\cos ^{2}x_{1}}{3(1+e^{|x|})}|u|^{\frac{1}{3}} , \quad \forall ~(x, u)\in (\mathbb {R}^{3}\times \mathbb {R}), \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} F(x,u)&=\frac{\sin ^{2}x_{1}}{1+e^{|x|}}|u|^{\frac{5}{4}}+\frac{\cos ^{2}x_{1}}{1+e^{|x|}}|u|^{\frac{4}{3}}\\&\ge \frac{\cos ^{2}1}{1+e}|u|^{\frac{4}{3}}, ~~\forall ~(x, u)\in J\times [-1.1], \end{aligned} \end{aligned}$$

where

$$\begin{aligned} \frac{5}{4}=\alpha _{1}<\alpha _{2}=\frac{4}{3},~~c_{1}(x)=\frac{\sin ^{2}x_{1}}{1+e^{|x|}},~~ c_{2}(x)=\frac{\cos ^{2}x_{1}}{1+e^{|x|}}, \end{aligned}$$

and

$$\begin{aligned} a_{1}=1,\quad a_{2}=\frac{\cos ^{2}1}{1+e},\quad a_{3}=\frac{4}{3},\quad J=B(0,1). \end{aligned}$$

Notation 1.1. Throughout this paper, we shall denote by $\Vert \cdot \Vert _{r}$ the $L^{r}$-norm and C various positive generic constants, which may vary from line to line. $2_{\alpha }^{*}=\frac{6}{3-2\alpha }$ is the critical Sobolev exponent. Also, if we take a subsequence of a sequence $\{u_{n}\}$, we shall denote it again by $\{u_{n}\}$.

The remainder of this paper is as follows. In Sect. 2, some preliminary results are presented. In Sect. 3, we give the proofs of Theorems 1.1 and 1.2. In Sect. 4, we study the concentration of solutions and prove Theorem 1.3.

2 Preliminaries

In this section, we recall some preliminary results, which will be useful along the paper. First, we will give some useful facts of the fractional order Sobolev spaces.

The fractional Sobolev space $W^{\alpha ,p}(\mathbb {R}^{3})$ is defined for any $p\in [1,+\infty )$ and $\alpha \in (0,1)$ as

$$\begin{aligned} W^{\alpha ,p}(\mathbb {R}^{3})=\bigg \{u\in L^{p}(\mathbb {R}^{3})\big |\int _{\mathbb {R}^{3}} \frac{|u(x)-u(y)|^{p}}{|x-y|^{\alpha p+3}}\,\mathrm{d}x\,\mathrm{d}y<\infty \bigg \}. \end{aligned}$$

This space is endowed with the Gagliardo norm

$$\begin{aligned} \Vert u\Vert _{W^{\alpha ,p}}=\bigg (\int _{\mathbb {R}^{3}}|u|^{p}\,\mathrm{d}x+\int _{\mathbb {R}^{3}} \frac{|u(x)-u(y)|^{p}}{|x-y|^{\alpha p+3}}\,\mathrm{d}x\,\mathrm{d}y\bigg )^{\frac{1}{p}}. \end{aligned}$$

When $p=2$, these spaces are also denoted by $H^{\alpha }(\mathbb {R}^{3})$.

If $p=2$, an equivalent definition of the fractional Sobolev space is possible, based on Fourier analysis. Indeed, it turns out that

$$\begin{aligned} H^{\alpha }(\mathbb {R}^{3})=\bigg \{u\in L^{2}(\mathbb {R}^{3}) \big |\int _{\mathbb {R}^{3}}(1+|\xi |^{2\alpha })|\hat{u}|^{2}\,\mathrm{d}\xi <\infty \bigg \}, \end{aligned}$$

where $\hat{u}=\mathcal {F}(u)$, and the norm can be equivalently written by

$$\begin{aligned} \Vert u\Vert _{H^{\alpha }}=\bigg (\Vert u\Vert _{2}^{2}+\int _{\mathbb {R}^{3}}|\xi |^{2\alpha }|\hat{u}|^{2}\,\mathrm{d}\xi \bigg )^{\frac{1}{2}}. \end{aligned}$$

Furthermore, by Plancherel’s theorem, we have $\Vert u\Vert _{2}=\Vert \hat{u}\Vert _{2}$, and

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{\alpha }{2}}u|^{2}&=\int _{\mathbb {R}^{3}}\big (\widehat{(-\Delta )^ {\frac{\alpha }{2}}u(\xi )}\big )^{2}\,\mathrm{d}\xi =\int _{\mathbb {R}^{3}}\big (|\xi |^{\alpha }\hat{u}(\xi )\big )^{2}\,\mathrm{d}\xi \\&=\int _{\mathbb {R}^{3}}|\xi |^{2\alpha }\hat{u}^{2}\,\mathrm{d}\xi <\infty ,\quad \forall ~u\in H^{\alpha }(\mathbb {R}^{3}). \end{aligned} \end{aligned}$$

Then,

$$\begin{aligned} \Vert u\Vert _{H^{\alpha }}=\bigg (\int _{\mathbb {R}^{3}}\big (u^{2}+|(-\Delta )^{\frac{\alpha }{2}}u|^{2}\big )\,\mathrm{d}x \bigg )^{\frac{1}{2}}. \end{aligned}$$

In this paper, in view of the potential V(x), we consider the space

$$\begin{aligned} E=\bigg \{u\in H^{\alpha }(\mathbb {R}^{3})\big |\int _{\mathbb {R}^{3}}\bigg (|(-\Delta )^{\frac{\alpha }{2}}u|^{2}+V(x)u^{2}\bigg )\,\mathrm{d}x<\infty \bigg \}. \end{aligned}$$

Then, by Laskin [15], E is a Hilbert space with the inner product

$$\begin{aligned} (u,v)_{E}=\int _{\mathbb {R}^{3}}\big (|\xi |^{2\alpha }\hat{u}(\xi )\hat{v}(\xi )+\hat{u}(\xi )\hat{v}(\xi )\big )\,\mathrm{d}\xi +\int _{\mathbb {R}^{3}}V(x)u(x)v(x)\,\mathrm{d}x, \end{aligned}$$

and the norm

$$\begin{aligned} \Vert u\Vert _{E}=\bigg (\int _{\mathbb {R}^{3}}\big (|\xi |^{2\alpha }\hat{u}^{2}+\hat{u}^{2}\big )\,\mathrm{d}\xi + \int _{\mathbb {R}^{3}}V(x)u^{2}\,\mathrm{d}x\bigg )^{\frac{1}{2}}. \end{aligned}$$

Furthermore, we know that $\Vert u\Vert _{E}$ is equivalent to the following norm:

$$\begin{aligned} \Vert u\Vert =\bigg (\int _{\mathbb {R}^{3}}\big (|(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ V(x)u^{2}\big )\,\mathrm{d}x\bigg )^{\frac{1}{2}}. \end{aligned}$$

The corresponding norm is

$$\begin{aligned} (u,v)=\int _{\mathbb {R}^{3}}\big ((-\Delta )^{\frac{\alpha }{2}}u(-\Delta )^{\frac{\alpha }{2}}v +V(x)uv\big )\,\mathrm{d}x,\quad \forall ~ u,~v\in E. \end{aligned}$$

Throughout the paper, we will use the norm $\Vert \cdot \Vert$ in E.

For $\lambda >0$, we also need the following inner product:

$$\begin{aligned} (u,v)_{\lambda }=\int _{\mathbb {R}^{3}}\big ((-\Delta )^{\frac{\alpha }{2}}u(-\Delta )^{\frac{\alpha }{2}}v +\lambda V(x)uv\big )\,\mathrm{d}x,\quad \forall ~ u,~v\in E, \end{aligned}$$

and the corresponding norm

$$\begin{aligned} \Vert u\Vert _{\lambda }=\bigg (\int _{\mathbb {R}^{3}}\big (|(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda V(x)u^{2}\big )\,\mathrm{d}x\bigg )^{\frac{1}{2}}. \end{aligned}$$

Obviously, if $\lambda \ge 1$, then we have $\Vert u\Vert \le \Vert u\Vert _{\lambda }$.

Set $E_{\lambda }=(E, \Vert \cdot \Vert _{\lambda })$, then it follows from $(V_{1})-(V_{2})$ that the embedding $E_{\lambda }\hookrightarrow H^{\alpha }(\mathbb {R}^{3})$ is continuous. Therefore, for every $r\in [2,2_{\alpha }^{*}]$, there exist $\tau _{r}, ~\lambda _{0}>0$ (independent of $\lambda \ge 1$) such that

$$\begin{aligned} \Vert u\Vert _{r}\le \tau _{r}\Vert u\Vert \le \tau _{r}\Vert u\Vert _{\lambda },\quad \text { for all }u\in E,~\lambda \ge \lambda _{0}. \end{aligned}$$

(2.1)

The space $D^{\alpha ,2}(\mathbb {R}^{3})$ is defined as follows:

$$\begin{aligned} D^{\alpha ,2}(\mathbb {R}^{3})=\bigg \{u\in L^{2_{\alpha }^{*}} (\mathbb {R}^{3})\bigg \Vert \xi |^{\alpha }\hat{u}(\xi )\in L^{2}(\mathbb {R}^{3})\bigg \}, \end{aligned}$$

which is defined as the completion of $C_{0}^{\infty }(\mathbb {R}^{3})$ under the norms

$$\begin{aligned} \Vert u\Vert _{D^{\alpha ,2}}=\bigg (\int _{\mathbb {R}^{3}}|\xi |^{2\alpha }\hat{u}^{2}d\xi \bigg )^{\frac{1}{2}}=\bigg (\int _{\mathbb {R}^{3}} |(-\Delta )^{\frac{\alpha }{2}}u|^{2}\,\mathrm{d}x\bigg )^{\frac{1}{2}}. \end{aligned}$$

As usual, for $1\le p<+\infty$, we let

$$\begin{aligned} \Vert u\Vert _{p}=\bigg (\int _{\mathbb {R}^{3}}|u|^{p}\,\mathrm{d}x\bigg )^{\frac{1}{p}},\quad u\in L^{p}(\mathbb {R}^{3}), \end{aligned}$$

and

$$\begin{aligned} \Vert u\Vert _{\infty }=ess ~\sup \limits _{x\in \mathbb {R}^{3}}|u|,\quad u\in L^{\infty }(\mathbb {R}^{3}). \end{aligned}$$

Lemma 2.1

[13] For $1<p<\infty$ and $0<\alpha <\frac{3}{p}$, we have

$$\begin{aligned} \Vert u\Vert _{\frac{p3}{3-p\alpha }}\le B\Vert (-\Delta )^{\frac{\alpha }{2}}u\Vert _{p} \end{aligned}$$

(2.2)

with best constant

$$\begin{aligned} B=2^{-\alpha }\pi ^{\frac{-\alpha }{2}}\frac{\Gamma (\frac{3-\alpha }{2})}{\Gamma (\frac{3+\alpha }{2})} \bigg (\frac{\Gamma (3)}{\Gamma (\frac{3}{2})}\bigg )^{\frac{\alpha }{3}}. \end{aligned}$$

Lemma 2.2

For any $u\in H^{\alpha }(\mathbb {R}^{3})$ and for any $h\in D^{-\alpha ,2}(\mathbb {R}^{3})$, there exists a unique solution $\phi =\big ((-\Delta )^{\alpha }+u^{2}\big )^{-}h\in D^{\alpha ,2}(\mathbb {R}^{3})$ of the equation

$$\begin{aligned} (-\Delta )^{\alpha }\phi +u^{2}\phi =h, \end{aligned}$$

(being $D^{-\alpha ,2}(\mathbb {R}^{3}$) the dual space of $D^{\alpha ,2}(\mathbb {R}^{3}))$. Moreover, for every $u\in H^{\alpha }(\mathbb {R}^{3})$ and for every $h,~g\in D^{-\alpha ,2}(\mathbb {R}^{3})$,

$$\begin{aligned} \langle h, \big ((-\Delta )^{\alpha }+u^{2}\big )^{-}g\rangle =\langle g,\big ((-\Delta )^{\alpha }+u^{2}\big )^{-}h\rangle . \end{aligned}$$

(2.3)

Proof

If $u\in H^{\alpha }(\mathbb {R}^{3})$, then by (2.1) and the Hölder inequality, we have

$$\begin{aligned} \int _{\mathbb {R}^{3}}u^{2}\phi ^{2}\,\mathrm{d}x\le \Vert u\Vert _{2p}^{2}\Vert \phi \Vert _{2q}^{2}\le B^{2}\Vert u\Vert _{2p}^{2}\Vert \phi \Vert _{D^{\alpha },2}^{2}, \end{aligned}$$

(2.4)

where $\frac{1}{p}+\frac{1}{q}=1$, $q=\frac{3}{3-2\alpha }$. Then, $\big (\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{\alpha }{2}}\phi |^{2}+u^{2}\phi ^{2})\,\mathrm{d}x\big )^{\frac{1}{2}}$ is equivalent to $\Vert \phi \Vert _{D^{\alpha ,2}}$. Thus, by using the Lax–Milgram lemma, we obtain the existence result. For every $u\in H^{\alpha }(\mathbb {R}^{3})$ and for every $h,~g\in D^{-\alpha }(\mathbb {R}^{3})$, we have $\phi _{g}=\big ((-\Delta )^{\alpha }+u^{2}\big )^{-}g$, $\phi _{h}=\big ((-\Delta )^{\alpha }+u^{2}\big )^{-}h$. Then,

$$\begin{aligned} \langle h, \big ((-\Delta )^{\alpha }+u^{2}\big )^{-}g\rangle= & {} \int _{\mathbb {R}^{3}}h \big ((-\Delta )^{\alpha }+u^{2}\big )^{-}g\,\mathrm{d}x\\= & {} \int _{\mathbb {R}^{3}}h\phi _{g} \,\mathrm{d}x=\int _{\mathbb {R}^{3}}((-\Delta )^{\alpha }+u^{2}\big )\phi _{h}\phi _{g}\,\mathrm{d}x\\= & {} \int _{\mathbb {R}^{3}}((-\Delta )^{\alpha }\phi _{h}+u^{2}\phi _{h}\big )\phi _{g}\,\mathrm{d}x\\= & {} \int _{\mathbb {R}^{3}}((-\Delta )^{\alpha }\phi _{g}+u^{2}\phi _{g}\big )\phi _{h}\,\mathrm{d}x\\= & {} \int _{\mathbb {R}^{3}}g\phi _{h}\,\mathrm{d}x=\int _{\mathbb {R}^{3}}g \big ((-\Delta )^{\alpha }+u^{2}\big )^{-}h\,\mathrm{d}x\\= & {} \langle g,\big ((-\Delta )^{\alpha }+u^{2}\big )^{-}h\rangle . \end{aligned}$$

The proof is complete. $\square$

Lemma 2.3

[11] Let f be a function in $C_{0}^{\infty }(\mathbb {R}^{3})$ and let $0<\alpha <n$. Then, with

$$\begin{aligned}&c_{\alpha }=\pi ^{\frac{-\alpha }{2}}\Gamma (\frac{-\alpha }{2}), \end{aligned}$$

(2.5)

$$\begin{aligned}&c_{\alpha }(\xi ^{-\alpha }\hat{f}(\xi ))^{\vee }(x) =c_{n-\alpha }\int _{\mathbb {R}^{3}}|x-y|^{\alpha -n}f(y)\,\mathrm{d}y. \end{aligned}$$

(2.6)

Lemma 2.4

For every $u\in H^{\alpha }(\mathbb {R}^{3})$, there exists a unique $\phi =\phi _{u}\in D^{\alpha ,2}(\mathbb {R}^{3})$, which solves the equation $(-\Delta )^{\alpha } \phi =G(x)K_{\alpha }u^{2}, \text { }x\in \mathbb {R}^{3}$. Furthermore, $\phi _{u}$ is given by

$$\begin{aligned} \phi _{u}=\int _{\mathbb {R}^{3}}G(x)|x-y|^{2\alpha -N}u^{2}(y)\,\mathrm{d}y. \end{aligned}$$

(2.7)

As a consequence, the map $\Phi : u\in H^{\alpha }(\mathbb {R}^{3})\mapsto \phi _{u}\in D^{\alpha ,2}(\mathbb {R}^{3})$ is of class $C^{1}$ and

$$\begin{aligned} {[}\phi _{u}]'(v)=2\int _{\mathbb {R}^{3}}G(x)|x-y|^{2\alpha -N}u(y)v(y)\,\mathrm{d}y,\quad \forall ~u,v\in H^{\alpha } (\mathbb {R}^{3}). \end{aligned}$$

(2.8)

Proof

By Lemma 2.2, the existence and uniqueness parts are proved. By Lemma 2.3 and the Fourier transform for the second equation of problem (1.1), the representation form (2.7) is proved for any $u\in C_{0}^{\infty }(\mathbb {R}^{3})$. Therefore, by density it can be extended for any $u\in H^{\alpha }(\mathbb {R}^{3})$. Then, (2.8) holds. The proof is complete. $\square$

Evidently, from (1.1) and (2.2) and the Hölder inequality, we have

$$\begin{aligned} \Vert \phi _{u}\Vert _{D^{\alpha ,2}}^{2}= & {} K_{\alpha }G(x)\int _{\mathbb {R}^{3}}\phi _{u}u^{2}\,\mathrm{d}x \le K_{\alpha }G_{\infty }\Vert \phi _{u}\Vert _{q}\Vert u\Vert _{2p}^{2}\nonumber \\\le & {} C\Vert \phi _{u}\Vert _{D^{\alpha ,2}}\Vert u\Vert _{2p}^{2}, \end{aligned}$$

(2.9)

where $\frac{1}{p}+\frac{1}{q}=1$, $q=2_{\alpha }^{*}$. By (2.9), we obtain

$$\begin{aligned}&\Vert \phi _{u}\Vert _{D^{\alpha ,2}}\le C\Vert u\Vert _{2p}^{2}\le C\Vert u\Vert ^{2}, \end{aligned}$$

(2.10)

$$\begin{aligned}&\int _{\mathbb {R}^{3}}G(x)\phi _{u}u^{2}\,\mathrm{d}x\le C\Vert u\Vert _{2p}^{4}\le C\Vert u\Vert ^{4}. \end{aligned}$$

(2.11)

Furthermore, by the Hardy–Littlewood–Sobolev inequality [16], we have the following inequality:

$$\begin{aligned} \int _{\mathbb {R}^{3}}\frac{|u(x)u(y)|}{|x-y|^{3-2\alpha }}\,\mathrm{d}x\,\mathrm{d}y\le C\Vert u\Vert _{\frac{6}{3+2\alpha }}\Vert v\Vert _{\frac{6}{3+2\alpha }},\quad \forall ~u,~v\in L^{\frac{6}{3+2\alpha }}(\mathbb {R}^{3}). \end{aligned}$$

(2.12)

Lemma 2.5

Assume that a sequence $\{u_{n}\}\subset E$, $u_{n}\rightharpoonup u$ in E, as $n\rightarrow \infty$. Then,

$$\begin{aligned} \bigg |\int _{\mathbb {R}^{3}}G(x)\big (\phi _{u_{n}}u_{n}-\phi _{u}u\big )(u_{n}-u)\,\mathrm{d}x\bigg |\rightarrow 0,\quad ~\mathrm{as}~n\rightarrow \infty . \end{aligned}$$

(2.13)

Proof

It follows from $u_{n}\rightharpoonup u$ in E that $\{u_{n}\}$ is a bounded sequence, $u_{n}\rightarrow u$ in $L_{loc}^{r}(\mathbb {R}^{3})$ and $u_{n}(x)\rightarrow u(x)$ a.e. on $\mathbb {R}^{3}$. For any $\varepsilon >0$, from the assumption (G), there exists $R_{\varepsilon }>0$ such that $|G(x)|\le \varepsilon$ for $|x|\ge R_{\varepsilon }$. Then,

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}|G(x)(u_{n}-u)|^{\frac{12}{3+2\alpha }}\,\mathrm{d}x&= \int _{R_{\varepsilon }}|G(x)(u_{n}-u)|^{\frac{12}{3+2\alpha }}\,\mathrm{d}x+ \int _{\mathbb {R}^{3}{\setminus } R_{\varepsilon }}|G(x)(u_{n}-u)|^{\frac{12}{3+2\alpha }}\,\mathrm{d}x \\&\le G_{\infty }^{\frac{12}{3+2\alpha }}\int _{R_{\varepsilon }}|u_{n}-u|^{\frac{12}{3+2\alpha }}\,\mathrm{d}x+ \varepsilon ^{\frac{12}{3+2\alpha }}\int _{\mathbb {R}^{3}{\setminus } R_{\varepsilon }}|u_{n}-u)|^{\frac{12}{3+2\alpha }}\,\mathrm{d}x\\&\le o(1)+\varepsilon ^{\frac{12}{3+2\alpha }}\Vert u_{n}-u\Vert _{\frac{12}{3+2\alpha }}^{\frac{12}{3+2\alpha }}\\&\le o(1)+C \varepsilon ^{\frac{12}{3+2\alpha }}. \end{aligned} \end{aligned}$$

Since $\varepsilon >0$ is arbitrary, we have

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\int _{\mathbb {R}^{3}}|G(x)(u_{n}-u)|^{\frac{12}{3+2\alpha }}\,\mathrm{d}x=0. \end{aligned}$$

Then, by the condition (G) and (2.5), we obtain

$$\begin{aligned}&\int _{\mathbb {R}^{3}}|G(x)\phi _{u_{n}}u_{n}(u_{n}-u)|\,\mathrm{d}x\nonumber \\&\quad =\int _{\mathbb {R}^{3}}\int _{\mathbb {R}^{3}}\frac{|G(x)G(y)(u_{n}(x)-u(x))|u_{n}^{2}(y)}{|x-y|^{3-2\alpha }}\,\mathrm{d}x\,\mathrm{d}y\nonumber \\&\quad \le G_{\infty }\int _{\mathbb {R}^{3}}\int _{\mathbb {R}^{3}}\frac{|G(x)(u_{n}(x)-u(x))|u_{n}^{2}(y)}{|x-y|^{3-2\alpha }}\,\mathrm{d}x\,\mathrm{d}y\nonumber \\&\quad \le C G_{\infty }\big (\int _{\mathbb {R}^{3}}|G(x)u_{n}(x)(u_{n}(x)-u(x))|^{\frac{6}{3+2\alpha }}\,\mathrm{d}x\big )^{\frac{3+2\alpha }{6}}\Vert u_{n}\Vert _{\frac{12}{3+2\alpha }}^{2}\nonumber \\&\quad \le C G_{\infty }\big (\int _{\mathbb {R}^{3}}|G(x)(u_{n}(x)-u(x))|^{\frac{12}{3+2\alpha }}\mathrm{d}x\big )^{\frac{3+2\alpha }{12}} \Vert u_{n}\Vert _{\frac{12}{3+2\alpha }}\Vert u_{n}\Vert _{\frac{12}{3+2\alpha }}^{2}\nonumber \\&\quad =o(1). \end{aligned}$$

(2.14)

On the other hand, since $G(x)\phi _{u}u\in L^{2}(\mathbb {R}^{3})$ and $u_{n}\rightharpoonup u$ in $L^{2}(\mathbb {R}^{3})$, then we have

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\int _{\mathbb {R}^{3}}|G(x)\phi _{u}u(u_{n}-u)|\,\mathrm{d}x=0. \end{aligned}$$

(2.15)

Then, it follows from (2.14) and (2.15) that the conclusion holds. The proof is complete. $\square$

Problem (1.1) is the Euler–Lagrange equations corresponding to the functional $J_{\lambda }:H^{\alpha }(\mathbb {R}^{3})\times D^{\alpha }(\mathbb {R}^{3})\rightarrow \mathbb {R}$ defined by

$$\begin{aligned} \begin{aligned} J_{\lambda }(u,\phi )&=\frac{1}{2}\int _{\mathbb {R}^{3}}\bigg (|(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda V(x)u^{2}- \frac{1}{2}|(-\Delta )^{\frac{\alpha }{2}}\phi |^{2} + K_{\alpha } \phi u^{2} \bigg )\,\mathrm{d}x \\&\quad -\,\int _{\mathbb {R}^{3}}F(x, u)\,\mathrm{d}x. \end{aligned} \end{aligned}$$

From (2.11) and $(F_{1})$, it is not difficult to verify that the function $J_{\lambda }$ is well defined and belongs to $C^{1}(H^{\alpha }(\mathbb {R}^{3})\times D^{\alpha }(\mathbb {R}^{3}), \mathbb {R})$ and the partial derivatives in $(u, \phi )$ are given, for $\xi \in H^{\alpha }(\mathbb {R}^{3})$ and $\eta \in D^{\alpha }(\mathbb {R}^{3})$, we have

$$\begin{aligned}&\begin{aligned} \bigg \langle \frac{\partial J_{\lambda }}{\partial u}(u, \phi ), \xi \bigg \rangle&=\int _{\mathbb {R}^{3}} \big ((-\Delta )^{\frac{\alpha }{2}}u(-\Delta )^{\frac{\alpha }{2}}\xi +\lambda V(x)u\xi +K_{\alpha }\phi u\xi \big )\,\mathrm{d}x\\&\quad -\,\int _{\mathbb {R}^{3}}f(x,u)\xi \,\mathrm{d}x, \end{aligned} \\&\bigg \langle \frac{\partial J_{\lambda }}{\partial \phi }(u, \phi ), \eta \bigg \rangle =\frac{1}{2 }\int _{\mathbb {R}^{3}} \big (-(-\Delta )^{\frac{\alpha }{2}}\phi (-\Delta )^{\frac{\alpha }{2}}\eta +K_{\alpha }u^{2}\eta \big )\,\mathrm{d}x. \end{aligned}$$

Thus, we have the following result:

Proposition 2.1

The pair $(u,\phi )$ is a weak solution of problem (1.1) if and only if it is a critical point of $J_{\lambda }$ in $H^{\alpha }(\mathbb {R}^{3})\times D^{\alpha }(\mathbb {R}^{3})$.

Therefore, we can consider the functional $I_{\lambda }:H^{\alpha }(\mathbb {R}^{3})\rightarrow \mathbb {R}$ defined by $I_{\lambda }(u)=J_{\lambda }(u,\phi _{u})$. After multiplying equation $(-\Delta )^{\alpha } \phi _{u}=K_{\alpha }u^{2}$ by $\phi _{u}$ and integration by parts over $\mathbb {R}^{3}$, we have

$$\begin{aligned} \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{\alpha }{2}}\phi _{u}|^{2}dx=K_{\alpha } \int _{\mathbb {R}^{3}}G(x)\phi _{u}u^{2}\,\mathrm{d}x. \end{aligned}$$

(2.16)

Therefore, by (2.16), the reduced functional takes the form

$$\begin{aligned} I_{\lambda }(u)= & {} \frac{1}{2}\int _{\mathbb {R}^{3}}\big (|(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda V(x)u^{2}\big )\,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x\nonumber \\&-\,\int _{\mathbb {R}^{3}}F(x, u)\,\mathrm{d}x. \end{aligned}$$

(2.17)

Evidently, $I_{\lambda }$ is well defined and belongs to $C^{1}(E,\mathbb {R})$ with the derivative given by

$$\begin{aligned} \big \langle I'_{\lambda }(u),v\big \rangle= & {} \int _{\mathbb {R}^{3}}(-\Delta )^{\frac{\alpha }{2}}u(-\Delta )^{\frac{\alpha }{2}}v\,\mathrm{d}x +\lambda \int _{\mathbb {R}^{3}}V(x)uv\,\mathrm{d}x\nonumber \\&+\,K_{\alpha }\int _{\mathbb {R}^{3}}G(x)\phi _{u}uv\,\mathrm{d}x-\int _{\mathbb {R}^{3}}f(x, u)v\,\mathrm{d}x. \end{aligned}$$

(2.18)

It can be proved that $(u,\phi )\in E\times D^{\alpha }(\mathbb {R}^{3})$ is a solution of problem (1.1) if and only if $u\in E$ is a critical point of the functional $I_{\lambda }$ and $\phi =\phi _{u}$.

Lemma 2.6

[10] Let E be a real Banach space and $I\in C^{1}(E, \mathbb {R})$ satisfy the (PS) condition. If I is bounded from below, then $c=\inf \nolimits _{E}I$ is a critical value of I.

Lemma 2.7

Assume that $(V_{1})-(V_{3})$, (G) and $(F_{1})-(F_{2})$ hold. Then, there exists $\Lambda _{0}>0$ such that $I_{\lambda }$ is bounded from below whenever $\lambda \ge \Lambda _{0}$.

Proof

It follows from (2.1), (2.17), $(F_{1})$ and the Hölder inequality that

$$\begin{aligned} I_{\lambda }(u)= & {} \frac{1}{2}\int _{\mathbb {R}^{3}}\big (|(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda \int _{\mathbb {R}^{3}}V(x)u^{2}\big )\,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x\\&-\,\int _{\mathbb {R}^{3}}F(x, u)\,\mathrm{d}x\\\ge & {} \frac{1}{2}\Vert u\Vert _{\lambda }^{2}-\int _{\mathbb {R}^{3}}F(x, u)\,\mathrm{d}x\\\ge & {} \frac{1}{2}\Vert u\Vert ^{2}-\int _{\mathbb {R}^{3}}c_{1}(x)|u|^{\alpha _{1}}\,\mathrm{d}x -\int _{\mathbb {R}^{3}}c_{2}(x)|u|^{\alpha _{2}}\,\mathrm{d}x\\\ge & {} \frac{1}{2}\Vert u\Vert ^{2}-\sum \limits _{i=1}^{2}\tau _{2}^{\alpha _{i}} \Vert c_{i}\Vert _{\frac{2}{2-\alpha _{i}}}\Vert u\Vert ^{\alpha _{i}}, \end{aligned}$$

which implies that $I_{\lambda }(u)\rightarrow +\infty ,~\mathrm{as} ~\Vert u\Vert \rightarrow \infty$, since $\alpha _{1}$, $\alpha _{2}\in (1,2)$. Consequently, there is $\Lambda _{0}=\max \{1,\lambda _{0}\}$ such that for every $\lambda \ge \Lambda _{0}$, $I_{\lambda }$ is bounded from below. The proof is complete. $\square$

3 Proofs of Main Results

Lemma 3.1

Assume that $(V_{1})-(V_{3})$, (G) and $(F_{1})-(F_{2})$ hold, then $I_{\lambda }$ satisfies the (PS) condition.

Proof

Assume that $\{u_{n}\}$ is a (PS) sequence of $I_{\lambda }$ such that $I_{\lambda }(u_{n})$ is bounded and $\Vert I_{\lambda }'(u_{n})\Vert \rightarrow 0, as ~n\rightarrow \infty$. By Lemma 2.7, it is clear that $\{u_{n}\}$ is bounded in $E_{\lambda }$. Then, there exists a constant $C>0$ such that

$$\begin{aligned} \Vert u_{n}\Vert _{r}\le \tau _{r}\Vert u_{n}\Vert _{\lambda }\le C,\quad n\in \mathbb {N},\quad \lambda \ge \Lambda _{0},\quad 2\le r\le 2_{\alpha }^{*}. \end{aligned}$$

(3.1)

Then, there exists $u\in E$ such that

$$\begin{aligned}&u_{n}\rightharpoonup u \quad \mathrm {in} ~E_{\lambda }, \end{aligned}$$

(3.2)

$$\begin{aligned}&u_{n}\rightarrow u\quad \mathrm {in}~L^{r}_{loc}(\mathbb {R}^{3}),\quad r\in [2,2_{\alpha }^{*}), \end{aligned}$$

(3.3)

$$\begin{aligned}&u_{n}\rightarrow u\quad \mathrm {a.e. ~in}\quad \mathbb {R}^{3}. \end{aligned}$$

(3.4)

On the other hand, for any given $\varepsilon >0$, by the condition $(F_{1})$, we can choose $R_{\varepsilon }>0$ such that

$$\begin{aligned} \left( \int _{|x|>R_{\varepsilon }}|c_{i}(x)|^{\frac{2}{2-\alpha _{i}}}\,\mathrm{d}x \right) ^{\frac{2-\alpha _{i}}{2}}<\varepsilon , \quad i=1,2. \end{aligned}$$

(3.5)

It follows from (3.3) that there exists $n_{0}>0$ such that

$$\begin{aligned} \int _{|x|\le R_{\varepsilon }}|u_{n}-u|^{2}\,\mathrm{d}x<\varepsilon ^{2}, \quad \mathrm{for} \quad n\ge n_{0}. \end{aligned}$$

(3.6)

Then, it follows from (3.1), (3.6), $(F_{1})$ and the Hölder inequality, for any $n\ge n_{0}$, one has

$$\begin{aligned}&\int _{|x|\le R_{\varepsilon }}|f(x,u_{n})-f(x,u)\Vert u_{n}-u|\,\mathrm{d}x\nonumber \\&\quad \le \left( \int _{|x|\le R_{\varepsilon }}|f(x,u_{n})-f(x,u)|^{2}\,\mathrm{d}x\right) ^{\frac{1}{2}}\left( \int _{|x|\le R_{\varepsilon }}|u_{n}-u|^{2}\,\mathrm{d}x\right) ^{\frac{1}{2}}\nonumber \\&\quad \le \varepsilon \left[ \int _{|x|\le R_{\varepsilon }}2\left( |f(x,u_{n})|^{2}+|f(x,u)|^{2}\right) \,\mathrm{d}x\right] ^{\frac{1}{2}}\nonumber \\&\quad \le \varepsilon \left[ 4\sum \limits _{i=1}^{2}\alpha _{i}^{2}\int _{|x|\le R_{\varepsilon }}|c_{i}(x)|^{2}\left( |u_{n}|^{2(\alpha _{i}-1)}+|u|^{2(\alpha _{i}-1)}\right) \,\mathrm{d}x\right] ^{\frac{1}{2}}\nonumber \\&\quad \le C\varepsilon \big [\sum \limits _{i=1}^{2}\alpha _{i}^{2}\Vert c_{i}\Vert _{\frac{2}{2-\alpha _{i}}}^{2} \left( \Vert u_{n}\Vert _{2}^{2\left( \alpha _{i}-1\right) }+\Vert u\Vert _{2}^{2\left( \alpha _{i}-1\right) }\big )\right] ^{\frac{1}{2}}\nonumber \\&\quad \le C\varepsilon \left[ \sum \limits _{i=1}^{2}\alpha _{i}^{2}\Vert c_{i}\Vert _{\frac{2}{2-\alpha _{i}}}^{2} \left( C^{2(\alpha _{i}-1)}+\Vert u\Vert _{2}^{2(\alpha _{i}-1)}\right) \right] ^{\frac{1}{2}}. \end{aligned}$$

(3.7)

On the other hand, for $n\in \mathbb {N}$, it follows from $(F_{1})$, (3.1), (3.5) and the Hölder inequality that

$$\begin{aligned}&\int _{|x|> R_{\varepsilon }}|f(x,u_{n})-f(x,u)\Vert u_{n}-u|\,\mathrm{d}x\nonumber \\&\quad \le \sum \limits _{i=1}^{2}\alpha _{i}\int _{|x|> R_{\varepsilon }}|c_{i}(x)|\left( |u_{n}|^{\alpha _{i}-1}+|u|^{\alpha _{i}-1}\right) \left( |u_{n}|+|u|\right) \,\mathrm{d}x\nonumber \\&\quad \le 2\sum \limits _{i=1}^{2}\alpha _{i}\int _{|x|> R_{\varepsilon }} |c_{i}(x)|\left( |u_{n}|^{\alpha _{i}}+|u|^{\alpha _{i}}\right) \,\mathrm{d}x\nonumber \\&\quad \le 2\sum \limits _{i=1}^{2}\alpha _{i}\left( \int _{|x|> R_{\varepsilon }} |c_{i}(x)|^{{\frac{2}{2-\alpha _{i}}}}\,\mathrm{d}x\right) ^{\frac{2-\alpha _{i}}{2}} \big (\Vert u_{n}\Vert _{2}^{\alpha _{i}}+\Vert u\Vert _{2}^{\alpha _{i}}\big )\nonumber \\&\quad \le 2\sum \limits _{i=1}^{2}\alpha _{i}\left( \int _{|x|> R_{\varepsilon }} |c_{i}(x)|^{{\frac{2}{2-\alpha _{i}}}}\,\mathrm{d}x\right) ^{\frac{2-\alpha _{i}}{2}} \left( C^{\alpha _{i}}+\Vert u\Vert _{2}^{\alpha _{i}}\right) \nonumber \\&\quad \le 2\varepsilon \sum \limits _{i=1}^{2}\alpha _{i}\left( C^{\alpha _{i}} +\Vert u\Vert _{2}^{\alpha _{i}}\right) . \end{aligned}$$

(3.8)

Since $\varepsilon$ is arbitrary, combining (3.7) and (3.8), we have

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\int _{\mathbb {R}^{3}}\left( f(x,u_{n}) -f(x,u)\right) \left( u_{n}-u\right) \,\mathrm{d}x=0. \end{aligned}$$

(3.9)

Then, by (2.18), (3.9), Lemma 2.5 and the weak convergence of $\{u_{n}\}$, one has

$$\begin{aligned} \begin{aligned} o_{n}(1)&=\langle I_{\lambda }'(u_{n})-I_{\lambda }'(u), u_{n}-u\rangle \\&=\int _{\mathbb {R}^{3}}|(-\triangle )^{\frac{\alpha }{2}}(u_{n}-u)|^{2}\,\mathrm{d}x +\lambda \int _{\mathbb {R}^{3}}V(x)\left( u_{n}-u\right) ^{2}\,\mathrm{d}x\\&\quad +\,\int _{\mathbb {R}^{3}}G(x)\left( \phi _{u_{n}}u_{n}-\phi _{u}u\right) \left( u_{n}-u\right) \,\mathrm{d}x\\&\quad -\,\int _{\mathbb {R}^{3}}\left( f(x,u_{n})-f(x,u)\right) \left( u_{n}-u\right) \,\mathrm{d}x\\&=\Vert u_{n}-u\Vert _{\lambda }^{2}+o_{n}(1), \end{aligned} \end{aligned}$$

which implies that $u_{n}\rightarrow u$ in $E_{\lambda }$. Then, $I_{\lambda }$ satisfies the (PS) condition. The proof is complete. $\square$

In order to find the multiplicity of nontrivial critical points of $J_{\lambda }$, we will use the “genus” properties, so we recall the following definitions and results (see [12]).

Let E be a Banach space, $c\in \mathbb {R}$ and $I\in C^{1}(E, \mathbb {R})$. Set

$$\begin{aligned} \Sigma= & {} \{A\subset E{\setminus }\{0\}: \mathrm{A~ is~ closed~ in}~ E \mathrm{~and~ symmetric ~with ~respect~ to}~ 0\}, \\ K_{c}= & {} \left\{ u\in E: I_{\lambda }(u)=c, I_{\lambda }'(u)=0\right\} ,~~~~ I_{\lambda }^{c}=\left\{ u\in E:I_{\lambda }(u)\le c\right\} . \end{aligned}$$

Definition 3.1

For $A\in \Sigma$, we say genus of A is n (denoted by $\gamma (A)=n$) if there is an odd map $\varphi \in C(A,\mathbb {R}^{3}{\setminus } \{0\})$ and n is the smallest integer with this property.

Lemma 3.2

Let I be an even $C^{1}$ functional on E and satisfy the (PS) condition. For any $n\in \mathbb {N}$, set

$$\begin{aligned} \Sigma _{n}=\{A\in \Sigma :\gamma (A)\ge n\},\quad c_{n}=\inf \limits _{A\in \Sigma _{n}}\sup \limits _{u\in A}I(u). \end{aligned}$$

• (i) If $\Sigma _{n}\ne \emptyset$ and $c_{n}\in \mathbb {R}$, then $c_{n}$ is a critical value of I.

• (ii) If there exists $r\in \mathbb {N}$ such that $c_{n}=c_{n+1}=\cdots =c_{n+r}=c\in \mathbb {R}$ and $c\ne I(0)$, then $\gamma (K_{c})\ge r+1.$

Proof of Theorem 1.1

By Lemmas 2.7 and  3.1, the conditions of Lemma 2.6 are satisfied. Thus, $c=\inf \nolimits _{E_{\lambda }}I_{\lambda }(u)$ is a critical value of $I_{\lambda }$, that is, there exists a critical point $u^{*}$ such that $I_{\lambda }(u^{*})=c$. Now, we show that $u^{*}\ne 0$. Let $u\in (H_{0}^{\alpha }(\Lambda )\bigcap E_{\lambda }){\setminus }\{0\}$ and $\Vert u\Vert _{\infty }\le 1$, then by (2.11) and $(F_{2})$, we have

$$\begin{aligned} I_{\lambda }(tu)= & {} \frac{t^{2}}{2}\int _{\mathbb {R}^{3}}\left( |(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda \int _{\mathbb {R}^{3}}V(x)u^{2}\right) \,\mathrm{d}x\nonumber \\&+\, \frac{t^{4}}{4}K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x -\int _{\mathbb {R}^{3}}F(x, tu)\,\mathrm{d}x\nonumber \\= & {} \frac{t^{2}}{2}\Vert u\Vert ^{2}_{\lambda }+\frac{t^{4}}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x-\int _{J}F(x, tu)\,\mathrm{d}x\nonumber \\\le & {} \frac{t^{2}}{2}\Vert u\Vert ^{2}_{\lambda }+\frac{t^{4}}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x- a_{2}t^{a_{3}}\int _{J}|u|^{a_{3}}\,\mathrm{d}x, \end{aligned}$$

(3.10)

where $0<t<a_{1}$, $a_{1}$ be given in $(f_{2})$. Since $1<a_{3}<2$, it follows from (3.10) that $I(tu)<0$ for $t>0$ small enough. Therefore, $I_{\lambda }(u^{*})=c<0$, that is, $u^{*}$ is a nontrivial critical point of $I_{\lambda }$, and so $u^{*}$ is a nontrivial solution of problem (1.1). The proof is complete. $\square$

Proof of Theorem 1.2

By Lemmas 2.7 and 3.1, $I_{\lambda }\in C^{1}(E, \mathbb {R})$ is bounded from below and satisfies the (PS)-condition. It follows from (2.17) and $(F_{3})$ that $I_{\lambda }$ is even and $I_{\lambda }(0)=0$. In order to apply Lemma 3.2, we now show that for any $n\in \mathbb {N}$, there exists $\varepsilon >0$ such that

$$\begin{aligned} \gamma (I_{\lambda }^{-\varepsilon })\ge n. \end{aligned}$$

(3.11)

For any $n\in \mathbb {N}$, we take n disjoint open sets $\Lambda _{i}$ such that

$$\begin{aligned} \bigcup \limits _{i=1}^{n}\Lambda _{i}\subset \Lambda . \end{aligned}$$

For $i=1,2,\ldots , n$, let $u_{i}\in (H_{0}^{\alpha }(\Lambda _{i})\bigcap E_{\lambda }){\setminus }\{0\}$, $\Vert u_{i}\Vert _{\infty }\le \infty$ and $\Vert u_{i}\Vert =1$, and

$$\begin{aligned} E_{n}=\text {span}\{u_{1},u_{2},\ldots ,u_{n}\},~~~~S_{n}=\{u\in E_{n}:\Vert u\Vert _{\lambda }=1\}. \end{aligned}$$

Then, for any $u\in E_{n}$, there exist $\mu _{i}\in \mathbb {R}, i=1,2,\ldots ,n$ such that

$$\begin{aligned} u(x)=\sum \limits _{i=1}^{n}\mu _{i}u_{i}(x),\quad x\in \mathbb {R}^{3}. \end{aligned}$$

(3.12)

Then, we get

$$\begin{aligned} \Vert u\Vert _{a_{3}}=\bigg (\int _{\mathbb {R}^{3}}|u|^{a_{3}}\,\mathrm{d}x\bigg )^{\frac{1}{a_{3}}}= \bigg (\sum \limits _{i=1}^{n}|\mu _{i}|^{a_{3}}\int _{J_{i}}|u|^{a_{3}}\,\mathrm{d}x\bigg )^{\frac{1}{a_{3}}}, \end{aligned}$$

(3.13)

and

$$\begin{aligned} \Vert u\Vert ^{2}_{\lambda }= & {} \int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{\alpha }{2}} u|^{2}+ \lambda V(x)|u|^{2})\,\mathrm{d}x\nonumber \\= & {} \sum \limits _{i=1}^{n}\mu _{i}^{2}\int _{J_{i}}(|(-\Delta )^{\frac{\alpha }{2}}u_{i}|^{2}+ \lambda V(x)|u_{i}|^{2})\,\mathrm{d}x\nonumber \\\le & {} \sum \limits _{i=1}^{n}\mu _{i}^{2}\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{\alpha }{2}} u_{i}|^{2}+ \lambda V(x)|u_{i}|^{2})\,\mathrm{d}x\nonumber \\= & {} \sum \limits _{i=1}^{n}\mu _{i}^{2}\Vert u_{i}\Vert ^{2}_{\lambda } =\sum \limits _{i=1}^{n}\mu _{i}^{2}. \end{aligned}$$

(3.14)

Since all norms are equivalent in a finite dimensional normed space, there exists $d_{1}>0$ such that

$$\begin{aligned} d_{1}\Vert u\Vert _{\lambda }\le \Vert u\Vert _{a_{3}},~\text {for any }~u\in E_{n}. \end{aligned}$$

(3.15)

Then, by (2.17), $(F_{2})$, (3.12)–(3.15) and the Sobolev embedding inequality, for $u\in S_{n}$, we have

$$\begin{aligned} I_{\lambda }(tu)= & {} \frac{t^{2}}{2}\int _{\mathbb {R}^{3}}\left( |(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda \int _{\mathbb {R}^{3}}V(x)u^{2}\right) \,\mathrm{d}x\nonumber \\&+\, \frac{t^{4}}{4}K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x-\int _{\mathbb {R}^{3}}F(x, tu)\,\mathrm{d}x\nonumber \\= & {} \frac{t^{2}}{2}\Vert u\Vert ^{2}_{\lambda }+ \frac{t^{4}}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u} u^{2}\,\mathrm{d}x- \sum \limits _{i=1}^{n}\int _{J_{i}}F(x, t\mu _{i}u_{i})\,\mathrm{d}x\nonumber \\\le & {} \frac{t^{2}}{2}\Vert u\Vert ^{2}_{\lambda }+ \frac{Ct^{4}}{4} \Vert u\Vert ^{4}_{\lambda }-a_{2}t^{a_{3}}\sum \limits _{i=1}^{n} |\lambda _{i}|^{a_{3}}\int _{J_{i}}|u_{i}|^{a_{3}}\,\mathrm{d}x\nonumber \\= & {} \frac{t^{2}}{2}\Vert u\Vert ^{2}_{\lambda }+ \frac{Ct^{4}}{4} \Vert u\Vert ^{4}_{\lambda }-a_{2}t^{a_{3}}\Vert u\Vert _{a_{3}}^{a_{3}}\nonumber \\\le & {} \frac{t^{2}}{2}\Vert u\Vert ^{2}_{\lambda }+ \frac{Ct^{4}}{4} \Vert u\Vert ^{4}_{\lambda }-a_{2}(d_{1}t)^{a_{3}}\Vert u\Vert ^{a_{3}}_{\lambda }\nonumber \\= & {} \frac{t^{2}}{2}+\frac{Ct^{4}}{4}-a_{2}(d_{1}t)^{a_{3}}. \end{aligned}$$

(3.16)

Since $0<t\le a_{1}$ and $1<a_{3}<2$, it follows from (3.16) that there exist $\varepsilon >0$ and $\delta >0$ such that

$$\begin{aligned} I_{\lambda }(\delta u)<-\varepsilon ,\quad \mathrm {for~any}\quad u\in S_{n}. \end{aligned}$$

(3.17)

Let

$$\begin{aligned} S_{n}^{\delta }=\{\delta u:u\in S_{n}\},\quad \Omega =\left\{ (\mu _{1},\mu _{2},\ldots ,\mu _{n})\in \mathbb {R}^{3}: \sum \limits _{i=1}^{n}\mu _{i}^{2}<\delta ^{2}\right\} . \end{aligned}$$

It follows from (3.17) that

$$\begin{aligned} I_{\lambda }(u)<-\varepsilon , \quad \mathrm {for~any} ~u\in S_{n}^{\delta }, \end{aligned}$$

which, together with the fact that $I_{\lambda }\in C^{1}(E,\mathbb {R} )$ and is even, implies that

$$\begin{aligned} S_{n}^{\delta }\subset I_{\lambda }^{-\varepsilon }\in \Sigma . \end{aligned}$$

(3.18)

On the other hand, by (3.12) and (3.14), there exists an odd homeomorphism mapping $\phi \in C(S_{n}^{\delta },\partial \Omega )$. By some properties of the genus (see $3^{0}$ of the Propositions of 7.5 and 7.7 in [12]), we have

$$\begin{aligned} \gamma (I_{\lambda }^{-\varepsilon })\ge \gamma (S_{n}^{\delta })=n. \end{aligned}$$

(3.19)

Thus, the proof of (3.11) holds. Set

$$\begin{aligned} c_{n}=\inf \limits _{A\in \Sigma _{n}}\sup \limits _{u\in A}I_{\lambda }(u). \end{aligned}$$

(3.20)

It follows from (3.20) and the fact that $I_{\lambda }$ is bounded from below on $E_{\lambda }$ that $-\infty<c_{n}\le -\varepsilon <0$, that is to say, for any $n\in \mathbb {N}$, $c_{n}$ is a real negative number. By Lemma 3.2, $I_{\lambda }$ has infinitely many nontrivial critical points; therefore, problem (1.1) possesses infinitely many nontrivial solutions. The proof is complete. $\square$

4 Concentration of Solutions

In the following, we study the concentration of solutions for problem (1.1) as $\lambda \rightarrow \infty$. Define

$$\begin{aligned} \tilde{c}=\inf \limits _{u\in H_{0}^{\alpha }(\Omega )}I_{\lambda }(u)\big |_{H_{0}^{\alpha }(\Omega )}, \end{aligned}$$

where $I_{\lambda }(u)\big |_{H_{0}^{\alpha }(\Omega )}$ is a restriction of $I_{\lambda }(u)$ on $H_{0}^{\alpha }(\Omega )$, that is,

$$\begin{aligned} \begin{aligned} I_{\lambda }(u)\big |_{H_{0}^{\alpha }(\Omega )}&=\frac{1}{2}\int _{\Omega }\left( |(-\Delta )^{\frac{\alpha }{2}}u|^{2}+ \lambda V(x)u^{2}\right) \,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\Omega }G(x) \phi _{u} u^{2}\,\mathrm{d}x\\&\quad -\int _{\Omega }F(x, u)\,\mathrm{d}x. \end{aligned} \end{aligned}$$

Similar to the proof of Theorem 1.1, it is easy to prove that $\tilde{c}<0$ can be achieved. Since $H_{0}^{\alpha }(\Omega )\subset E_{\lambda }$ for all $\lambda >0$, we derive

$$\begin{aligned} c\le \tilde{c}<0,~~\text { for all} ~\lambda >\Lambda _{0}. \end{aligned}$$

Proof of Theorem 1.3

We mainly borrow the ideas of Che and Chen [5] and Sun and Wu [23]. For any $\lambda _{n}\rightarrow \infty$, let $u_{n}:=u(\lambda _{n})$ be critical points of $I_{\lambda {n}}$ obtained in Theorem 1.1, then

$$\begin{aligned} I_{\lambda _{n}}(u_{n})\le \tilde{c}<0, \end{aligned}$$

(4.1)

and

$$\begin{aligned} \begin{aligned} I_{\lambda _{n}}(u_{n})&=\frac{1}{2}\int _{\mathbb {R}^{3}}\left( |(-\Delta )^{\frac{\alpha }{2}}u_{n}|^{2}+ \lambda V(x)u_{n}^{2}\right) \,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{u_{n}} u_{n}^{2}\,\mathrm{d}x\\&\quad -\,\int _{\mathbb {R}^{3}}F(x, u_{n})dx\\&\ge \frac{1}{2}\Vert u_{n}\Vert ^{2}_{\lambda _{n}}-\sum _{i=1}^{2}\tau _{2}^{\alpha _{i}}\Vert c_{i}\Vert _{\frac{2}{2-\alpha _{i}}} \Vert u_{n}\Vert _{\lambda _{n}}^{\alpha _{i}}, \end{aligned} \end{aligned}$$

showing that

$$\begin{aligned} \Vert u_{n}\Vert _{\lambda _{n}}\le C, \end{aligned}$$

(4.2)

where C is independent of $\lambda _{n}$. Thus, we may assume that $u_{n}\rightharpoonup \tilde{u}$ in $E_{\lambda }$ and $u_{n}\rightarrow \tilde{u}$ in $L_{loc}^{r}(\mathbb {R}^{3})$ for $r\in [2,2_{\alpha }^{*})$. It follows from Fatou’s lemma that

$$\begin{aligned} \int _{\mathbb {R}^{3}}V(x)|\tilde{u}|^{2}\,\mathrm{d}x\le \liminf \limits _{n\rightarrow \infty } \int _{\mathbb {R}^{3}}V(x)|\tilde{u}_{n}|^{2}\,\mathrm{d}x\le \liminf \limits _{n\rightarrow \infty } \frac{\Vert u_{n}\Vert _{\lambda _{n}}^{2}}{\lambda _{n}}=0, \end{aligned}$$

which implies that $\tilde{u}=0$ a.e. in $\mathbb {R}^{3}{\setminus } V^{-1}(0)$ and $u\in H_{0}^{\alpha }(\Omega )$ by $(V_{3})$. Now for any $\varphi \in C_{0}^{\infty }(\Omega )$, since $\langle I_{\lambda _{n}}'(u_{n}),\varphi \rangle =0$, it is easy to verify that

$$\begin{aligned} \int _{\Omega }(-\Delta )^{\frac{\alpha }{2}} \tilde{u}(-\Delta )^{\frac{\alpha }{2}}\varphi \,\mathrm{d}x +\int _{\Omega }G(x)\phi _{\tilde{u}}\tilde{u}^{2}dx-\int _{\Omega }f(x,\tilde{u})\varphi \,\mathrm{d}x=0, \end{aligned}$$

which implies that $\tilde{u}$ is a weak solution of problem (1.4) by the density of $C_{0}^{\infty }(\Omega )$ in $H_{0}^{\alpha }(\Omega )$.

Next we show that $u_{n}\rightarrow \tilde{u}$ in $L^{r}(\mathbb {R}^{3})$ for $r\in [2,2_{\alpha }^{*})$. Otherwise, by Lions vanishing lemma [29], there exist $\delta >0$, $\rho >0$ and $x_{n}\in \mathbb {R}^{3}$ such that

$$\begin{aligned} \int _{B_{\rho }(x_{n})}|u_{n}-\tilde{u}|^{2}\,\mathrm{d}x\ge \delta . \end{aligned}$$

Since $u_{n}\rightarrow \tilde{u}$ in $L_{loc}^{2}(\mathbb {R}^{3})$, $|x_{n}|\rightarrow \infty ,~~n\rightarrow \infty .$ Therefore, $\text {meas}\{B_{\rho }(x_{n})\cap V_{b}\}\rightarrow 0,~~n\rightarrow \infty .$ Then, by the Hölder inequality, we obtain

$$\begin{aligned} \int _{B_{\rho }(x_{n})}|u_{n}-\tilde{u}|^{2}\,\mathrm{d}x\le (meas\{B_{\rho }(x_{n})\cap V_{b}\})^{\frac{2_{\alpha }^{*}-2}{2_{\alpha }^{*}}}\left( \int _{\mathbb {R}^{3}} |u_{n}-\tilde{u}|^{2_{\alpha }^{*}}\,\mathrm{d}x\right) ^{\frac{2}{2_{\alpha }^{*}}}\rightarrow 0, \end{aligned}$$

as $n\rightarrow \infty$. Therefore,

$$\begin{aligned} \begin{aligned} \Vert u_{n}\Vert _{\lambda _{n}}^{2}&\ge \lambda _{n}b\int _{B_{\rho }(x_{n})\cap \{x\in \mathbb {R}^{3}:V(x)\ge b\}}|u_{n}|^{2}\,\mathrm{d}x\\&=\lambda _{n}b\int _{B_{\rho }(x_{n})\cap \{x\in \mathbb {R}^{3}:V(x)\ge b\}}|u_{n}-\tilde{u}|^{2}\,\mathrm{d}x\\&=\lambda _{n}b\left( \int _{B_{\rho }(x_{n})}|u_{n}-\tilde{u}|^{2}\,\mathrm{d}x- \int _{B_{\rho }(x_{n})\cap V_{b}}|u_{n}-\tilde{u}|^{2}\,\mathrm{d}x\right) \rightarrow \infty , \end{aligned} \end{aligned}$$

as $n\rightarrow \infty$, which is a contradiction with (4.2). Next, we prove that $u_{n}\rightarrow \tilde{u}$ in $H^{\alpha }(\mathbb {R}^{3})$. By virtue of $\langle I_{\lambda _{n}}'(u_{n},u_{n})\rangle =\langle I_{\lambda _{n}}'(u_{n},\tilde{u})\rangle =0$ and the fact that $u_{n}\rightarrow \tilde{u}$ in $L^{r}(\mathbb {R}^{3})$ for $r\in [2,2_{\alpha }^{*})$, we derive

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\Vert u_{n}\Vert _{\lambda _{n}}^{2}= \lim \limits _{n\rightarrow \infty }(u_{n},\tilde{u})_{\lambda _{n}}=\lim \limits _{n\rightarrow \infty }(u_{n},\tilde{u}) =\Vert \tilde{u}\Vert ^{2}. \end{aligned}$$

Observe that $\Vert u_{n}\Vert \le \Vert u_{n}\Vert _{\lambda _{n}}$, therefore

$$\begin{aligned} \limsup \limits _{n\rightarrow \infty }\Vert u_{n}\Vert ^{2}\le \Vert \tilde{u}\Vert ^{2}. \end{aligned}$$

On the other hand, from the weak semi-continuity of norm, we have

$$\begin{aligned} \Vert \tilde{u}\Vert ^{2}\le \limsup \limits _{n\rightarrow \infty }\Vert u_{n}\Vert ^{2}. \end{aligned}$$

Therefore,

$$\begin{aligned} u_{n}\rightarrow \tilde{u}\quad \text {in}~~H^{\alpha }(\mathbb {R}^{3}). \end{aligned}$$

Then, by (4.1), we have

$$\begin{aligned} \frac{1}{2}\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{\alpha }{2}}\tilde{u}|^{2}\,\mathrm{d}x+ \frac{1}{4} K_{\alpha }\int _{\mathbb {R}^{3}}G(x) \phi _{\tilde{u}} \tilde{u}^{2}\,\mathrm{d}x-\int _{\mathbb {R}^{3}}F(x, \tilde{u})\,\mathrm{d}x\le \tilde{c}<0, \end{aligned}$$

which implies that $\tilde{u}\ne 0$. The proof is complete. $\square$