Positive Solutions for Fractional Kirchhoff Type Problem with Steep Potential Well

Abstract

In this paper, we study the following nonlinear fractional Kirchhoff type problem

$$\begin{aligned} \left\{ \begin{array}{ll} \left( a+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) (-\Delta )^{s}u+\lambda V(x)u=|u|^{p-2}u, &{} {\mathrm{in }}\mathbb {R}^3, \\ u\in H^{s}(\mathbb {R}^3), \end{array} \right. \end{aligned}$$

where $s\in (\frac{3}{4}, 1)$, $a, b, \lambda$ are positive parameters, $2<p<4$ and the potential V(x) is a nonnegative continuous function with a potential well $\Omega =int V^{-1}(0)$. By applying truncation technique, the existence and asymptotic behavior of positive solutions are established.

1 Introduction

In this paper, we are concerned with the existence and asymptotic behavior of positive solutions for the following fractional Kirchhoff type problem

$$\begin{aligned} \left\{ \begin{array}{llll} \left( a+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) (-\Delta )^{s}u+\lambda V(x)u=|u|^{p-2}u, &{} \mathrm{in } \mathbb {R}^3, \\ u\in H^{s}(\mathbb {R}^3), \end{array} \right. \end{aligned}$$

(1.1)

where $s\in (\frac{3}{4}, 1)$, $2<p<4$, $a, b, \lambda$ are positive parameters and the potential V(x) satisfies the following conditions:

$(V_{1})$ $V(x)\in C(\mathbb {R}^{3}, \mathbb {R})$ and $V(x)\ge 0$ on $\mathbb {R}^{3}$;

$(V_{2})$ there exists $c>0$ such that $\mathcal {V}_{c}:=\{x\in \mathbb {R}^{3}: V(x)<c\}$ is nonempty and has finite measure;

$(V_{3})$ $\Omega =int V^{-1}(0)$ is a nonempty open set with locally Lipschitz boundary and $\overline{\Omega }=V^{-1}(0)$.

It’s well known that the fractional Laplacian $(-\Delta )^{s} (s\in (0, 1))$ can be defined by

$$\begin{aligned} (-\Delta )^{s}v(x)=C_{3, s}P.V.\int _{\mathbb {R}^{3}}\frac{v(x)-v(y)}{|x-y|^{3+2s}}\mathrm{d}y=C_{3, s}\lim _{\varepsilon \rightarrow 0}\int _{\mathbb {R}^{3}\backslash B_{\varepsilon }(x)}\frac{v(x)-v(y)}{|x-y|^{3+2s}}\mathrm{d}y \end{aligned}$$

for $v\in \mathcal {S}(\mathbb {R}^{3})$, where P.V. denotes a Principal Value, $\mathcal {S}(\mathbb {R}^{3})$ is the Schwartz space of rapidly decaying $\mathcal {C}^{\infty }$ function, $B_{\varepsilon }(x)$ denotes an open ball of radius $\varepsilon$ centered at x and the normalization constant $C_{3, s}=\left( \int _{\mathbb {R}^{3}}\frac{1-\cos (\zeta _{1})}{|\zeta |^{3+2s}}\right) ^{-1}$(see e.g., [8, 22, 27] and reference therein). For $u\in \mathcal {S}(\mathbb {R}^{3})$, the fractional Laplacian $(-\Delta )^{s} (s\in (0, 1))$ can be defined by the Fourier transform $(-\Delta )^{s}u=\mathfrak {F}^{-1}(|\xi |^{2s}\mathfrak {F}u)$, $\mathfrak {F}$ being the usual Fourier transform. The application background of operator $(-\Delta )^{s}$ can be founded in several areas such as fractional quantum mechanics [16, 17], physics and chemistry [20], obstacle problems [23], optimization and finance [2], conformal geometry and minimal surfaces [3] and so on.

When $a=\lambda =1$, $b=0$, $\mathbb {R}^{3}$ is replaced by the more general space $\mathbb {R}^{N}$ and $|u|^{p-2}u$ is replaced by the more general nonlinear term f(x, u), problem (1.1) reduces to a fractional Schrödinger equation

$$\begin{aligned} (-\Delta )^{s}u+V(x)u=f(x, u), \quad { x\in \mathbb {R}^N,} \end{aligned}$$

(1.2)

which has been introduced by Laskin [16] as a result of expanding the Feynman path integral. In the past few years, under different assumptions on V and f, the existence and concentration of solutions for equation (1.2) are established by variational methods. See, for instance, [4, 10, 11, 24] and the references therein.

In the case $s=1$ and $|u|^{p-2}u$ is replaced by the more general nonlinear term f(x, u), problem (1.1) reduces to the following Kirchhoff equation

$$\begin{aligned} \left\{ \begin{array}{ll} -\left( a+b\int _{\mathbb {R}^{3}}|\nabla u|^{2}\mathrm{d}x\right) \Delta u+\lambda V(x)u=f(x, u), &{} \mathrm{in }\mathbb {R}^3, \\ u\in H^{1}(\mathbb {R}^3), \end{array} \right. \end{aligned}$$

(1.3)

where $V: \mathbb {R}^{3}\rightarrow \mathbb {R}$ is a potential function, $a, b, \lambda >0$ are parameters and $f\in C(\mathbb {R}^{3}, \mathbb {R})$. This kind of problem has been widely studied by many scholars in recent years. In [14], He and Zou proved the existence of positive ground state solutions for (1.3) by using the Mountain Pass Theorem and the method of Nehari manifold when $f(x, u)=|u|^{p-2}u$ $(4<p<6)$ and $\lambda =1$. Wang [26] used a method similar to that in reference [14], the existence of positive ground state solutions for (1.3) is established with $f(x, u)=\lambda f(u)+|u|^{4}u$ and $\lambda =1$. In [18], Li and Ye obtained the existence of positive ground state solutions for (1.3) when $f(x, u)=|u|^{p-2}u$ $(3<p<5)$ satisfies the Ambrosetti–Rabinowitz type condition and $\lambda =1$. Sun and Wu [25] proved the existence and the nonexistence of nontrivial solutions for (1.3) by using variational methods when the V satisfies the assumptions $(V_{1})-(V_{3})$ and the f(x, s) is asymptotically k-linear $(k=1, 3, 4)$ with respect to s at infinity. In [5], Du obtained the existence and asymptotic behavior of ground state solutions for (1.3) when $f(x, u)=|u|^{p-2}u$ with $4<p<6$. For more Kirchhoff type problem, please see [6, 13] and the references therein for more details.

For the fractional Kirchhoff type problem, it seems that in the first study by Fiscella and Valdinoci [12], they considered the following fractional Kirchhoff type problem

$$\begin{aligned} \left\{ \begin{array}{ll} M\left( \int _{\mathbb {R}^{N}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) (-\Delta )^{s}u=\lambda f(x, u)+|u|^{2_{s}^{*}-2}u, &{} {x\in \Omega ,} \\ u=0,&{}{x\in \mathbb {R}^{N}\backslash \Omega ,} \end{array} \right. \end{aligned}$$

(1.4)

where $\Omega$ be bounded regular domains of $\mathbb {R}^{N}$. Under different assumptions on M and f, authors proved the existence of nonnegative solutions. In [21], Pucci and Saldi obtained the existence and multiplicity of nontrivial solutions of (1.4) via variational methods. Recently, there are few papers on the fractional Kirchhoff type problem. For instance, Ambrosio and Isernia [1] considered the following fractional Kirchhoff type equation

$$\begin{aligned} \left( a+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) (-\Delta )^{s}u=f(u), \quad \mathrm{in } \, \mathbb {R}^3, \end{aligned}$$

where f is an odd subcritical nonlinearity satisfying the Berestyki–Lions conditions. By using variational methods, authors proved the existence of multiple radial solutions when the parameter b small.

In [15], He and Zou are concerned with the following nonlinear fractional Kirchhoff equation

$$\begin{aligned} \left( \varepsilon ^{2s}a+\varepsilon ^{4s-3}b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) (-\Delta )^{s}u+V(x)u=f(u), \quad \mathrm{in } \, \mathbb {R}^3, \end{aligned}$$

where $\varepsilon >0$ is a positive parameter, $s\in (\frac{3}{4}, 1)$, a, b are positive constants and V is a positive potential such that $\inf _{\partial \Lambda }V>\inf _{\Lambda }V$ for some open bounded subset $\Lambda \subset \mathbb {R}^{3}$. They established the existence of positive solutions by using the Ljusternik–Schnirelmann theory.

To our best knowledge, there is no result for Equation (1.1) at $2<p<4$. The main difficulty is that the nonlinear term $u\mapsto f(u):=|u|^{p-2}u$ with $2<p<4$ does not satisfy the Ambrosetti–Rabinowitz condition, which would make it very difficult to prove the boundedness of Palais–Smale sequence or Cerami sequence. Moreover, we note that the function $\frac{f(s)}{s}$ is not increasing on $(-\infty , 0)$ and $(0, +\infty )$; hence, we can’t apply Nehari manifold and fibering methods and so on.

Inspired by [19], in this paper, we consider the existence result of solutions for Kirchhoff type problem (1.1) with $2<p<4$. Now, before we present our main results, we recall some facts that $H^{s}(\mathbb {R}^{3})$ denotes the fractional Sobolev space with norm

$$\begin{aligned} \Vert u\Vert ^{2}_{H^{s}}:=\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+u^{2})\mathrm{d}x \end{aligned}$$

and

$$\begin{aligned} D^{s,2}(\mathbb {R}^{3}):=\{u\in L^{2^{*}_{s}}(\mathbb {R}^{3}):(-\Delta )^{\frac{s}{2}}u\in L^{2}(\mathbb {R}^{3})\} \end{aligned}$$

denotes the homogeneous fractional Sobolev space with the norm

$$\begin{aligned} \Vert u\Vert ^{2}_{D^{s,2}}:=\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x. \end{aligned}$$

It is well known that $H^{s}(\mathbb {R}^{3})$ is continuously embedded into $L^{p}(\mathbb {R}^{3})$ for $2\le p\le 2^{*}_{s}(2^{*}_{s}=\frac{6}{3-2s})$, and for any $s\in (0,1)$, there exists a best constant $S_{s}>0$ such that

$$\begin{aligned} S_{s}=\inf _{u\in D^{s, 2}\backslash \{0\}}\frac{\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x}{(\int _{\mathbb {R}^{3}}|u(x)|^{2^{*}_{s}}\mathrm{d}x)^{\frac{2}{2^{*}_{s}}}}. \end{aligned}$$

For simplicity, we assume $a=1$. Finally, we will consider the following nonlinear fractional Kirchhoff equation

$$\begin{aligned} \left\{ \begin{array}{ll} \left( 1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}|^{2}\mathrm{d}x\right) (-\Delta )^{s}u+\lambda V(x)u=|u|^{p-2}u, &{} \mathrm{in } \, \mathbb {R}^3, \\ u\in H^{s}(\mathbb {R}^3). \end{array} \right. \end{aligned}$$

(1.5)

In order to find weak solutions to (1.5), we look for critical points of the functional $J_{b, \lambda }(u): E_{\lambda }\rightarrow \mathbb {R}$ associated with (1.5) which is defined by

$$\begin{aligned} J_{b, \lambda }(u)= & {} \frac{1}{2}\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+\lambda V(x)u^{2}) \mathrm{d}x\\&+\frac{b}{4}\left( \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x \right) ^{2}-\frac{1}{p}\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x, \end{aligned}$$

where

$$\begin{aligned} E_{\lambda }=\left\{ u\in H^{s}(\mathbb {R}^3): \int _{\mathbb {R}^{3}}V(x)u^{2}\mathrm{d}x<\infty \right\} \end{aligned}$$

with the norm

$$\begin{aligned} \Vert u\Vert _{\lambda }^{2}=\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+\lambda V(x)u^{2})\mathrm{d}x. \end{aligned}$$

Moreover, $u^{+}=\max \{u, 0\}$. Our first main result can be stated as follows.

Theorem 1.1

Suppose that $(V_{1})-(V_{3})$ hold with $2<p<4$. Then, there exist $\hat{b}>0$ and $\tilde{\lambda }>1$ such that for each $b\in (0, \hat{b})$ and $\lambda \in (\tilde{\lambda }, \infty )$, problem (1.5) has at least a positive solution $u_{b, \lambda }\in E_{\lambda }$. Moreover, there exist constants $\tau , T>0$ (independent of b, $\lambda$ and s) such that

$$\begin{aligned} \tau \le \Vert u_{b, \lambda }\Vert _{\lambda }\le T. \end{aligned}$$

(1.6)

Remark 1.2

Similar to the argument of Lemma 3.1, it is clear that functional $J_{b, \lambda }$ verifies the assumptions of the Mountain Pass Theorem when $b>0$ small, then we will get a Cerami sequence of functional $J_{b, \lambda }$ when $b>0$ small, but we can’t prove the boundedness of the Cerami sequence. In order to prove Theorem 1.1, firstly, for any $T>0$, we denote the truncated functional $J_{b, \lambda }^{T}: E_{\lambda }\rightarrow \mathbb {R}$:

$$\begin{aligned} \begin{aligned} J_{b, \lambda }^{T}(u)&=\frac{1}{2}\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+\lambda V(x)u^{2})\mathrm{d}x\\&\qquad +\,\frac{b}{4}\eta \left( \frac{\Vert u\Vert _{\lambda }^{2}}{T^{2}}\right) \left( \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) ^{2}-\frac{1}{p}\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x, \end{aligned} \end{aligned}$$

where $\eta$ is a smooth cut-off function such that

$$\begin{aligned} \eta \left( \frac{\Vert u\Vert _{\lambda }^{2}}{T^{2}}\right) = {\left\{ \begin{array}{ll} 1, &{} \Vert u\Vert _{\lambda }\le T,\\ 0, &{}\Vert u\Vert _{\lambda }>\sqrt{2}T. \end{array}\right. } \end{aligned}$$

By applying the Mountain Pass Theorem, we can obtain a Cerami sequence $\{u_{n}\}$ of $J_{b, \lambda }^{T}$ at the mountainpass level $c_{b, \lambda }^{T}$ when $b>0$ small.

Secondly, we will show that $c_{b, \lambda }^{T}$ has an upper bounded, after passing to a subsequence, such that $\Vert u_{n}\Vert _{\lambda }\le T$ for all n when $b>0$ small; then, we get that $\{u_{n}\}$ is a bounded Cerami sequence of $J_{b, \lambda }^{T}$ at the mountainpass level $c_{b, \lambda }^{T}$, that is,

$$\begin{aligned} \sup _{n\in \mathbb {N}}\Vert u_{n}\Vert _{\lambda }\le T, \quad J_{b, \lambda }(u_{n})\rightarrow c_{b, \lambda }^{T} \quad and \quad (1+\Vert u_{n}\Vert _{\lambda })\Vert J'_{b, \lambda }(u_{n})\Vert _{E'_{\lambda }}\rightarrow 0, \end{aligned}$$

where $E'_{\lambda }$ is the dual space of $E_{\lambda }$.

Finally, we will prove that $u_{n}\rightarrow u_{b, \lambda }$ in $E_{\lambda }$ when $\lambda >0$ large; therefore, $u_{b, \lambda }$ is a solution of problem (1.5).

Next, we are concerned with the decay of the positive solutions at infinity. Clearly, it is possible that $\liminf _{|x|\rightarrow \infty }V(x)=0$ since $(V_{1})-(V_{3})$; hence, we need to replace $(V_{2})$ by the following condition:

$(V'_{2})$ there exists $c>0$ such that $\mathcal {V}_{c}:=\{x\in \mathbb {R}^{3}: V(x)<c\}$ is nonempty and bounded.

Clearly, $(V'_{2})$ is stronger than $(V_{2})$. Then, we have

Theorem 1.3

Suppose that $(V_{1})$, $(V'_{2})$ and $(V_{3})$ hold with $2<p<4$. Suppose in addition that $V(x)\in L^{\infty }(\mathbb {R}^{3})$. Let $u_{b, \lambda }$ be the positive solution of (1.5) for each $b\in (0, \hat{b})$ and $\lambda \in (\tilde{\lambda }, \infty )$ satisfying (1.6). Then, there exists $\tilde{\Lambda }>\tilde{\lambda }$ such that for each $b\in (0, \hat{b})$ and $\lambda \in (\tilde{\Lambda }, \infty )$, we have

$$\begin{aligned} 0<u_{b, \lambda }(x)<\frac{C}{1+|x|^{3+2s}} \quad \mathrm{for} \quad |x|>R \end{aligned}$$

(1.7)

where $C, R>0$ independent of b and $\lambda$.

Remark 1.4

As far as we know, Theorem 1.3 is a new result for fractional Kirchhoff type problem with steep potential well. Moreover, it is noteworthy that the potential function V(x) satisfying condition $(V_{1})$, $(V'_{2})$ and $(V_{3})$ may be bounded or unbounded. For example, the bounded potential function:

$$\begin{aligned} V(x)= {\left\{ \begin{array}{ll} 0, &{} |x|\le 1,\\ (|x|-1)^{2}, &{}1<|x|\le 2,\\ 1,&{}|x|>2, \end{array}\right. } \end{aligned}$$

and the unbounded potential function:

$$\begin{aligned} V(x)= {\left\{ \begin{array}{ll} 0, &{} |x|\le 1,\\ (|x|-1)^{2}, &{}|x|>1. \end{array}\right. } \end{aligned}$$

However, here Theorem 1.3 only obtains that the decay rate of positive solution $u_{b, \lambda }$ at infinity when V(x) is bounded, we also are interesting to know the decay rate of positive solution $u_{b, \lambda }$ at infinity when V(x) is unbounded, and however, we cannot solve this question now.

Finally, we give the asymptotic behavior of the positive solutions as $b\rightarrow 0$ and $\lambda \rightarrow \infty$.

Theorem 1.5

Let $u_{b, \lambda }$ be the positive solution of (1.5) obtained by Theorem 1.1. Then, let $b\in (0, \hat{b})$ be fixed, $u_{b, \lambda }\rightarrow u_{b}$ in $H^{s}(\mathbb {R}^3)$ as $\lambda \rightarrow \infty$ up to a subsequence, where $u_{b}\in H_{0}^{s}(\Omega )$ is a solution of

$$\begin{aligned} \left\{ \begin{array}{ll} \left( 1+b\int _{\Omega }|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) (-\Delta )^{s}u=|u|^{p-2}u, &{} \mathrm{in} \, \Omega , \\ u=0,&{} \mathrm{on } \, \partial \Omega . \end{array} \right. \end{aligned}$$

(1.8)

Theorem 1.6

Let $u_{b, \lambda }$ be the positive solution of (1.5) obtained by Theorem 1.1. Then, let $\lambda \in (\tilde{\lambda }, \infty )$ be fixed, $u_{b, \lambda }\rightarrow u_{\lambda }$ in $E_{\lambda }$ as $b\rightarrow 0$ up to a subsequence, where $u_{\lambda }\in E_{\lambda }$ is a positive solution of

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s}u+\lambda V(x)u=|u|^{p-2}u, &{} \mathrm{in} \, \mathbb {R}^{3}, \\ u\in H^{s}(\mathbb {R}^3). \end{array} \right. \end{aligned}$$

(1.9)

Theorem 1.7

Let $u_{b, \lambda }$ be the positive solution of (1.5) obtained by Theorem 1.1. Then, $u_{b, \lambda }\rightarrow u_{0}$ in $H^{s}(\mathbb {R}^3)$ as $b\rightarrow 0$ and $\lambda \rightarrow \infty$ up to a subsequence, where $u_{0}\in H_{0}^{s}(\Omega )$ is a positive solution of

$$\begin{aligned} \left\{ \begin{array}{ll} (-\Delta )^{s}u=|u|^{p-2}u, &{} \mathrm{in} \, \Omega , \\ u=0,&{} \mathrm{on} \, \partial \Omega . \end{array} \right. \end{aligned}$$

(1.10)

In the sequel, we use the following notations:

• C denotes a universal positive constant.

• $|u|_{s}:=(\int _{\mathbb {R}^{3}}|u|^{s}\mathrm{d}x)^{\frac{1}{s}}$, $1\le s\le \infty$.

• For $\rho >0$ and $z\in \mathbb {R}^{3}$, $B_{\rho }(z)$ denotes the ball of radius $\rho$ centered at z.

• |M| is the Lebesgue measure of the set M.

• $X'$ denotes the dual space of X.

The paper is organized as follows. In Sect. 2, we set up the variational framework and present some preliminaries results. In Sect. 3, we prove Theorem 1.1. In Sect. 4, we prove Theorem 1.3. Section 5 is devoted to proving Theorems 1.5, 1.6 and 1.7.

2 Preliminaries

Let

$$\begin{aligned} E=\left\{ u\in H^{s}(\mathbb {R}^3): \int _{\mathbb {R}^{3}}V(x)u^{2}\mathrm{d}x<\infty \right\} \end{aligned}$$

with the inner product

$$\begin{aligned} \langle u, v\rangle =\int _{\mathbb {R}^{3}}((-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v+V(x)uv)\mathrm{d}x \end{aligned}$$

with the norm

$$\begin{aligned} \Vert u\Vert ^{2}=\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+V(x)u^{2})\mathrm{d}x. \end{aligned}$$

It is clear that $E\hookrightarrow H^{s}(\mathbb {R}^3)$. In fact, by virtue of $(V_{1})-(V_{2})$, Hölder’s inequality and Sobolev inequality, it is easy to deduce that

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+u^{2})\mathrm{d}x \le&\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x +|\mathcal {V}_{c}|^{\frac{2_{s}^{*}-2}{2_{s}^{*}}} \left( \int _{\mathcal {V}_{c}}|u|^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{2}{2_{s}^{*}}}\\&+\frac{1}{c}\int _{\mathbb {R}^{3}\backslash \mathcal {V}_{c}}V(x)u^{2}\mathrm{d}x\\ \le&\max \{1+|\mathcal {V}_{c}|^{\frac{2_{s}^{*}-2}{2_{s}^{*}}}S_{s}^{-1}, c^{-1}\}\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+V(x)u^{2})\mathrm{d}x. \end{aligned} \end{aligned}$$

Now, we set

$$\begin{aligned} E_{\lambda }=(E, \Vert u\Vert _{\lambda }), \end{aligned}$$

where

$$\begin{aligned} \Vert u\Vert _{\lambda }^{2}=\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+\lambda V(x)u^{2})\mathrm{d}x. \end{aligned}$$

Thus, there exists $L_{s}>0$ (independent of $\lambda \ge 1$) such that

$$\begin{aligned} |u|_{s}\le L_{s}\Vert u\Vert \le L_{s}\Vert u\Vert _{\lambda } \quad for \quad s\in [2, 2_{s}^{*}]. \end{aligned}$$

(2.1)

Clearly, the functional $J_{b, \lambda }(u): E_{\lambda }\rightarrow \mathbb {R}$ is given by

$$\begin{aligned}&J_{b, \lambda }(u)=\frac{1}{2}\int _{\mathbb {R}^{3}}(|(-\Delta )^{\frac{s}{2}}u|^{2}+\lambda V(x)u^{2})\mathrm{d}x\\&+\frac{b}{4}\left( \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) ^{2}-\frac{1}{p}\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x, \end{aligned}$$

it is easy to check that $J_{b, \lambda }$ is well defined, $J_{b, \lambda }\in C^{1}(E_{\lambda }, \mathbb {R})$ and its differential is given by

$$\begin{aligned} \langle J'_{b, \lambda }(u), v\rangle= & {} (1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x) \int _{\mathbb {R}^{3}}(-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v +\int _{\mathbb {R}^{3}}\lambda V(x)uv\mathrm{d}x\\&-\int _{\mathbb {R}^{3}}|u^{+}|^{p-2}u^{+}v\mathrm{d}x, \end{aligned}$$

for all $u, v\in E_{\lambda }$.

Now, let we recall a stronger version of the Mountain Pass Theorem.

Lemma 2.1

[9] Let X be a real Banach space with its dual space $X'$, and suppose that $J\in C^{1}(X, \mathbb {R})$ satisfies

$$\begin{aligned} \max \{J(0), J(e)\}\le \mu <\eta \le \inf _{\Vert u\Vert _{X}=\rho }J(u) \end{aligned}$$

for some $\mu <\eta$, $\rho >0$ and $e\in X$ with $\Vert e\Vert _{X}>\rho$. Let $c\ge \eta$ be characterized by

$$\begin{aligned} c=\inf _{\gamma \in \Gamma }\max _{t\in [0, 1]}J(\gamma (t)), \end{aligned}$$

where $\Gamma =\{\gamma \in C([0, 1], X): \gamma (0)=0, \gamma (1)=e\}$. Then, there exists a sequence $\{u_{n}\}\subset X$ such that

$$\begin{aligned} J(u_{n})\rightarrow c\ge \eta \quad and \quad (1+\Vert u_{n}\Vert _{X})\Vert J'(u_{n})\Vert _{X'}\rightarrow 0, \end{aligned}$$

as $n\rightarrow \infty$.

Lemma 2.2

Suppose that $(V_{1})-(V_{2})$ hold with $2<p<4$. Then, every nontrivial critical point of $u_{b, \lambda }$ is a positive solution of problem (1.5).

Proof

Let $u\in E_{\lambda }$ be a nontrivial critical point of $J_{b, \lambda }$, then we obtain that

$$\begin{aligned}&\left( 1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) \int _{\mathbb {R}^{3}}(-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v\mathrm{d}x +\int _{\mathbb {R}^{3}}\lambda V(x)uv\mathrm{d}x\nonumber \\&\qquad -\int _{\mathbb {R}^{3}}|u^{+}|^{p-2}u^{+}v\mathrm{d}x=0, \end{aligned}$$

(2.2)

for every $v\in E_{\lambda }$. Taking $v=u^{-}=\max \{-u, 0\}$, we have

$$\begin{aligned}&\left( 1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) \int \int _{\mathbb {R}^{6}}\frac{(u(x)-u(y))(u^{-}(x)-u^{-}(y))}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\\&\qquad -\int _{\mathbb {R}^{3}}\lambda V(x)|u^{-}|^{2}\mathrm{d}x=0, \end{aligned}$$

which implies that

$$\begin{aligned} \int \int _{\mathbb {R}^{6}}\frac{(u(x)-u(y))(u^{-}(x)-u^{-}(y))}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\ge 0. \end{aligned}$$

However, by direct computation, we deduce that

$$\begin{aligned} \begin{aligned}&\int \int _{\mathbb {R}^{6}}\frac{(u(x)-u(y))(u^{-}(x)-u^{-}(y))}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\\&\quad =\int _{\{u(x)\ge 0\}\times \{u(y)<0\}}\frac{(u(x)-u(y))u(y)}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\\&\quad \qquad +\int _{\{u(x)<0\}\times \{u(y)\ge 0\}}\frac{(u(y)-u(x))u(x)}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\\&\qquad -\int _{\{u(x)<0\}\times \{u(y)<0\}}\frac{(u(x)-u(y))^{2}}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\le 0. \end{aligned} \end{aligned}$$

Then, we get that

$$\begin{aligned} \int \int _{\mathbb {R}^{6}}\frac{(u(x)-u(y))(u^{-}(x)-u^{-}(y))}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y=0, \end{aligned}$$

which leads to $u^{-}=0$, so, $u\ge 0$ and $u\not \equiv 0$. Next, we show that $u>0$. Assume by contradiction that there exists $x_{0}\in \mathbb {R}^{3}$ such that $u(x_{0})=0$, then since $u\ge 0$ and $u\not \equiv 0$, we can see that

$$\begin{aligned} (-\Delta )^{s}u(x_{0})=-\frac{1}{2}C(3, s)\int _{\mathbb {R}^{3}}\frac{u(x_{0}+y)+u(x_{0}-y)}{|x_{0}-y|^{3+2s}}\mathrm{d}y<0. \end{aligned}$$

However, it is easy to see that

$$\begin{aligned} (-\Delta )^{s}u(x_{0})=\frac{1}{1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u(x_{0})|^{2}\mathrm{d}x}(-\lambda V(x_{0})u(x_{0})+(u(x_{0}))^{p-1}u(x_{0}))=0, \end{aligned}$$

which gives a contradiction. Hence, $u>0$ for all $x\in \mathbb {R}^{3}$ and this ends the proof of Lemma. $\square$

Lemma 2.3

[7] Assume that $\{u_{n}\}$ is bounded in $H^{s}(\mathbb {R}^{N})$, and it satisfies

$$\begin{aligned} \lim _{n\rightarrow \infty }\sup _{y\in \mathbb {R}^{N}}\int _{B_{\rho }(y)}|u_{n}(x)|^{2}\mathrm{d}x=0, \end{aligned}$$

where $\rho >0$. Then, $u_{n}\rightarrow 0$ in $L^{r}(\mathbb {R}^{N})$ for $2<r<2_{s}^{*}$.

3 Proof of Theorem 1.1

In this section, we give the proof of Theorem 1.1. As mentioned in introduction, we define a cut-off function $\eta \in C^{1}([0, \infty ), \mathbb {R})$ satisfying $0\le \eta \le 1$, $\eta (t)=1$ if $0\le t\le 1$, $\eta (t)=0$ if $t\ge 2$, $\max _{t>0}|\eta '(t)|\le 2$ and $\eta '(t)\le 0$ for each $t>0$.

Now, for $T>0$, we consider the truncated functional $J_{b, \lambda }^{T}: E_{\lambda }\rightarrow \mathbb {R}$:

$$\begin{aligned} J_{b, \lambda }^{T}(u)=\frac{1}{2}\Vert u\Vert _{\lambda }^{2}+\frac{b}{4}\eta \left( \frac{\Vert u\Vert _{\lambda }^{2}}{T^{2}}\right) \left( \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) ^{2}-\frac{1}{p}\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x, \end{aligned}$$

(3.1)

it is easy to check that $J_{b, \lambda }^{T}$ is well defined, $J_{b, \lambda }^{T}\in C^{1}(E_{\lambda }, \mathbb {R})$ and its differential is given by

$$\begin{aligned} \begin{aligned} \langle (J_{b, \lambda }^{T})'(u), v\rangle =&\int _{\mathbb {R}^{3}}((-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v+\lambda V(x)uv)\mathrm{d}x\\&+b\eta \left( \frac{\Vert u\Vert _{\lambda }^{2}}{T^{2}}\right) \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\int _{\mathbb {R}^{3}}(-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v\mathrm{d}x\mathrm{d}y\\&+\frac{b}{2T^{2}}\eta '\left( \frac{\Vert u\Vert _{\lambda }^{2}}{T^{2}}\right) \left( \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\right) ^{2} \int _{\mathbb {R}^{3}}((-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v\\&+\lambda V(x)uv)\mathrm{d}x\\&-\int _{\mathbb {R}^{3}}|u^{+}|^{p-2}u^{+}v\mathrm{d}x, \end{aligned} \end{aligned}$$

(3.2)

for all $u, v\in E_{\lambda }$. Clearly, if a Cerami sequence $\{u_{n}\}$ of $J_{b, \lambda }^{T}$ satisfying $\Vert u_{n}\Vert _{\lambda }\le T$, then $\{u_{n}\}$ is also a Cerami sequence of $J_{b, \lambda }$ satisfying $\Vert u_{n}\Vert _{\lambda }\le T$.

Now we show that the functional $J_{b, \lambda }^{T}$ possesses a Mountain Pass geometry.

Lemma 3.1

Suppose that $2<p<4$ and $(V_{1})-(V_{3})$ hold. Then, the functional $J_{b, \lambda }^{T}$ satisfies the following conditions:

(i) for each $T, b>0$ and $\lambda \ge 1$, there exists $\alpha , \rho >0$ (independent of T, b and $\lambda$) such that $J_{b, \lambda }^{T}(u)\ge \alpha$ for all $u\in E_{\lambda }$ with $\Vert u\Vert _{\lambda }=\rho$;

(ii) there exists $\bar{b}>0$ such that for each $T, \lambda >0$ and $b\in (0, \bar{b})$, we have $J_{b, \lambda }^{T}(e_{0})<0$ for some $e_{0}\in C_{0}^{\infty }(\Omega )$ with $\Vert e_{0}\Vert _{\lambda }>\rho$.

Proof

(i) For $u\in E_{\lambda }$, by using (2.1), we have

$$\begin{aligned} J_{b, \lambda }^{T}(u)\ge \frac{1}{2}\Vert u\Vert _{\lambda }^{2}-\frac{1}{p}L_{p}^{p}\Vert u\Vert _{\lambda }^{p}, \end{aligned}$$

where $L_{p}>0$ is independent of T, b and $\lambda$. Since $p>2$, the conclusion (i) follows by choosing $\rho >0$ sufficiently small.

(ii) Define the functional $J_{\lambda }: E_{\lambda }\rightarrow \mathbb {R}$ by

$$\begin{aligned} J_{\lambda }(u)=\frac{1}{2}\Vert u\Vert _{\lambda }^{2}-\frac{1}{p}\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x. \end{aligned}$$

Since $2<p<4$, then (2.1) shows that $J_{\lambda }$ is well defined. Let $e\in C_{0}^{\infty }(\Omega )$ be a positive smooth function, it is easy to see that

$$\begin{aligned} J_{\lambda }(te)=\frac{t^{2}}{2}\int _{\Omega }|(-\Delta )^{\frac{s}{2}}e|^{2}\mathrm{d}x-\frac{t^{p}}{p}\int _{\Omega }|e|^{p}\mathrm{d}x. \end{aligned}$$

Since $p>2$, we have $J_{\lambda }(te)\rightarrow -\infty$ as $t\rightarrow \infty$. Hence, there exists $e_{0}\in C_{0}^{\infty }(\Omega )$ with $\Vert e_{0}\Vert _{\lambda }>\rho$ such that $J_{\lambda }(e_{0})\le -1$. Note that

$$\begin{aligned} J_{b, \lambda }^{T}(e_{0})=J_{\lambda }(e_{0})+\frac{b}{4}\eta \left( \frac{\Vert e_{0}\Vert _{\lambda }^{2}}{T^{2}}\right) \left( \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}e_{0}|^{2}\mathrm{d}x\right) ^{2}\le -1+\frac{b}{4}|(-\Delta )^{\frac{s}{2}}e_{0}|_{2}^{4}. \end{aligned}$$

Therefore, there exists $\bar{b}>0$ such that for each $T, \lambda >0$ and $b\in (0, \bar{b})$, we have $J_{b, \lambda }^{T}(e_{0})<0$ for some $e_{0}\in C_{0}^{\infty }(\Omega )$ with $\Vert e_{0}\Vert _{\lambda }>\rho$. $\square$

In view of Lemma 2.1 and Lemma 3.1, we can see that for each $T>0$, $\lambda \ge 1$ and $b\in (0, \bar{b})$, there exists a Cerami sequence $\{u_{n}\}\subset E_{\lambda }$ such that

$$\begin{aligned} J_{b, \lambda }^{T}(u_{n})\rightarrow c_{b, \lambda }^{T} \quad and \quad (1+\Vert u_{n}\Vert _{\lambda })\Vert (J_{b, \lambda }^{T})'(u_{n})\Vert _{E'_{\lambda }}\rightarrow 0, \end{aligned}$$

(3.3)

where

$$\begin{aligned} c_{b, \lambda }^{T}=\inf _{\gamma \in \Gamma }\max _{t\in [0, 1]}J_{b, \lambda }^{T}(\gamma (t)), \end{aligned}$$

where $\Gamma =\{\gamma \in C([0, 1], E_{\lambda }): \gamma (0)=0, \gamma (1)=e\}$.

Lemma 3.2

Suppose that $2<p<4$ and $(V_{1})-(V_{3})$ hold. Then, for each $T>0$, $\lambda \ge 1$ and $b\in (0, \bar{b})$, there exists $M>0$ (independent of T, b and $\lambda$) such that $c_{b, \lambda }^{T}\le M$.

Proof

Since $e_{0}\in C_{0}^{\infty }(\Omega )$, then we have

$$\begin{aligned} J_{b, \lambda }^{T}(te_{0})\le \frac{t^{2}}{2}\int _{\Omega }|(-\Delta )^{\frac{s}{2}}e_{0}|^{2}\mathrm{d}x+\frac{\bar{b}t^{4}}{4}\left( \int _{\Omega }|(-\Delta )^{\frac{s}{2}}e_{0}|^{2}\mathrm{d}x\right) ^{2}-\frac{t^{p}}{p}\int _{\Omega }|e_{0}|^{p}\mathrm{d}x, \end{aligned}$$

which implies that there exists $M>0$ (independent of T, b and $\lambda$) such that

$$\begin{aligned} c_{b, \lambda }^{T}\le \max _{t\in [0, 1]}J_{b, \lambda }^{T}(te_{0})\le M. \end{aligned}$$

$\square$

Lemma 3.3

Suppose that $2<p<4$ and $(V_{1})-(V_{3})$ hold, and let $T=\sqrt{\frac{2p(M+1)}{p-2}}$. Then, there exists $\hat{b}\in (0, \bar{b})$ such that for each $\lambda \ge 1$ and $b\in (0, \hat{b})$, if $\{u_{n}\}\subset E_{\lambda }$ is a sequence satisfying (3.3), then we have, up to a subsequence, $\Vert u_{n}\Vert _{\lambda }\le T$ that is $\{u_{n}\}$ is also a Cerami sequence at level $c_{b, \lambda }^{T}$ for $J_{b, \lambda }$.

Proof

Firstly, we aim to show that $\Vert u_{n}\Vert _{\lambda }\le \sqrt{2}T$ for n large enough. Assume by contradiction that there exists a subsequence, still denoted by $\{u_{n}\}$, such that $\Vert u_{n}\Vert _{\lambda }>\sqrt{2}T$. It is easy to see that

$$\begin{aligned} \begin{aligned} c_{b, \lambda }^{T}=&\lim _{n\rightarrow \infty }(J_{b, \lambda }^{T}(u_{n})-\frac{1}{p}\langle (J_{b, \lambda }^{T})'(u_{n}), u_{n}\rangle )\\ =&\lim _{n\rightarrow \infty }((\frac{1}{2}-\frac{1}{p})\Vert u_{n}\Vert _{\lambda }^{2}-(\frac{b}{p}-\frac{b}{4})\eta \left( \frac{\Vert u_{n}\Vert _{\lambda }^{2}}{T^{2}}\right) |(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{4}\\&-\frac{b}{2pT^{2}}\eta '\left( \frac{\Vert u_{n}\Vert _{\lambda }^{2}}{T^{2}}\right) |(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{4}\Vert u_{n}\Vert _{\lambda }^{2})\\ \ge&\frac{p-2}{2p}2T^{2}=2\frac{p-2}{2p}\frac{2p(M+1)}{p-2}=2(M+1), \end{aligned} \end{aligned}$$

(3.4)

which gives a contradiction by Lemma 3.2. Therefore, $\Vert u_{n}\Vert _{\lambda }\le \sqrt{2}T$ for n large enough. Now, we show that $\Vert u_{n}\Vert _{\lambda }\le T$. Assume by contradiction that there exists a subsequence, still denoted by $\{u_{n}\}$, such that $T<\Vert u_{n}\Vert _{\lambda }\le \sqrt{2}T$ for n large enough. According to the definition of $\eta$, we can see that

$$\begin{aligned} \begin{aligned} c_{b, \lambda }^{T}&=\lim _{n\rightarrow \infty }(J_{b, \lambda }^{T}(u_{n})-\frac{1}{p}\langle (J_{b, \lambda }^{T})'(u_{n}), u_{n}\rangle )\\&\ge \liminf _{n\rightarrow \infty }((\frac{1}{2}-\frac{1}{p})\Vert u_{n}\Vert _{\lambda }^{2}-(\frac{b}{p}-\frac{b}{4})\Vert u_{n}\Vert _{\lambda }^{4})\\&\ge (M+1)-\frac{4(4-p)pb(M+1)^{2}}{(p-2)^{2}}, \end{aligned} \end{aligned}$$

(3.5)

which implies a contradiction by choosing $\hat{b}>0$ small. $\square$

Now, we give a compact lemma about Cerami sequence $\{u_{n}\}$, which is crucial to prove our main result.

Lemma 3.4

Suppose that $2<p<4$ and $(V_{1})-(V_{3})$ hold, and let $T=\sqrt{\frac{2p(M+1)}{p-2}}$. Then, there exists $\tilde{\lambda }>1$ such that for each $b\in (0, \hat{b})$ and $\lambda \in (\tilde{\lambda }, \infty )$, if $\{u_{n}\}\subset E_{\lambda }$ is a sequence satisfying (3.3), then $\{u_{n}\}$ has a convergent subsequence in $E_{\lambda }$.

Proof

In view of Lemma 3.3, we can obtain, up to a subsequence, $\Vert u_{n}\Vert _{\lambda }\le T$. Up to a subsequence again, we may assume that there exist $u\in E_{\lambda }$ and $A\in \mathbb {R}$ such that

$$\begin{aligned}&u_{n}\rightarrow u \quad in \quad E_{\lambda }, \\&\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u_{n}|^{2}\mathrm{d}x\rightarrow A^{2}, \\&\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x\le A^{2}. \end{aligned}$$

By $I'_{b, \lambda }(u_{n})\rightarrow 0$, we can see that

$$\begin{aligned}&(1+bA^{2})\int _{\mathbb {R}^{3}}(-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}v\mathrm{d}x+\int _{\mathbb {R}^{3}}\lambda V(x)uv\mathrm{d}x\nonumber \\&-\int _{\mathbb {R}^{3}}|u^{+}|^{p-2}u^{+}v\mathrm{d}x=0, {\forall v\in E_{\lambda }.} \end{aligned}$$

(3.6)

Taking $v=u$ as test function in (3.6), we get

$$\begin{aligned} (1+bA^{2})\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x+\int _{\mathbb {R}^{3}}\lambda V(x)u^{2}\mathrm{d}x-\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x=0. \end{aligned}$$

(3.7)

Now, we show that $u_{n}\rightarrow u$ in $E_{\lambda }$. Let $v_{n}:=u_{n}-u$. By $(V_{2})$, we get

$$\begin{aligned} |v_{n}|_{2}^{2}=\int _{\mathbb {R}^{3}\backslash \mathcal {V}_{c}}v_{n}^{2}\mathrm{d}x+\int _{\mathcal {V}_{c}}v_{n}^{2}\mathrm{d}x\le \frac{1}{\lambda c}\Vert v_{n}\Vert _{\lambda }^{2}+o(1). \end{aligned}$$

(3.8)

By applying (3.8), Hölder’s inequality and Sobolev inequality, we can deduce that

$$\begin{aligned} \begin{aligned} |v_{n}|_{p}&\le |v_{n}|_{2}^{\theta }|v_{n}|_{2_{s}^{*}}^{1-\theta }\le S_{s}^{\theta -1}|v_{n}|_{2}^{\theta }|(-\Delta )^{\frac{s}{2}}v_{n}|_{2}^{1-\theta }\\&\le S_{s}^{\theta -1}(\lambda c)^{-\frac{\theta }{2}}\Vert v_{n}\Vert _{\lambda }^{\theta }|(-\Delta )^{\frac{s}{2}}v_{n}|_{2}^{1-\theta }+o(1)\\&\le S_{s}^{\theta -1}(\lambda c)^{-\frac{\theta }{2}}\Vert v_{n}\Vert _{\lambda }^{\theta }\Vert v_{n}\Vert _{\lambda }^{1-\theta }+o(1)\\&=S_{s}^{\theta -1}(\lambda c)^{-\frac{\theta }{2}}\Vert v_{n}\Vert _{\lambda }+o(1), \end{aligned} \end{aligned}$$

(3.9)

where $\theta$ satisfies that $\frac{1}{p}=\frac{\theta }{2}+\frac{1-\theta }{2_{s}^{*}}$.

By (2.1), (3.7) and (3.9), we can see that

$$\begin{aligned} \begin{aligned} o(1)&=\langle J'_{b, \lambda }(u_{n}), u_{n}\rangle \\&-\left[ (1+bA^{2})\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u|^{2}\mathrm{d}x+\int _{\mathbb {R}^{3}}\lambda V(x)u^{2}\mathrm{d}x-\int _{\mathbb {R}^{3}}|u^{+}|^{p}\mathrm{d}x\right] \\&=\Vert u_{n}\Vert _{\lambda }^{2}+b|(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{4}-|u_{n}^{+}|_{p}^{p}-\Vert u\Vert _{\lambda }^{2}-bA^{2}|(-\Delta )^{\frac{s}{2}}u|_{2}^{2}+|u^{+}|_{p}^{p}\\&=\Vert v_{n}\Vert _{\lambda }^{2}-|v_{n}^{+}|_{p}^{p}+bA^{4}-bA^{2}|(-\Delta )^{\frac{s}{2}}u|_{2}^{2}+o(1)\\&\ge \Vert v_{n}\Vert _{\lambda }^{2}-|v_{n}|_{p}^{p-2}|v_{n}|_{p}^{2}+o(1)\\&\ge \Vert v_{n}\Vert _{\lambda }^{2}-L_{p}^{p-2}\Vert v_{n}\Vert _{\lambda }^{p-2}S_{s}^{2\theta -2}(\lambda c)^{-\theta }\Vert v_{n}\Vert _{\lambda }^{2}+o(1)\\&\ge [1-(2L_{p}T)^{p-2}S_{s}^{2\theta -2}(\lambda c)^{-\theta }]\Vert v_{n}\Vert _{\lambda }^{2}+o(1). \end{aligned} \end{aligned}$$

Therefore, there exists $\tilde{\lambda }>1$ such that $\lambda >\tilde{\lambda }$, then $u_{n}\rightarrow u$ in $E_{\lambda }$.

$\square$

Proof of Theorem 1.1

Defined T in Lemma 3.3. Form Lemma 2.1 and Lemma 3.1, we have that there exists $\bar{b}>0$ such that for $\lambda \ge 1$ and $b\in (0, \bar{b})$, $I_{b, \lambda }^{T}$ possesses a Cerami sequence $\{u_{n}\}\subset E_{\lambda }$ at mountain pass level $c_{b, \lambda }^{T}$. By Lemma 3.2 and Lemma 3.3, it is easy to see that there exists $\hat{b}\in (0, \bar{b})$ such that for $\lambda \ge 1$ and $b\in (0, \hat{b})$, $\{u_{n}\}$ is also a Cerami sequence at level $c_{b, \lambda }^{T}$ for $J_{b, \lambda }$ satisfying $\Vert u_{n}\Vert _{\lambda }\le T$, that is,

$$\begin{aligned} \sup _{n\in \mathbb {N}}\Vert u_{n}\Vert _{\lambda }\le T, \quad J_{b, \lambda }(u_{n})\rightarrow c_{b, \lambda }^{T} \quad and \quad (1+\Vert u_{n}\Vert _{\lambda })\Vert J'_{b, \lambda }(u_{n})\Vert _{E'_{\lambda }}\rightarrow 0. \end{aligned}$$

By applying Lemma 3.4, we know that there exists $\tilde{\lambda }>1$ such that for each $b\in (0, \hat{b})$ and $\lambda \in (\tilde{\lambda }, \infty )$, then $\{u_{n}\}$ has a convergent subsequence in $E_{\lambda }$. We assume that $u_{n}\rightarrow u_{b, \lambda }$ as $n\rightarrow \infty$; then, we have

$$\begin{aligned} \Vert u_{b, \lambda }\Vert _{\lambda }\le T, \quad J_{b, \lambda }(u_{b, \lambda })=c_{b, \lambda }^{T} \quad and \quad J'_{b, \lambda }(u_{b, \lambda })=0. \end{aligned}$$

By using Lemma 2.2, we can get that $u_{b, \lambda }$ is a positive solution of problem (1.5). Finally, since $\langle J'_{b, \lambda }(u_{b, \lambda }), u_{b, \lambda }\rangle$ and $u_{b, \lambda }\ne 0$, we have

$$\begin{aligned} \Vert u_{b, \lambda }\Vert _{\lambda }^{2}\le |u_{b, \lambda }^{+}|_{p}^{p}\le L_{p}^{p}\Vert u_{b, \lambda }\Vert _{\lambda }^{p}, \end{aligned}$$

which implies that there exists $\tau >0$ (independent of b and $\lambda$) such that $\Vert u_{b, \lambda }\Vert _{\lambda }\ge \tau$. $\square$

4 Proof of Theorem 1.3

In this section, we will prove Theorem 1.3. For this purpose, let $u_{b, \lambda }$ be the positive solution of (1.5) obtained by Theorem 1.1. Firstly, let us give an important estimate involving the $L^{\infty }$-norm of $u_{b, \lambda }$ under the assumptions of Theorem 1.3.

Lemma 4.1

Under the assumptions of Theorem 1.3, then $u_{b, \lambda }\in L^{\infty }(\mathbb {R}^{3})$ and there exists $C_{0}>0$ such that

$$\begin{aligned} |u_{b, \lambda }|_{\infty }\le C_{0}, \quad \mathrm{for all } \, b, \lambda . \end{aligned}$$

Moreover,

$$\begin{aligned} \lim _{|x|\rightarrow \infty }u_{b, \lambda }(x)=0. \end{aligned}$$

Proof

For $\beta \ge 1$ and $\widehat{T}>0$, we define

$$\begin{aligned} \begin{aligned}&\varphi (t)=\left\{ \begin{array}{ll} 0, &{} t\le 0, \\ t^{\beta }, &{} 0<t<\widehat{T}, \\ \beta \widehat{T}^{\beta -1}(t-\widehat{T})+\widehat{T}^{\beta }, &{} t\ge \widehat{T}. \end{array} \right. \end{aligned} \end{aligned}$$

Note that $\varphi$ is convex and Lipschitz, then

$$\begin{aligned} (-\Delta )^{s}\varphi (u_{b, \lambda })\le \varphi '(u_{b, \lambda })(-\Delta )^{s}u_{b, \lambda }, \end{aligned}$$

(4.1)

in the weak sense. Moreover, since

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}\varphi (u_{b, \lambda })|^{2}\mathrm{d}x&=\int \int _{\mathbb {R}^{6}}\frac{|\varphi (u_{b, \lambda })(x)-\varphi (u_{b, \lambda })(y)|^{2}}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\\&\le \beta \widehat{T}^{\beta -1}\int \int _{\mathbb {R}^{6}}\frac{|u_{b, \lambda }(x)-u_{b, \lambda }(y)|^{2}}{|x-y|^{3+2s}}\mathrm{d}x\mathrm{d}y\\&<+\infty , \end{aligned} \end{aligned}$$

which implies that $\varphi (u_{b, \lambda })\in D^{s, 2}(\mathbb {R}^{3})$. By Sobolev inequality, integrate by parts, $u_{b, \lambda }>0$, $(V_{1})$ and (4.1), we can see that

$$\begin{aligned} \begin{aligned} |\varphi (u_{b, \lambda })|_{2_{s}^{*}}^{2}\le&S_{s}^{-1}\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}\varphi (u_{b, \lambda })|^{2}\mathrm{d}x\\ =\,\,&S_{s}^{-1}\int _{\mathbb {R}^{3}}\varphi (u_{b, \lambda })(-\Delta )^{s}\varphi (u_{b, \lambda })\mathrm{d}x\\ \le&S_{s}^{-1}\int _{\mathbb {R}^{3}}\varphi (u_{b, \lambda })\varphi '(u_{b, \lambda })(-\Delta )^{s}u_{b, \lambda }\mathrm{d}x\\ =\,\,&\frac{S_{s}^{-1}}{1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|^{2} \mathrm{d}x}\int _{\mathbb {R}^{3}}\varphi (u_{b, \lambda })\varphi '(u_{b, \lambda })\\&{}[-\lambda V(x)u_{b, \lambda }+|u_{b, \lambda }^{+}|^{p-2}u_{b, \lambda }^{+}]\mathrm{d}x\\ \le \,\,&C_{1}\int _{\mathbb {R}^{3}}\varphi (u_{b, \lambda })\varphi '(u_{b, \lambda })(1+u_{b, \lambda }^{2_{s}^{*}-1})\mathrm{d}x\\ =\,\,&C_{1}\left( \int _{\mathbb {R}^{3}}\varphi (u_{b, \lambda })\varphi '(u_{b, \lambda })\mathrm{d}x+\int _{\mathbb {R}^{3}}\varphi (u_{b, \lambda })\varphi '(u_{b, \lambda })u_{b, \lambda }^{2_{s}^{*}-1}\mathrm{d}x\right) , \end{aligned} \end{aligned}$$

where $C_{1}$ is a constant independent of $b, \lambda$ and $\beta$. Now by applying the fact that

$$\begin{aligned} \varphi '(u_{b, \lambda })\varphi (u_{b, \lambda })\le \beta u_{b, \lambda }^{2\beta -1}, \end{aligned}$$

and

$$\begin{aligned} u_{b, \lambda }\varphi '(u_{b, \lambda })\le \beta \varphi (u_{b, \lambda }), \end{aligned}$$

then we can obtain that

$$\begin{aligned} |\varphi (u_{b, \lambda })|_{2_{s}^{*}}^{2}\le C_{1}\beta \left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta -1}\mathrm{d}x+\int _{\mathbb {R}^{3}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x\right) . \end{aligned}$$

(4.2)

Clearly, the last integral is well defined for every $\widehat{T}$. In fact,

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x&=\int _{\{u_{b, \lambda }\le \widehat{T}\}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x+\int _{\{u_{b, \lambda }>\widehat{T}\}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x\\&\,\le \widehat{T}^{2\beta -2}\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x+C\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x<+\infty , \end{aligned} \end{aligned}$$

where we have used that $\beta \ge 1$ and that $\varphi (u_{b, \lambda })$ is linear when $u_{b, \lambda }\ge \widehat{T}$.

Let $\beta$ in (4.2) such that $2\beta -1=2_{s}^{*}$ and denote $\beta _{1}=\frac{2_{s}^{*}+1}{2}$. Moreover, let $R>0$ be fixed later, then by applying Hölder’s inequality, it is easy to deduce that

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x&=\int _{\{u_{b, \lambda }\le R\}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x+\int _{\{u_{b, \lambda }>R\}}(\varphi (u_{b, \lambda }))^{2}u_{b, \lambda }^{2_{s}^{*}-2}\mathrm{d}x\\&\,\le R^{2_{s}^{*}-1}\int _{\{u_{b, \lambda }\le R\}}\frac{(\varphi (u_{b, \lambda }))^{2}}{u_{b, \lambda }}\mathrm{d}x\\&\quad \,+\left( \int _{\{u_{b, \lambda }>R\}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{2_{s}^{*}-2}{2_{s}^{*}}}\left( \int _{\mathbb {R}^{3}}(\varphi (u_{b, \lambda }))^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{2}{2_{s}^{*}}}. \end{aligned} \end{aligned}$$

(4.3)

From (1.6), we know that $u_{b, \lambda }$ is bounded in $E_{\lambda }$, so we can choose R sufficiently large such that

$$\begin{aligned} \left( \int _{\{u_{b, \lambda }>R\}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{2_{s}^{*}-2}{2_{s}^{*}}}\le \frac{1}{2C_{1}\beta _{1}}. \end{aligned}$$

(4.4)

By applying (4.2)–(4.4), we can deduce that

$$\begin{aligned} \left( \int _{\mathbb {R}^{3}}(\varphi (u_{b, \lambda }))^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{2}{2_{s}^{*}}}\le 2C_{1}\beta _{1}\left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x+R^{2_{s}^{*}-1}\int _{\mathbb {R}^{3}}\frac{(\varphi (u_{b, \lambda }))^{2}}{u_{b, \lambda }}\mathrm{d}x\right) . \end{aligned}$$

(4.5)

Now, by using $\varphi (u_{b, \lambda })\le u_{b, \lambda }^{\beta _{1}}$ and let $\widehat{T}\rightarrow \infty$, we can see that

$$\begin{aligned} \left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta _{1}}\mathrm{d}x\right) ^{\frac{2}{2_{s}^{*}}}\le 2C_{1}\beta _{1}\left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x+R^{2_{s}^{*}-1}\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x\right) <\infty , \end{aligned}$$

which implies that

$$\begin{aligned} u_{b, \lambda }\in L^{2_{s}^{*}\beta _{1}}(\mathbb {R}^{3}). \end{aligned}$$

(4.6)

Now, let $\beta >\beta _{1}$. Using $\varphi (u_{b, \lambda })\le u_{b, \lambda }^{\beta }$, (4.2), and letting $\widehat{T}\rightarrow \infty$, we get that

$$\begin{aligned} \left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta }\mathrm{d}x\right) ^{\frac{2}{2_{s}^{*}}}\le C_{1}\beta \left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta -1}\mathrm{d}x+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta +2_{s}^{*}-2}\mathrm{d}x\right) . \end{aligned}$$

(4.7)

Let

$$\begin{aligned} u_{b, \lambda }^{2\beta -1}=u_{b, \lambda }^{l}u_{b, \lambda }^{k}, \end{aligned}$$

where $l=\frac{2_{s}^{*}(2_{s}^{*}-1)}{2(\beta -1)}$ and $k=2\beta -1-l$. Moreover, $\beta >\beta _{1}$ implies that $0<l, k<2_{s}^{*}$; then, by Young’s inequality with exponents

$$\begin{aligned} r=\frac{2_{s}^{*}}{l}\quad and \quad r'=\frac{2_{s}^{*}}{2_{s}^{*}-l}, \end{aligned}$$

we have

$$\begin{aligned} \begin{aligned} \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta -1}\mathrm{d}x&\le \frac{l}{2_{s}^{*}}\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x+\frac{2_{s}^{*}-l}{2_{s}^{*}}\int _{\mathbb {R}^{3}}u_{b, \lambda }^{\frac{2_{s}^{*}k}{2_{s}^{*}-l}}\mathrm{d}x\\&\le \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta +2_{s}^{*}-2}\mathrm{d}x\\&\le C\left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta +2_{s}^{*}-2}\mathrm{d}x\right) . \end{aligned} \end{aligned}$$

(4.8)

By (4.7) and (4.8), we have

$$\begin{aligned} \left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta }\mathrm{d}x\right) ^{\frac{2}{2_{s}^{*}}}\le C\beta \left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta +2_{s}^{*}-2}\mathrm{d}x\right) . \end{aligned}$$

(4.9)

So, we can obtain that

$$\begin{aligned} \left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta }\mathrm{d}x\right) ^{\frac{1}{2_{s}^{*}(\beta -1)}}\le (C\beta )^{\frac{1}{2(\beta -1)}}\left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2\beta +2_{s}^{*}-2}\mathrm{d}x\right) ^{\frac{1}{2(\beta -1)}}. \end{aligned}$$

(4.10)

Iterating this argument, we obtain

$$\begin{aligned} \left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta _{i+1}}\mathrm{d}x\right) ^{\frac{1}{2_{s}^{*}(\beta _{i+1}-1)}}\le (C\beta _{i+1})^{\frac{1}{2(\beta _{i+1}-1)}}\left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta _{i}}\mathrm{d}x\right) ^{\frac{1}{2(\beta _{i}-1)}}, \end{aligned}$$

(4.11)

where

$$\begin{aligned} 2\beta _{i+1}+2_{s}^{*}-2=2_{s}^{*}\beta _{i}, \quad \beta _{i+1}=(\frac{2_{s}^{*}}{2})^{i}(\beta _{i}-1)+1. \end{aligned}$$

Denoting $C_{i+1}=C\beta _{i+1}$ and

$$\begin{aligned} K_{i}=\left( 1+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta _{i}}\mathrm{d}x\right) ^{\frac{1}{2_{s}^{*}(\beta _{i}-1)}}. \end{aligned}$$

We can see that there exists a constant $C_{2}>0$ independent of i , such that

$$\begin{aligned} K_{i+1}\le \Pi _{i=2}^{i+1}C_{i}^{\frac{1}{2(\beta _{i}-1)}}K_{1}\le C_{2}K_{1}. \end{aligned}$$

Therefore, we can deduce that

$$\begin{aligned} |u_{b, \lambda }|_{\infty }\le C_{0}, \quad \mathrm{for\, all } \, b, \lambda . \end{aligned}$$

Now, using (4.6) and (4.7), we can see that

$$\begin{aligned} \begin{aligned} |u_{b, \lambda }|_{\infty }&\le C\left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x+\int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}\beta _{1}}\mathrm{d}x\right) ^{\frac{1}{2_{s}^{*}(\beta _{1}-1)}}\\&\le C\left[ \left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{1}{2_{s}^{*}(\beta _{1}-1)}}+\left( \int _{\mathbb {R}^{3}}u_{b, \lambda }^{2_{s}^{*}}\mathrm{d}x\right) ^{\frac{1}{2(\beta _{1}-1)}}\right] , \end{aligned} \end{aligned}$$

where $C>0$ independent of $u_{b, \lambda }$ and $\beta _{1}=\frac{2_{s}^{*}+1}{2}$. This yields $u_{b, \lambda }\in L^{r}(\mathbb {R}^{3})$ for all $r\in [2, +\infty ]$. Moreover,

$$\begin{aligned} (-\Delta )^{s}u_{b, \lambda }=\frac{1}{1+b|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|_{2}^{2}}(-\lambda V(x)u_{b, \lambda }+|u_{b, \lambda }^{+}|^{p-2}u_{b, \lambda }):=f(u_{b, \lambda }), \end{aligned}$$

since $V(x)\in L^{\infty }(\mathbb {R}^{3})$, hence $f(u_{b, \lambda })\in L^{\infty }(\mathbb {R}^{3})$. According to Proposition 2.1.9 in [23], we can get $u_{b, \lambda }\in C^{1, \alpha }(\mathbb {R}^{3})$ for all $\alpha <2s-1$ when $\frac{3}{4}<s<1$. Finally, the fact $u_{b, \lambda }\in L^{r}(\mathbb {R}^{3})\cap C^{1, \alpha }$ for all $2\le r\le \infty$ implies that $\lim _{|x|\rightarrow \infty }u_{b, \lambda }(x)=0$.

$\square$

Proof of Theorem 1.3

Since $u_{b, \lambda }$ is bounded in $E_{\lambda }$, then there exists some constant $A_{1}>0$ such that

$$\begin{aligned} 1+b\int _{\mathbb {R}^{3}}|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|^{2}\mathrm{d}x\le 1+bA_{1}^{2}. \end{aligned}$$

By Lemma 4.3 in [11], there exists a function w such that

$$\begin{aligned} 0<w\le \frac{C}{1+|x|^{3+2s}}, \end{aligned}$$

(4.12)

and

$$\begin{aligned} (-\Delta )^{s}w+\frac{1}{2(1+bA_{1}^{2})}w\ge 0, \quad \mathrm{in } \mathbb {R}^3\backslash B_{R_{1}}(0), \end{aligned}$$

(4.13)

for some suitable $R_{1}>0$. Note that

$$\begin{aligned} \begin{aligned}&(-\Delta )^{s}u_{b, \lambda }+\frac{1}{2(1+bA_{1}^{2})}u_{b, \lambda }=\frac{-\lambda V(x)u_{b, \lambda }+|u_{b, \lambda }^{+}|^{p-2}u_{b, \lambda }}{1+b|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|_{2}^{2}}+\frac{u_{b, \lambda }}{2(1+bA_{1}^{2})}\\&\quad \le \frac{-\lambda V(x)u_{b, \lambda }+|u_{b, \lambda }^{+}|^{p-2}u_{b, \lambda }}{1+b|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|_{2}^{2}}+\frac{u_{b, \lambda }}{2(1+b|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|_{2}^{2})}\\&\quad =\frac{1}{1+b|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|_{2}^{2}}\left[ |u_{b, \lambda }^{+}|^{p-2}u_{b, \lambda }-(\lambda V(x)-\frac{1}{2})u_{b, \lambda }\right] . \end{aligned} \end{aligned}$$

(4.14)

By $(V'_{2})$, there exists $R_{2}>0$ such that $\mathcal {V}_{c}\subset B_{R_{2}}(0)$ and so

$$\begin{aligned} V(x)\ge c, \quad \mathrm{for } \, |x|\ge R_{2}. \end{aligned}$$

(4.15)

By (4.14), (4.15) and Lemma 4.1, we get

$$\begin{aligned} (-\Delta )^{s}u_{b, \lambda }+\frac{1}{2(1+bA_{1}^{2})}u_{b, \lambda }\le \frac{1}{1+b|(-\Delta )^{\frac{s}{2}}u_{b, \lambda }|_{2}^{2}}\left[ u_{b, \lambda }(C_{0}^{p-2}-\lambda c+\frac{1}{2})\right] .\nonumber \\ \end{aligned}$$

(4.16)

Hence, we choose $\tilde{\Lambda }>\tilde{\lambda }>1$ large such that $\lambda \in (\tilde{\Lambda }, \infty )$; we have $C_{0}^{p-2}-\lambda c+\frac{1}{2}\le 0$. Then,

$$\begin{aligned} (-\Delta )^{s}u_{b, \lambda }+\frac{1}{2(1+bA_{1}^{2})}u_{b, \lambda }\le 0, \end{aligned}$$

(4.17)

for $|x|\ge R_{2}$ and $\lambda \in (\tilde{\Lambda }, \infty )$.

Now, let $R_{3}=\max \{R_{1}, R_{2}\}$ and set

$$\begin{aligned} \gamma :=\inf _{B_{R_{3}}(0)}w>0, \quad w_{b, \lambda }=(k+1)w-\gamma u_{b, \lambda }, \end{aligned}$$

(4.18)

where $k=\sup |u_{b, \lambda }|_{\infty }<\infty$. Now, we claim that $w_{b, \lambda }\ge 0$ in $\mathbb {R}^{3}$. In fact, suppose by contradiction that there exists a sequence $x_{j}$ such that

$$\begin{aligned} \inf _{x\in \mathbb {R}^{3}}w_{b, \lambda }(x)=\lim _{j\rightarrow \infty }w_{b, \lambda }(x_{j})<0. \end{aligned}$$

(4.19)

By Lemma 4.1 and (4.12), we have that

$$\begin{aligned} \lim _{|x|\rightarrow \infty }w(x)=\lim _{|x|\rightarrow \infty }u_{b, \lambda }(x)=0, \end{aligned}$$

so (4.18) implies that

$$\begin{aligned} \lim _{|x|\rightarrow \infty }w_{b, \lambda }(x)=0. \end{aligned}$$

(4.20)

Putting together (4.19) and (4.20), we can see that $\{x_{j}\}$ is bounded and therefore, up to a subsequence, we may suppose that $x_{j}\rightarrow x_{*}$ for some $x_{*}\in \mathbb {R}^{3}$ as $j\rightarrow \infty$. So, by (4.19), we have

$$\begin{aligned} \inf _{x\in \mathbb {R}^{3}}w_{b, \lambda }(x)=w_{b, \lambda }(x_{*})<0, \end{aligned}$$

(4.21)

then

$$\begin{aligned} (-\Delta )^{s}w_{b, \lambda }(x_{*})=\frac{C_{3, s}}{2}\int _{\mathbb {R}^{3}}\frac{2w_{b, \lambda }(x_{*})-w_{b, \lambda }(x_{*}+y)-w_{b, \lambda }(x_{*}-y)}{|y|^{3+2s}}\mathrm{d}x\le 0.\nonumber \\ \end{aligned}$$

(4.22)

By (4.18), we can obtain that

$$\begin{aligned} w_{b, \lambda }(x)\ge k\gamma +w-k\gamma >0, \quad \mathrm{in } B_{R_{3}}(0); \end{aligned}$$

hence, by (4.19), we have

$$\begin{aligned} x_{*}\in \mathbb {R}^{3}\backslash B_{R_{3}}(0). \end{aligned}$$

(4.23)

Putting together (4.12), (4.13) and (4.17), we can see that

$$\begin{aligned} (-\Delta )^{s}w_{b, \lambda }+\frac{1}{2(1+bA_{1}^{2})}w_{b, \lambda }\ge 0,\quad \mathrm{in } \, \mathbb {R}^{3}\backslash B_{R_{3}}(0). \end{aligned}$$

(4.24)

So by (4.21)–(4.24), we can see that

$$\begin{aligned} 0\le (-\Delta )^{s}w_{b, \lambda }(x_{*})+\frac{1}{2(1+bA_{1}^{2})}w_{b, \lambda }(x_{*})<0, \end{aligned}$$

which is a contradiction; then, $w_{b, \lambda }\ge 0$ in $\mathbb {R}^{3}$. Therefore,

$$\begin{aligned} u_{b, \lambda }\le (k+1)\gamma ^{-1}w. \end{aligned}$$

Finally, (4.12) implies that

$$\begin{aligned} 0<u_{b, \lambda }\le \frac{C}{1+|x|^{3+2s}} \quad for \quad |x|>R, \end{aligned}$$

where $R\ge R_{3}$. The proof of Theorem 1.3 is complete. $\square$

5 Proof of Theorems 1.5, 1.6 and 1.7

In this last section, we study the asymptotic behavior of positive solutions for (1.5) and give the proofs of Theorems 1.5, 1.6 and 1.7.

Proof of Theorem 1.5

Let $b\in (0, \hat{b})$ be fixed, then for any sequence $\lambda _{n}\rightarrow \infty$, let $u_{n}:=u_{b, \lambda _{n}}$ be the positive solution of (1.5) obtained by Theorem 1.1. By (1.6), we know that

$$\begin{aligned} \tau \le \Vert u_{n}\Vert _{\lambda _{n}}\le T. \end{aligned}$$

(5.1)

Up to a subsequence, we may assume that

$$\begin{aligned} \begin{aligned} \left\{ \begin{array}{ll} u_{n}\rightharpoonup u_{b}, &{} \mathrm{in } \, E, \\ u_{n}\rightarrow u_{b}, &{} \mathrm{in } \, L_{loc}^{s}(\mathbb {R}^{3}) \quad \mathrm{for }\, s\in [2, 2_{s}^{*}),\\ u_{n}\rightarrow u_{b}, &{} \mathrm{a.e.\, on } \, \mathbb {R}^{3}. \end{array} \right. \end{aligned} \end{aligned}$$

(5.2)

By applying Fatou’s Lemma and (5.1), we can deduce that

$$\begin{aligned} \int _{\mathbb {R}^{3}}V(x)u_{b}^{2}\mathrm{d}x\le \liminf _{n\rightarrow \infty }\int _{\mathbb {R}^{3}}V(x)u_{n}^{2}\mathrm{d}x\le \liminf _{n\rightarrow \infty }\frac{\Vert u_{n}\Vert _{\lambda _{n}}}{\lambda _{n}}=0, \end{aligned}$$

which implies that $u_{b}=0$ a.e. in $\mathbb {R}^{3}\backslash V^{-1}(0)$. Hence, by $(V_{3})$, we have that $u_{b}\in H_{0}^{s}(\Omega )$.

Next, we show that $u_{n}\rightarrow u_{b}$ in $L^{s}(\mathbb {R}^{3})$ for $s\in (2, 2_{s}^{*})$. If not, recalling that Lemma 2.3, we can see that there exist $\delta , r>0$ and $x_{n}\in \mathbb {R}^{3}$ such that

$$\begin{aligned} \int _{B_{r}(x_{n})}(u_{n}-u_{b})^{2}\mathrm{d}x\ge \delta , \end{aligned}$$

which implies that $|x_{n}|\rightarrow \infty$, then $|B_{r}(x_{n})\cap \mathcal {V}_{c}|\rightarrow 0$. Hence, by using Hölder’s inequality, it is easy to deduce that

$$\begin{aligned} \int _{B_{r}(x_{n})\cap \mathcal {V}_{c}}(u_{n}-u_{b})^{2}\mathrm{d}x\rightarrow 0. \end{aligned}$$

Thus,

$$\begin{aligned} \begin{aligned} \Vert u_{n}\Vert _{\lambda _{n}}^{2}\ge&\lambda _{n}c\int _{B_{r}(x_{n})\cap \{V\ge c\}}u_{n}^{2}\mathrm{d}x=\lambda _{n}c\int _{B_{r}(x_{n})\cap \{V\ge c\}}(u_{n}-u_{b})^{2}\mathrm{d}x\\ =&\lambda _{n}c\left( \int _{B_{r}(x_{n})}(u_{n}-u_{b})^{2}\mathrm{d}x-\int _{B_{r}(x_{n})\cap \mathcal {V}_{c}}(u_{n}-u_{b})^{2}\mathrm{d}x\right) \\&\rightarrow \infty , \end{aligned} \end{aligned}$$

which is a contradicts since (5.1).

Now, we show that $u_{n}\rightarrow u_{b}$ in E. Using $\langle J'_{b, \lambda _{n}}(u_{n}), u_{n}\rangle =\langle J'_{b, \lambda _{n}}(u_{n}), u_{b}\rangle =0$, we can obtain that

$$\begin{aligned} \Vert u_{n}\Vert _{\lambda _{n}}^{2}+b|(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{4}=|u_{n}^{+}|_{p}^{p}, \end{aligned}$$

(5.3)

and

$$\begin{aligned} \Vert u_{b}\Vert ^{2}+b|(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{2}|(-\Delta )^{\frac{s}{2}}u_{b}|_{2}^{2}=|u_{n}^{+}|_{p}^{p}+o(1). \end{aligned}$$

(5.4)

Up to a subsequence, we may assume that $\Vert u_{n}\Vert _{\lambda _{n}}^{2}\rightarrow l_{1}$ and $|(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{2}\rightarrow l_{2}$. It is easy see that

$$\begin{aligned} |(-\Delta )^{\frac{s}{2}}u_{b}|_{2}^{2}\le \liminf _{n\rightarrow \infty }|(-\Delta )^{\frac{s}{2}}u_{n}|_{2}^{2}=l_{2}, \end{aligned}$$

(5.5)

and

$$\begin{aligned} |u_{n}^{+}|_{p}^{p}=|u_{b}^{+}|_{p}^{p}+o(1). \end{aligned}$$

(5.6)

Putting together (5.3)–(5.6), we obtain

$$\begin{aligned} l_{1}+bl_{2}^{2}=\Vert u_{b}\Vert ^{2}+bl_{2}|(-\Delta )^{\frac{s}{2}}u_{b}|_{2}^{2}+o(1)\le \Vert u_{b}\Vert ^{2}+bl_{2}^{2}, \end{aligned}$$

which implies that $l_{1}\le \Vert u_{b}\Vert ^{2}$. Then, the weakly lower semi-continuity of norm shows that

$$\begin{aligned} \Vert u_{b}\Vert ^{2}\le \liminf _{n\rightarrow \infty }\Vert u_{n}\Vert ^{2}\le \limsup _{n\rightarrow \infty }\Vert u_{n}\Vert ^{2}\le \lim _{n\rightarrow \infty }\Vert u_{n}\Vert _{\lambda _{n}}^{2}=l_{1}\le \Vert u_{b}\Vert ^{2}, \end{aligned}$$

(5.7)

which implies that $u_{n}\rightarrow u_{b}$ in E.

Finally, we prove that $u_{b}$ is a positive solution of (1.8). For any $v\in C_{0}^{\infty }(\Omega )$, by using $\langle J'_{b, \lambda _{n}}(u_{n}), v\rangle =0$, we can see that

$$\begin{aligned} \left( 1+b\int _{\Omega }|(-\Delta )^{\frac{s}{2}}u_{b}|^{2}\mathrm{d}x\right) \int _{\Omega }(-\Delta )^{\frac{s}{2}}u_{b}(-\Delta )^{\frac{s}{2}}v\mathrm{d}x=\int _{\Omega }|u_{b}^{+}|^{p-2}u_{b}^{+}v\mathrm{d}x, \end{aligned}$$

which implies that $u_{b}$ is a nonnegative solution of (1.8) by the density of $C_{0}^{\infty }(\Omega )$ in $H_{0}^{s}(\Omega )$. Moreover, by (5.1) and (5.7), we can obtain that

$$\begin{aligned} \Vert u_{b}\Vert =\lim _{n\rightarrow \infty }\Vert u_{n}\Vert _{\lambda _{n}}\ge \tau >0, \end{aligned}$$

which implies that $u_{b}\ne 0$. Similar to the proof Lemma 2.2, we obtain $u_{b}>0$ in $\mathbb {R}^{3}$ and this ends the proof of Theorem. $\square$

Proof of Theorem 1.6

Let $\lambda \in (\tilde{\lambda }, \infty )$ be fixed, then for any sequence $b_{n}\rightarrow 0$, let $u_{n}:=u_{b_{n}, \lambda }$ be the positive solution of (1.5) obtained by Theorem 1.1. By (1.6), we know that

$$\begin{aligned} \tau \le \Vert u_{n}\Vert _{\lambda }\le T. \end{aligned}$$

(5.8)

Up to a subsequence, we may assume that

$$\begin{aligned} u_{n}\rightharpoonup u_{\lambda },\quad \mathrm{in } \, E_{\lambda }; \end{aligned}$$

(5.9)

since $J'_{b_{n}, \lambda }(u_{n})=0$, then by Lemma 3.4, we can see that $u_{n}\rightarrow u_{\lambda }$ in $E_{\lambda }$.

To complete the proof, it suffices to show that $u_{\lambda }$ is a positive solution of (1.9). For any $v\in E_{\lambda }$, by using $\langle J'_{b_{n}, \lambda }(u_{n}), v\rangle =0$, we can see that

$$\begin{aligned} \int _{\mathbb {R}^{3}}((-\Delta )^{\frac{s}{2}}u_{\lambda }(-\Delta )^{\frac{s}{2}}v+\lambda V(x)u_{\lambda }v)\mathrm{d}x=\int _{\mathbb {R}^{3}}|u_{\lambda }^{+}|^{p-2}u_{\lambda }^{+}v\mathrm{d}x, \end{aligned}$$

which implies that $u_{b}$ is a nonnegative solution of (1.9). Moreover, (5.8) shows that $u_{b}\ne 0$. Finally, similar to the proof Lemma 2.2, we obtain $u_{\lambda }>0$ in $\mathbb {R}^{3}$ and this ends the proof of Theorem. $\square$

Proof of Theorem 1.7

We can proceed exactly as in the proof of Theorem 1.5 and omit the details. $\square$