Ground state sign-changing solutions for a class of nonlinear fractional Schrödinger–Poisson system in ${\mathbb {R}}^{3}$

Abstract

In this paper, we are concerned with the existence of the least energy sign-changing solutions for the following fractional Schrödinger–Poisson system:

$$\begin{aligned} \left\{ \begin{aligned}&(-\Delta )^{s} u+V(x)u+\lambda \phi (x)u=f(x, u),\quad&\text {in}\, \ {\mathbb {R}}^{3},\\&(-\Delta )^{t}\phi =u^{2},&\text {in}\,\ {\mathbb {R}}^{3}, \end{aligned} \right. \end{aligned}$$

where $\lambda \in {\mathbb {R}}^{+}$ is a parameter, $s, t\in (0, 1)$ and $4s+2t>3$, $(-\Delta )^{s}$ stands for the fractional Laplacian. By constraint variational method and quantitative deformation lemma, we prove that the above problem has one least energy sign-changing solution. Moreover, for any $\lambda >0$, we show that the energy of the least energy sign-changing solutions is strictly larger than two times the ground state energy. Finally, we consider $\lambda$ as a parameter and study the convergence property of the least energy sign-changing solutions as $\lambda \searrow 0$.

1 Introduction

In this article, we are interested in the existence, energy property of the least energy sign-changing solution $u_{\lambda }$ and a convergence property of $u_{\lambda }$ as $\lambda \searrow 0$ for the nonlinear fractional Schrödinger–Poisson system

$$\begin{aligned} \left\{ \begin{aligned}&(-\Delta )^{s} u+V(x)u+\lambda \phi (x)u=f(x, u),\quad&\text {in}\, \ {\mathbb {R}}^{3},\\&(-\Delta )^{t}\phi =u^{2},&\text {in}\,\ {\mathbb {R}}^{3}. \end{aligned} \right. \end{aligned}$$

(1.1)

where $\lambda >0$ is a parameter, $s, t\in (0, 1)$ and $4s+2t>3$, $(-\Delta )^{s}$ stands for the fractional Laplacian and the potential V(x) satisfies the following assumptions:

$(V_{1})$:

$V\in C({\mathbb {R}}^{3})$ satisfies ${\inf }_{x\in {\mathbb {R}}^{3}}V(x)\ge V_{0}>0$, where $V_{0}$ is a positive constant;

$(V_{2})$:

There exists $h>0$ such that ${\lim _{\vert y\vert \rightarrow \infty }}\text {meas}(\{x\in B_{h}(y): V(x)\le c\})=0$ for any $c>0$;

where $B_{h}(y)$ denotes an open ball of ${\mathbb {R}}^{3}$ centered at y with radius $h>0$, and $\text {meas}(A)$ denotes the Lebesgue measure of set A. Condition $(V_{2})$, which is weaker than the coercivity assumption: $V(x)\rightarrow \infty$ as $\vert x\vert \rightarrow \infty$, was originally introduced by Bartsch and Wang [1] to overcome the lack of compactness for the local elliptic equations and then was used by Pucci, Xia and Zhang [18] for the fractional Schrödinger–Kirchhoff type equations. Moreover, on the nonlinearity f, we assume that

$(f_1)$:

$f: {\mathbb {R}}^{3}\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ is a Carathéodory function and $f(x, u)=o(\vert u\vert )$ as $u\rightarrow 0$ for $x\in {\mathbb {R}}^{3}$ uniformly;

$(f_2)$:

For some $1<p<2_{s}^{*}-1$, there exits $C>0$ such that

$$\begin{aligned} \vert f(x, u)\vert \le C(1+\vert u\vert ^{p}), \end{aligned}$$

where $2_{s}^{*}=\frac{6}{3-2s}$;

$(f_3)$:

${\lim }_{\vert u\vert \rightarrow \infty }\frac{F(x, u)}{u^{4}}=+\infty$, where $F(x, u)=\int _{0}^{u}f(x, s)\hbox {d}s$;

$(f_4)$:

$\frac{f(x, t)}{\vert t\vert ^{3}}$ is an increasing function of t on ${\mathbb {R}}\setminus \{0\}$ for a.e. $x\in {\mathbb {R}}^{3}$.

When $s=t=1$, the system (1.1) reduces to the following Schrödinger–Poisson system

$$\begin{aligned} \left\{ \begin{aligned}&-\Delta u+V(x)u+\lambda \phi (x)u=f(x, u),\quad&\text {in}\, \ {\mathbb {R}}^{3},\\&-\Delta \phi =u^{2},&\text {in}\,\ {\mathbb {R}}^{3}. \end{aligned} \right. \end{aligned}$$

This kind of system has a strong physical meaning. For instance, they appear in quantum mechanics models [4, 6] and in semiconductor theory [2, 3]. For the research of Schrödinger–Poisson system, we may refer to [9, 10, 13, 19, 23].

In recent years, there has been a great deal works dealing with the nonlinear equations or systems involving fractional Laplacian, which arise in fractional quantum mechanics [11, 12], physics and chemistry [14], obstacle problems [21], optimization and finance [8] and so on. In the remarkable work of Caffarelli and Silvestre [5], the authors express this nonlocal operator $(-\Delta )^{s}$ as a Dirichlet–Neumann map for a certain elliptic boundary value problem with local differential operators defined on the upper half space. This technique is a valid tool to deal with the equations involving fractional operators in the respects of regularity and variational methods. For some results on the fractional differential equations, we refer to [7, 16, 18, 25, 26]. Recently, using Caffarelli and Silvestre’s method in [5] and variational method, in [22], Teng studied the ground state for the fractional Schrödinger–Poisson system with critical Sobolev exponent. To the best of our knowledge, there are few papers which considered the least energy sign-changing solutions of system (1.1). In [20], Combining constraint variational methods and quantitative deformation lemma, Shuai firstly studied the least energy sign-changing solutions for a class of Kirchhoff problems in bounded domain, where a stronger condition that $f\in C^{1}$ was assumed. In virtue of the fractional operator and Poisson equation which are included in (1.1), our problem is more complicated and difficult.

Now, we recall some theory of the fractional Sobolev spaces. We firstly define the homogeneous fractional Sobolev space ${\mathcal {D}}^{\alpha , 2}({\mathbb {R}}^{3})(\alpha \in (0, 1))$ as follows

$$\begin{aligned} {\mathcal {D}}^{\alpha , 2}({\mathbb {R}}^{3})=\Big \{u\in L^{2_{\alpha }^{*}}({\mathbb {R}}^{3}): \frac{\vert u(x)-u(y)\vert }{\vert x-y\vert ^{\frac{3}{2}+\alpha }}\in L^{2}({\mathbb {R}}^{3}\times {\mathbb {R}}^{3})\Big \} \end{aligned}$$

which is the completion of ${\mathcal {C}}_{0}^{\infty }({\mathbb {R}}^{3})$ under the norm

$$\begin{aligned} \Vert u\Vert _{ {\mathcal {D}}^{\alpha , 2}({\mathbb {R}}^{3})}=\Vert (-\Delta )^{\frac{\alpha }{2}} u\Vert _{L^{2}({\mathbb {R}}^{3})}=\Big (\iint _{{\mathbb {R}}^{3}\times {\mathbb {R}}^{3}}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{3+2\alpha }}\hbox {d}x\hbox {d}y\Big )^{\frac{1}{2}}. \end{aligned}$$

The embedding ${\mathcal {D}}^{\alpha , 2}({\mathbb {R}}^{3})\hookrightarrow L^{2_{\alpha }^{*}}$ is continuous and there exists a best constant ${\mathcal {S}}_{\alpha }>0$ such that

$$\begin{aligned} {\mathcal {S}}_{\alpha }=\underset{u\in {\mathcal {D}}^{\alpha , 2}({\mathbb {R}}^{3})}{\inf }=\frac{\int _{{\mathbb {R}}^{3}} \vert (-\Delta )^{\frac{\alpha }{2}}u\vert ^{2}\hbox {d}x}{\Big (\int _{{\mathbb {R}}^{3}} \vert u(x)\vert ^{2_{\alpha }^{*}}\hbox {d}x\Big )^{\frac{2}{2_{\alpha }^{*}}}}. \end{aligned}$$

(1.2)

The fractional Sobolev space $H^{\alpha }({\mathbb {R}}^{3})$ is defined by

$$\begin{aligned} H^\alpha ({\mathbb {R}}^{3})=\Big \{u\in L^{2}({\mathbb {R}}^{3}): \frac{\vert u(x)-u(y)\vert }{\vert x-y\vert ^{\frac{3}{2}+\alpha }}\in L^{2}({\mathbb {R}}^{3}\times {\mathbb {R}}^{3})\Big \}, \end{aligned}$$

endowed with the norm

$$\begin{aligned} \Vert u\Vert _{H^\alpha ({\mathbb {R}}^{3})}=\Big (\iint _{{\mathbb {R}}^{3}\times {\mathbb {R}}^{3}}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{3+2\alpha }}\hbox {d}x\hbox {d}y+\int _{{\mathbb {R}}^{3}} \vert u\vert ^{2}\hbox {d}x\Big )^{\frac{1}{2}}. \end{aligned}$$

In this paper, we denote the fractional Sobolev space for (1.1) by

$$\begin{aligned} H= \Big \{u\in H^s({\mathbb {R}}^{3}): \int _{{\mathbb {R}}^{3}}V(x)u^{2}\hbox {d}x< \infty \Big \}, \end{aligned}$$

with the norm

$$\begin{aligned} \Vert u\Vert =\Big (\iint _{{\mathbb {R}}^{3}\times {\mathbb {R}}^{3}}\frac{\vert u(x)-u(y)\vert ^{2}}{\vert x-y\vert ^{3+2s}}\hbox {d}x\hbox {d}y+\int _{{\mathbb {R}}^{3}} V(x)u^{2}\hbox {d}x\Big )^{\frac{1}{2}}. \end{aligned}$$

In the sequel, we need the following embedding lemma which is a special case of Lemma 1 in [18], so we omit its proof.

Lemma 1.1

(i) Suppose that $(V_{1})$ holds. Let $q\in [2, 2_{s}^{*}]$, then the embeddings

$$\begin{aligned} H\hookrightarrow H^{s}({\mathbb {R}}^{3})\hookrightarrow L^{q}({\mathbb {R}}^{3}) \end{aligned}$$

are continuous, with $\min \{1, V_{0}\}\Vert u\Vert _{H^s({\mathbb {R}}^{3})}^{2}\le \Vert u\Vert ^{2}$ for all $u\in H$. In particular, there exists a constant $C_{q}>0$ such that

$$\begin{aligned} \Vert u\Vert _{L^q({\mathbb {R}}^{3})}\le C_{q}\Vert u\Vert \quad \text {for all}\,\, u\in H. \end{aligned}$$

Moreover, if $q\in [1, 2_{s}^{*})$, then the embedding $H\hookrightarrow \hookrightarrow L^{q}(B_{R})$ is compact for any $R>0$.

(ii) Suppose that $(V_{1})-(V_{2})$ hold. Let $q\in [2, 2_{s}^{*})$ be fixed and $\{u_{n}\}_{n}$ be a bounded sequence in H, then there exists $u\in H\cap L^{q}({\mathbb {R}}^{N})$ such that, up to a subsequence,

$$\begin{aligned} u_{n}\rightarrow u \quad \text {strongly in}\,\, L^q({\mathbb {R}}^{3})\, \text {as}\,\, n\rightarrow \infty . \end{aligned}$$

Assume that $s, t\in (0, 1)$, if $4s+2t\ge 3$, there holds $2\le \frac{12}{3+2t} \le \frac{6}{3-2s}$ and thus $H\hookrightarrow L^{\frac{12}{3+2t}}({\mathbb {R}}^{3})$ by Lemma 1.1. For $u\in H$, the linear functional ${\mathcal {L}}_{u}:{\mathcal {D}}^{t, 2}({\mathbb {R}}^{3})\rightarrow {\mathbb {R}}$ is defined by

$$\begin{aligned} {\mathcal {L}}_{u}(v)=\int _{{\mathbb {R}}^{3}} u^{2}v\hbox {d}x, \end{aligned}$$

the Hölder’s inequality and (1.2) implies that

$$\begin{aligned} \vert {\mathcal {L}}_{u}(v)\vert \le \Big (\int _{{\mathbb {R}}^{3}} \vert u(x)\vert ^{\frac{12}{3+2t}}\hbox {d}x\Big )^{\frac{3+2t}{6}}\Big (\int _{{\mathbb {R}}^{3}} \vert v(x)\vert ^{2_{t}^{*}}\hbox {d}x\Big )^{\frac{1}{2_{t}^{*}}}\le C\Vert u\Vert ^{2}\Vert v\Vert _{ {\mathcal {D}}^{t, 2}({\mathbb {R}}^{3})}. \end{aligned}$$

By the Lax–Milgram theorem, there exists a unique $\phi _{u}^{t}\in {\mathcal {D}}^{t, 2}({\mathbb {R}}^{3})$ such that

$$\begin{aligned} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{t}{2}}u(-\Delta )^{\frac{t}{2}}v\hbox {d}x=\int _{{\mathbb {R}}^{3}}u^{2}v\hbox {d}x,\quad \forall v\in {\mathcal {D}}^{t, 2}({\mathbb {R}}^{3}), \end{aligned}$$

that is $\phi _{u}^{t}$ is the weak solution of

$$\begin{aligned} (-\Delta )^{t}\phi _{u}^{t}=u^{2},\quad x\in {\mathbb {R}}^{3} \end{aligned}$$

and the representation formula holds

$$\begin{aligned} \phi _{u}^{t}(x)=c_{t}\int _{{\mathbb {R}}^{3}} \frac{u^{2}(y)}{\vert x-y\vert ^{3-2t}}\hbox {d}y,\quad x\in {\mathbb {R}}^{3}, \end{aligned}$$

which is called t-Riesz potential, where

$$\begin{aligned} c_{t}=\pi ^{-\frac{3}{2}}2^{-2t}\frac{\Gamma (3-2t)}{\Gamma (t)}. \end{aligned}$$

(1.3)

In the sequel, we often omit the constant $c_{t}$ for convenience in (1.3). The properties of the function $\phi _{u}^{t}$ are given as follows.

Lemma 1.2

([22]) If $4s+2t\ge 3$, then for any $u\in H^s({\mathbb {R}}^{3})$, we have

• (1) $\phi _{u}^{t}: H^s({\mathbb {R}}^{3})\rightarrow {\mathcal {D}}^{t, 2}({\mathbb {R}}^{3})$ is continuous and maps bounded sets into bounded maps;

• (2) $\int _{{\mathbb {R}}^{3}}\phi _{u}^{t}{u}^{2}\hbox {d}x\le {\mathcal {S}}_{t}^{2}\Vert u\Vert ^{4}_{L^{\frac{12}{3+2t}}}$;

• (3) $\phi _{\tau u}^{t}=\tau ^{2}\phi _{u}^{t}$ for all $\tau \in {\mathbb {R}}$, $\phi _{u(\cdot +y)}^{t}=\phi _{u}^{t}(x+y)$;

• (4) $\phi _{u_{\theta }}=\theta ^{2s}(\phi _{u}^{t})_{\theta }$ for all $\theta >0$, where $u_{\theta }=u(\frac{\cdot }{\theta });$

• (5) If $u_{n}\rightharpoonup u$ in $H^s({\mathbb {R}}^{3}),$ then $\phi _{u_{n}}^{t}\rightharpoonup \phi _{u}^{t}$ in ${\mathcal {D}}^{t, 2}({\mathbb {R}}^{3})$;

• (6) If $u_{n}\rightarrow u$ in $H^s({\mathbb {R}}^{3}),$ then $\phi _{u_{n}}^{t}\rightarrow \phi _{u}^{t}$ in ${\mathcal {D}}^{t, 2}({\mathbb {R}}^{3})$ and $\int _{{\mathbb {R}}^{3}}\phi _{u_{n}}^{t}{u_{n}}^{2}\hbox {d}x\rightarrow \int _{{\mathbb {R}}^{3}}\phi _{u}^{t}{u}^{2}\hbox {d}x$.

If we substitute $\phi _{u}^{t}$ in (1.1), it leads to the following fractional Schrödinger equation

$$\begin{aligned} (-\Delta )^{s} u+V(x)u+\lambda \phi _{u}^{t}u=f(x, u),\quad \text {in}\, \ {\mathbb {R}}^{3}, \end{aligned}$$

(1.4)

whose solutions are the critical points of the functional $I_{\lambda }: H\rightarrow {\mathbb {R}}$ defined by

$$\begin{aligned} I_{\lambda }(u)=\frac{1}{2}\int _{{\mathbb {R}}^{3}}(\vert (-\Delta )^{\frac{s}{2}}u\vert ^{2}+V(x) u^{2})\hbox {d}x+\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi _{u}^{t}u^{2}\hbox {d}x-\int _{{\mathbb {R}}^{3}}F(x, u)\hbox {d}x \end{aligned}$$

where $F(x, u)=\int _{0}^{u}f(x, r)\hbox {d}r$. The functional $I_{\lambda }\in C^{1}(H, {\mathbb {R}})$ and for any $v\in H$

$$\begin{aligned} \langle I_{\lambda }'(u), \varphi \rangle =\int _{{\mathbb {R}}^{3}}\Big ((-\Delta )^{\frac{s}{2}}u(-\Delta )^{\frac{s}{2}}\varphi +V(x)uv\Big )\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u}^{t}u\varphi \hbox {d}x-\int _{{\mathbb {R}}^{3}}f(x,u)\varphi \hbox {d}x. \end{aligned}$$

We call u a least energy sign-changing solution to problem (1.1) if u is a solution of problem (1.4) with $u^{\pm }\ne 0$ and

$$\begin{aligned} I_{\lambda }(u)=\inf \{I_{\lambda }(v)\mid v^{\pm }\ne 0, I'_{\lambda }(v)=0\}, \end{aligned}$$

where $v^{+}=\max \{v(x), 0\}$ and $v^{-}=\min \{v(x), 0\}$.

For problem (1.4), due to the effect of the nonlocal term $\phi _{u}^{t}$ and $(-\Delta )^{s} u$, that is

$$\begin{aligned} \int _{{\mathbb {R}}^{3}}\Big ((-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\Bigg )\hbox {d}x>0\quad \text {and}\quad \int _{{\mathbb {R}}^{3}}\phi _{u}^{t}u^{2}\hbox {d}x>\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}+\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{-})}^{2} \end{aligned}$$

for $u^{\pm }\ne 0$, a straightforward computation yields that

$$\begin{aligned}&I_{\lambda }(u)>I_{\lambda }(u^{+})+ I_{\lambda }(u^{-}),\nonumber \\&\langle I_{\lambda }'(u), u^{+}\rangle>\langle I_{\lambda }'(u^{+}), u^{+}\rangle ,\quad \text {and}\quad \langle I_{\lambda }'(u), u^{-}\rangle >\langle I_{\lambda }'(u^{-}), u^{-}\rangle . \end{aligned}$$

(1.5)

So the methods to obtain sign-changing solutions of the local problems and to estimate the energy of the sign-changing solutions seem not suitable for our nonlocal one (1.4). In order to get a sign-changing solution for problem (1.4), we firstly try to seek a minimizer of the energy functional $I_{\lambda }$ over the following constraint:

$$\begin{aligned} {\mathcal {M}}_{\lambda }=\{u\in H: u^{\pm }\ne 0, \langle I_{\lambda }'(u), u^{+}\rangle =\langle I_{\lambda }'(u), u^{-}\rangle =0\} \end{aligned}$$

and then we show that the minimizer is a sign-changing solution of (1.4). To show that the minimizer of the constrained problem is a sign-changing solution, we will use the quantitative deformation lemma and degree theory.

The following are the main results of this paper.

Theorem 1.1

Let $(f_{1})-(f_{4})$ and $(V_{1})-(V_{2})$ hold. Then, for any $\lambda >0$, problem (1.1) has a least energy sign-changing solution $u_{\lambda }$, which has precisely two nodal domains.

Let

$$\begin{aligned} {\mathcal {N}}_{\lambda }:=\{u\in H\setminus \{0\}: \langle I_{\lambda }'(u), u\rangle =0\}, \end{aligned}$$

(1.6)

and

$$\begin{aligned} c_{\lambda }:=\underset{u\in {\mathcal {N}}_{\lambda }}{\inf }I_{\lambda }(u). \end{aligned}$$

(1.7)

Let $u_{\lambda }\in H$ be a sign-changing solution of problem (1.4), it is clear from (1.5) and (1.6) that $u_{\lambda }^{\pm }\not \in {\mathcal {N}}_{\lambda }$.

Theorem 1.2

Under the assumptions of Theorem 1.1, $c_{\lambda }>0$ is achieved and $I_{\lambda }(u_{\lambda })>2c_{\lambda }$, where $u_{\lambda }$ is the least energy sign-changing solution obtained in Theorem 1.1. In particular, $c_{\lambda }>0$ is achieved either by a nonpositive or a nonnegative function.

It is clear that the energy of the sign-changing solution $u_{\lambda }$ obtained in Theorem 1.1 depends on $\lambda$. Furthermore, we give a convergence property of $u_{\lambda }$ as $\lambda \searrow 0$, which reflects some relationship between $\lambda >0$ and $\lambda =0$ in problem (1.4).

Theorem 1.3

If the assumptions of Theorem 1.1 hold, then for any sequence $\{\lambda _{n}\}_{n}$ with $\lambda _{n}\searrow 0$ as $n\rightarrow \infty$, there exists a subsequence, still denoted by $\{\lambda _{n}\}_{n}$, such that $u_{\lambda _{n}}\rightarrow u_{0}$ strongly in H as $n\rightarrow \infty$, where $u_{0}$ is a least energy sign-changing solution of the problem

$$\begin{aligned} (-\Delta )^{s} u+V(x)u=f(x, u),\quad \text {in}\,\ {\mathbb {R}}^{3}, \end{aligned}$$

(1.8)

which has precisely two nodal domains.

This paper is organized as follows. In Sect. 2, we present some preliminary lemmas which are essential for this paper. In Sect. 3, we give the proofs of Theorems 1.1–1.3, respectively.

2 Some technical lemmas

We will use constraint minimization on ${\mathcal {M}}_{\lambda }$ to look for a critical point of $I_{\lambda }$. For this, we start with this section by claiming that the set ${\mathcal {M}}_{\lambda }$ is nonempty in H.

Lemma 2.1

Assume that $(f_{1})-(f_{4})$ and $(V_{1})$ hold, if $u\in H$ with $u^{\pm }\ne 0$, then there exists a unique pair $(\alpha _{u}, \beta _{u})\in {\mathbb {R}}_{+}\times {\mathbb {R}}_{+}$ such that $\alpha _{u}u^{+}+\beta _{u}u^{-}\in {\mathcal {M}}_{\lambda }$.

Proof

Fixed an $u\in H$ with $u^{\pm }\ne 0$. We first establish the existence of $\alpha _{u}$ and $\beta _{u}$. Let

$$\begin{aligned} g_{1}(\alpha , \beta )&=\langle I_{\lambda }'(\alpha u^{+}+\beta u^{-}), \alpha u^{+}\rangle \nonumber \\&=\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}(\alpha u^{+}+\beta u^{-})(-\Delta )^{\frac{s}{2}}(\alpha u^{+})\hbox {d}x +\alpha ^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\nonumber \\&\quad +\,\lambda \alpha ^{2}\int _{{\mathbb {R}}^{3}}\phi _{\alpha u^{+}+\beta u^{-}}^{t}{(u^{+})}^{2} \hbox {d}x-\int _{{\mathbb {R}}^{3}}f(x,\alpha u^{+})\alpha u^{+} \hbox {d}x\nonumber \\&=\alpha ^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x +\alpha \beta \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x\nonumber \\&\quad +\,\alpha ^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x+\lambda \alpha ^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda \alpha ^{2}\beta ^{2}\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x\nonumber \\&\quad -\,\int _{{\mathbb {R}}^{3}}f(x,\alpha u^{+})\alpha u^{+}\hbox {d}x, \end{aligned}$$

(2.1)

and

$$\begin{aligned} g_{2}(\alpha , \beta )&=\langle I_{\lambda }'(\alpha u^{+}+\beta u^{-}), \beta u^{-}\rangle \nonumber \\&=\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}(\alpha u^{+}+\beta u^{-})(-\Delta )^{\frac{s}{2}}(\beta u^{-})\hbox {d}x +\beta ^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{-})}^{2}\hbox {d}x\nonumber \\&\quad +\,\lambda \beta ^{2}\int _{{\mathbb {R}}^{3}}\phi _{\alpha u^{+}+\beta u^{-}}^{t}{(u^{-})}^{2} \hbox {d}x-\int _{{\mathbb {R}}^{3}}f(x,\beta u^{-})\beta u^{-} \hbox {d}x\nonumber \\&=\beta ^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{-}\vert ^{2} \hbox {d}x +\alpha \beta \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x\nonumber \\&\quad +\,\beta ^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{-})}^{2}\hbox {d}x+\lambda \beta ^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{-})}^{2}\hbox {d}x+\lambda \alpha ^{2}\beta ^{2}\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{-})}^{2}\hbox {d}x\nonumber \\&\quad -\,\int _{{\mathbb {R}}^{3}}f(x,\beta u^{-})\beta u^{-}\hbox {d}x. \end{aligned}$$

(2.2)

By $(f_{1})$ and $(f_{3})$, it is easy to see that $g_{1}(\alpha , \alpha )>0$ and $g_{2}(\alpha , \alpha )>0$ for $\alpha >0$ small and $g_{1}(\beta , \beta )<0$ and $g_{2}(\beta , \beta )<0$ for $\beta >0$ large. Thus, there exist $0<r<R$ such that

$$\begin{aligned} g_{1}(r, r)>0, \quad g_{2}(r, r)>0,\quad g_{1}(R, R)<0,\quad g_{2}(R, R)<0. \end{aligned}$$

(2.3)

From (2.1), (2.2) and (2.3), we have

$$\begin{aligned} g_{1}(r, \beta )>0, g_{1}(\beta , R)<0\quad \forall \beta \in [r, R] \end{aligned}$$

and

$$\begin{aligned} g_{2}(\alpha , r)>0, g_{2}(\alpha , R)<0\quad \forall \alpha \in [r, R]. \end{aligned}$$

By virtue of Miranda’s Theorem [15], there exists some point $(\alpha _{u}, \beta _{u})$ with $r<\alpha _{u}, \beta _{u}<R$ such that $g_{1}(\alpha _{u}, \beta _{u})=g_{2}(\alpha _{u}, \beta _{u})=0$. So $\alpha _{u}u^{+}+\beta _{u}u^{-}\in {\mathcal {M}}_{\lambda }$.

Now, we prove the uniqueness of the pair $(\alpha _{u}, \beta _{u})$ and consider two cases.

Case 1$u\in {\mathcal {M}}_{\lambda }$.

If $u\in {\mathcal {M}}_{\lambda }$, then $u^{+}+u^{-}=u\in {\mathcal {M}}_{\lambda }$. It means that

$$\begin{aligned} \langle I_{\lambda }'(u), u^{+}\rangle =\langle I_{\lambda }'(u), u^{-}\rangle =0, \end{aligned}$$

that is

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x +\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\nonumber \\&\quad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x =\int _{{\mathbb {R}}^{3}}f(x, u^{+}) u^{+}\hbox {d}x, \end{aligned}$$

(2.4)

and

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{-}\vert ^{2} \hbox {d}x +\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x +\int _{{\mathbb {R}}^{3}}V(x){(u^{-})}^{2}\hbox {d}x\nonumber \\&\quad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{-})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{-})}^{2}\hbox {d}x =\int _{{\mathbb {R}}^{3}}f(x, u^{-}) u^{-}\hbox {d}x. \end{aligned}$$

(2.5)

We show that $(\alpha _{u}, \beta _{u})=(1, 1)$ is the unique pair of numbers such that $\alpha _{u}u^{+}+\beta _{u}u^{-}\in {\mathcal {M}}_{\lambda }$.

Assume that $({\tilde{\alpha }}_{u}, {\tilde{\beta }}_{u})$ is another pair of numbers such that ${\tilde{\alpha }}_{u}u^{+}+{\tilde{\beta }}_{u}u^{-}\in {\mathcal {M}}_{\lambda }$. By the definition of ${\mathcal {M}}_{\lambda }$, we have

$$\begin{aligned}&{\tilde{\alpha }}_{u}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x +{\tilde{\alpha }}_{u}{\tilde{\beta }}_{u} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+{\tilde{\alpha }}_{u}^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x \nonumber \\&\quad \quad +\,\lambda {\tilde{\alpha }}_{u}^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda {\tilde{\alpha }}_{u}^{2}{\tilde{\beta }}_{u}^{2}\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x =\int _{{\mathbb {R}}^{3}}f(x,{\tilde{\alpha }}_{u} u^{+}){\tilde{\alpha }}_{u} u^{+}\hbox {d}x, \end{aligned}$$

(2.6)

and

$$\begin{aligned}&{\tilde{\beta }}_{u}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{-}\vert ^{2} \hbox {d}x +{\tilde{\alpha }}_{u}{\tilde{\beta }}_{u} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x +{\tilde{\beta }}_{u}^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{-})}^{2}\hbox {d}x\nonumber \\&\quad \quad +\,\lambda {\tilde{\beta }}_{u}^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{-})}^{2}\hbox {d}x+\lambda {\tilde{\alpha }}_{u}^{2}{\tilde{\beta }}_{u}^{2}\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{-})}^{2}\hbox {d}x=\int _{{\mathbb {R}}^{3}}f(x,{\tilde{\beta }}_{u} u^{-}){\tilde{\beta }}_{u}u^{-}\hbox {d}x. \end{aligned}$$

(2.7)

Without loss of generality, we may assume that $0<{\tilde{\alpha }}_{u}\le {\tilde{\beta }}_{u}$. Then, from (2.6), we have

$$\begin{aligned}&{\tilde{\alpha }}_{u}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x +{\tilde{\alpha }}_{u}^{2} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+{\tilde{\alpha }}_{u}^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\\&\quad \quad +\,\lambda {\tilde{\alpha }}_{u}^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda {\tilde{\alpha }}_{u}^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x \le \int _{{\mathbb {R}}^{3}}f(x,{\tilde{\alpha }}_{u} u^{+}){\tilde{\alpha }}_{u} u^{+}\hbox {d}x. \end{aligned}$$

Moreover, dividing the above inequality by ${\tilde{\alpha }}_{u}^{-4}$, we have

$$\begin{aligned}&{\tilde{\alpha }}_{u}^{-2}\Big (\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\Big )\nonumber \\&\quad \quad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x \le \int _{{\mathbb {R}}^{3}}\frac{f(x,{\tilde{\alpha }}_{u} u^{+})}{{\tilde{\alpha }}_{u}^{3}} u^{+}\hbox {d}x. \end{aligned}$$

(2.8)

By (2.8) and (2.4), one has

$$\begin{aligned}&({\tilde{\alpha }}_{u}^{-2}-1)\Big (\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\Big )\nonumber \\&\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \le \int _{{\mathbb {R}}^{3}}\Big (\frac{f(x,{\tilde{\alpha }}_{u} u^{+})}{({\tilde{\alpha }}_{u}u^{+})^{3}}- \frac{f(x, u^{+})}{(u^{+})^{3}}\Big )(u^{+})^{4}\hbox {d}x. \end{aligned}$$

(2.9)

By $(f_{4})$ and (2.9), it implies that $1\le {\tilde{\alpha }}_{u}\le {\tilde{\beta }}_{u}$. By (2.7) and the same method, we have that

$$\begin{aligned}&({\tilde{\beta }}_{u}^{-2}-1)\Big (\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{-}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){(u^{-})}^{2}\hbox {d}x\Big )\nonumber \\&\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \ge \int _{{\mathbb {R}}^{3}}\Big (\frac{f(x,{\tilde{\beta }}_{u} u^{-})}{({\tilde{\beta }}_{u}u^{-})^{3}}- \frac{f(x, u^{-})}{(u^{-})^{3}}\Big )(u^{-})^{4}\hbox {d}x. \end{aligned}$$

It is easy to see that ${\tilde{\beta }}_{u}\le 1$. This together with $1\le {\tilde{\alpha }}_{u}\le {\tilde{\beta }}_{u}$ shows that ${\tilde{\alpha }}_{u}={\tilde{\beta }}_{u}=1$.

Case 2$u\not \in {\mathcal {M}}_{\lambda }$.

If $u\not \in {\mathcal {M}}_{\lambda }$, then there exists a pair of positive numbers $(\alpha _{u}, \beta _{u})$ such that $\alpha _{u}u^{+}+\beta _{u}u^{-}\in {\mathcal {M}}_{\lambda }$. Suppose that there exists another pair of positive numbers $(\alpha '_{u}, \beta '_{u})$ such that $\alpha '_{u}u^{+}+\beta '_{u}u^{-}\in {\mathcal {M}}_{\lambda }$. Set $v:=\alpha _{u}u^{+}+\beta _{u}u^{-}$ and $v':=\alpha '_{u}u^{+}+\beta '_{u}u^{-}$, we have

$$\begin{aligned} \frac{\alpha '_{u}}{\alpha _{u}}v^{+}+\frac{\beta '_{u}}{\beta _{u}}v^{-}=\alpha '_{u}u^{+}+\beta '_{u}u^{-}=v'\in {\mathcal {M}}_{\lambda }. \end{aligned}$$

Since $v\in {\mathcal {M}}_{\lambda }$, we obtain that $\alpha _{u}=\alpha '_{u}$ and $\beta _{u}=\beta '_{u}$, which implies that $(\alpha _{u}, \beta _{u})$ is the unique pair of numbers such that $\alpha _{u}u^{+}+\beta _{u}u^{-}\in {\mathcal {M}}_{\lambda }$. The proof is completed. $\square$

Lemma 2.2

Assume that $(f_{1})-(f_{4})$ and (V) hold for a fixed $u\in H$ with $u^{\pm }\ne 0$. If $\langle I_{\lambda }'(u), u^{+}\rangle \le 0$ and $\langle I_{\lambda }'(u), u^{-}\rangle \le 0$, then there exists a unique pair $(\alpha _{u}, \beta _{u})\in (0, 1]\times (0, 1]$ such that $\langle I_{\lambda }'(\alpha _{u}u^{+}+ \beta _{u}u^{-} ), \alpha _{u}u^{+}\rangle =\langle I_{\lambda }'(\alpha _{u}u^{+}+ \beta _{u}u^{-}), \beta _{u}u^{-}\rangle =0$.

Proof

For $u\in H$ with $u^{\pm }\ne 0$, by Lemma 2.2, we know that there exist $\alpha _{u}$ and $\beta _{u}$ such that $\alpha _{u}u^{+}+\beta _{u}u^{-}\in {\mathcal {M}}_{\lambda }$. Without loss of generality, suppose that $\alpha _{u}\ge \beta _{u}>0$. Moreover, we have

$$\begin{aligned}&\alpha _{u}^{2}\Big (\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\Big )\nonumber \\&\qquad +\,\lambda \alpha _{u}^{4}\Big (\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x\Big )\nonumber \\&\ge \alpha _{u}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x +\alpha _{u}\beta _{u} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\alpha _{u}^{2}\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\nonumber \\&\qquad +\,\lambda \alpha _{u}^{4}\int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda \alpha _{u}^{2}\beta _{u}^{2}\int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x\nonumber \\&=\int _{{\mathbb {R}}^{3}}f(x,\alpha _{u} u^{+})\alpha _{u} u^{+}\hbox {d}x. \end{aligned}$$

(2.10)

Since $\langle I_{\lambda }'(u), u^{+}\rangle \le 0$, it yields that

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\nonumber \\&\quad \quad \quad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{+}}^{t}{(u^{+})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u^{-}}^{t}{(u^{+})}^{2}\hbox {d}x\le \int _{{\mathbb {R}}^{3}}f(x, u^{+})u^{+}\hbox {d}x. \end{aligned}$$

(2.11)

Combine (2.10) and (2.11), we have

$$\begin{aligned}&\left( \frac{1}{\alpha _{u}^{2}}-1\right) \Big (\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u^{+}(-\Delta )^{\frac{s}{2}}u^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){(u^{+})}^{2}\hbox {d}x\Big )\\&\quad \quad \quad \quad \quad \quad \ge \int _{{\mathbb {R}}^{3}}\Big (\frac{f(\alpha _{u}u^{+})}{(\alpha _{u}u^{+})^{3}}-\frac{f(u^{+})}{(u^{+})^{3}}\Big )(u^{+})^{4}\hbox {d}x. \end{aligned}$$

If $\alpha _{u}>1$, the left-hand side of this inequality is negative. But from $(f_{4})$, the right-hand side of this inequality is positive, so have $\alpha _{u}\le 1$. The proof is thus complete. $\square$

Lemma 2.3

For a fixed $u\in H$ with $u^{\pm }\ne 0$, then $(\alpha _{u}, \beta _{u})$ obtained in Lemma 2.1 is the unique maximum point of the function $\kappa :{\mathbb {R}}_{+}\times {\mathbb {R}}_{+}\rightarrow {\mathbb {R}}$ defined as $\kappa (\alpha , \beta )=I_{\lambda }(\alpha u^{+}+\beta u^{-})$.

Proof

From the proof of Lemma 2.1, we know that $(\alpha _{u}, \beta _{u})$ is the unique critical point of $\kappa$ in ${\mathbb {R}}_{+} \times {\mathbb {R}}_{+}$. By $(f_{3})$, we conclude that $\kappa (\alpha , \beta )\rightarrow -\infty$ uniformly as $\vert (\alpha , \beta )\vert \rightarrow \infty$, so it is sufficient to show that a maximum point cannot be achieved on the boundary of $({\mathbb {R}}_{+}, {\mathbb {R}}_{+})$. If we assume that $(0, {\bar{\beta }})$ is a maximum point of $\kappa$ with ${\bar{\beta }}\ge 0$. Then, since

$$\begin{aligned} \kappa (\alpha , {\bar{\beta }})&=I_{\lambda }(\alpha u^{+}+{\bar{\beta }}u^{-})\\&=\frac{1}{2}\int _{{\mathbb {R}}^{3}}\Big (\vert (-\Delta )^{\frac{s}{2}}(\alpha u^{+}+{\bar{\beta }}u^{-})\vert ^{2}+V(x)(\alpha u^{+}+{\bar{\beta }} u^{-})^{2}\Big )\hbox {d}x\\&\quad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{\alpha u^{+}+{\bar{\beta }} u^{-}}^{t}(\alpha u^{+}+{\bar{\beta }} u^{-})^{2}\hbox {d}x-\int _{{\mathbb {R}}^{3}}f(x,\alpha u^{+}+{\bar{\beta }} u^{-})(\alpha u^{+}+{\bar{\beta }} u^{-})\hbox {d}x \end{aligned}$$

is an increasing function with respect to $\alpha$ if $\alpha$ is small enough, the pair $(0, {\bar{\beta }})$ is not a maximum point of $\kappa$ in ${\mathbb {R}}_{+}\times {\mathbb {R}}_{+}$. The proof is now finished. $\square$

By Lemma 2.1, we define the minimization problem

$$\begin{aligned} m_{\lambda }:=\inf \Big \{I_{\lambda }(u): u\in {\mathcal {M}}_{\lambda }\Big \}. \end{aligned}$$

Lemma 2.4

Assume that $(f_{1})-(f_{4})$ and $(V_{1})-(V_{2})$ hold, then $m_{\lambda }>0$ can be achieved for any $\lambda >0$.

Proof

For every $u\in {\mathcal {M}}_{\lambda }$, we have $\langle I'_{\lambda }(u), u\rangle =0$. From $(f_{1})$, $(f_{2})$, for any $\epsilon >0$, there exists $C_{\epsilon }>0$ such that

$$\begin{aligned} \vert f(x, u)u\vert \le \epsilon u^{2}+C_{\epsilon }\vert u\vert ^{p+1} \quad \text{ for } \text{ all }\, u\in {\mathbb {R}}. \end{aligned}$$

(2.12)

By Sobolev embedding theorem, we get

$$\begin{aligned} \Vert u\Vert ^{2}&\le \int _{{\mathbb {R}}^{3}}\Big (\vert (-\Delta )^{\frac{s}{2}}u\vert ^{2}+V(x)u^{2}\Big )\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u}^{t}u^{2}\hbox {d}x=\int _{{\mathbb {R}}^{3}}f(x, u)u\hbox {d}x\nonumber \\&\le \epsilon \int _{{\mathbb {R}}^{3}}\vert u\vert ^{2}\hbox {d}x+C_{\epsilon }\int _{{\mathbb {R}}^{3}}\vert u\vert ^{p+1}\hbox {d}x\nonumber \\&\le C_{2}\epsilon \Vert u\Vert ^{2}+C_{\epsilon }'\Vert u\Vert ^{p+1}. \end{aligned}$$

(2.13)

Pick $\epsilon =\frac{1}{2C_{2}}$. So there exists a constant $\gamma >0$ such that $\Vert u\Vert ^{2}>\gamma$.

By $(f_{4})$, we have

$$\begin{aligned} f(x, u)u-4F(x, u)\ge 0, \end{aligned}$$

then

$$\begin{aligned} I_{\lambda }(u)= I_{\lambda }(u)-\frac{1}{4}\langle I'_{\lambda }(u), u\rangle \ge \frac{\Vert u\Vert ^{2}}{4}\ge \frac{\gamma }{4}. \end{aligned}$$

(2.14)

This implies that $I_{\lambda }(u)$ is coercive in ${\mathcal {M}}_{\lambda }$ and $m_{\lambda }\ge \frac{\gamma }{4}>0$.

Let $\{u_{n}\}_n\subset {\mathcal {M}}_{\lambda }$ be such that $I_{\lambda }(u_{n})\rightarrow m_{\lambda }$. Then $\{u_{n}\}_n$ is bounded in H by (2.14). Using Lemma 1.1, up to a subsequence, we have

$$\begin{aligned}&u_{n}^{\pm }\rightharpoonup u_{\lambda }^{\pm } \quad \text {weakly in}\, H,\nonumber \\&u_{n}^{\pm }\rightarrow u_{\lambda }^{\pm } \quad \text {strongly in}\, L^{q}({\mathbb {R}}^{3}),\quad \text {for}\,\, q\in [2, 2_{s}^{*}),\nonumber \\&u_{n}^{\pm }\rightarrow u_{\lambda }^{\pm } \quad \text {a.e.}\,\, \text {in}\, \, {\mathbb {R}}^{3},\nonumber \\&(-\Delta )^{\frac{s}{2}}u_{n}^{\pm }\rightarrow (-\Delta )^{\frac{s}{2}}u_{\lambda }^{\pm }\quad \text {a.e. in}\, {\mathbb {R}}^{3}. \end{aligned}$$

(2.15)

Moreover, the conditions $(f_{1})$, $(f_{2})$ and Lemma 1.1 imply that

$$\begin{aligned}&\underset{n\rightarrow \infty }{\lim } \int _{{\mathbb {R}}^{3}}f(x, u_{n}^{\pm })u_{n}^{\pm }\hbox {d}x=\int _{{\mathbb {R}}^{3}}f(x, u_{\lambda }^{\pm })u_{\lambda }^{\pm }\hbox {d}x,\quad \,\,\nonumber \\&\quad \underset{n\rightarrow \infty }{\lim } \int _{{\mathbb {R}}^{3}}F(x, u_{n}^{\pm })\hbox {d}x=\int _{{\mathbb {R}}^{3}}F(x, u_{\lambda }^{\pm })\hbox {d}x. \end{aligned}$$

(2.16)

Since $u_{n}\in {\mathcal {M}}_{\lambda }$, we have $\langle I'_{\lambda }(u_{n}), u_{n}^{\pm }\rangle =0$, that is

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u_{n}^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u_{n}^{+}(-\Delta )^{\frac{s}{2}}u_{n}^{-}\hbox {d}x +\int _{{\mathbb {R}}^{3}}V(x){(u_{n}^{+})}^{2}\hbox {d}x\nonumber \\&\qquad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u_{n}^{+}}^{t}{(u_{n}^{+})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u_{n}^{-}}^{t}{(u_{n}^{+})}^{2}\hbox {d}x =\int _{{\mathbb {R}}^{3}}f(x, u_{n}^{+}) u_{n}^{+}\hbox {d}x, \end{aligned}$$

(2.17)

and

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u_{n}^{-}\vert ^{2} \hbox {d}x +\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u_{n}^{+}(-\Delta )^{\frac{s}{2}}u_{n}^{-}\hbox {d}x +\int _{{\mathbb {R}}^{3}}V(x){(u_{n}^{-})}^{2}\hbox {d}x\nonumber \\&\qquad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u_{n}^{-}}^{t}{(u_{n}^{-})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u_{n}^{+}}^{t}{(u_{n}^{-})}^{2}\hbox {d}x =\int _{{\mathbb {R}}^{3}}f(x, u_{n}^{-}) u_{n}^{-}\hbox {d}x. \end{aligned}$$

(2.18)

Similar to (2.12) and (2.13), we also have $\Vert u_{n}^{\pm }\Vert ^{2}\ge \delta$ for all $n\in N$, where $\delta$ is a constant.

Since $u_{n}\in {\mathcal {M}}_{\lambda }$, by (2.17) and (2.18) again, we have

$$\begin{aligned} \delta \le \Vert u_{n}^{\pm }\Vert ^{2}< \int _{{\mathbb {R}}^{3}}f(x, u_{n}^{\pm })u_{n}^{\pm } \hbox {d}x&\le \epsilon \int _{{\mathbb {R}}^{3}}\vert u_{n}^{\pm }\vert ^{2}\hbox {d}x+ C_{\epsilon }\int _{{\mathbb {R}}^{3}}\vert u_{n}^{\pm }\vert ^{p+1}\hbox {d}x\\&\le \frac{\epsilon }{V_{0}}\int _{{\mathbb {R}}^{3}}\vert u_{n}^{\pm }\vert ^{2}\hbox {d}x+ C_{\epsilon }\int _{{\mathbb {R}}^{3}}\vert u_{n}^{\pm }\vert ^{p+1}\hbox {d}x. \end{aligned}$$

Using the boundedness of $\{u_{n}\}_n$, there is $C_{2}>0$ such that

$$\begin{aligned} \delta \le \epsilon C_{2}+C_{\epsilon }\int _{{\mathbb {R}}^{3}}\vert u_{n}^{\pm }\vert ^{p+1}\hbox {d}x. \end{aligned}$$

Choosing $\epsilon =\delta /(2C_{2})$, we get

$$\begin{aligned} \int _{{\mathbb {R}}^{3}}\vert u_{n}^{\pm }\vert ^{p+1}\hbox {d}x\ge \frac{\delta }{2{\bar{C}}}. \end{aligned}$$

(2.19)

where ${\bar{C}}$ is a positive constant, in fact, ${\bar{C}}=C_{\frac{\delta }{2C_{2}}}$.

By (2.19) and Lemma 1.1 (ii), we get

$$\begin{aligned} \int _{{\mathbb {R}}^{3}}\vert u_{\lambda }^{\pm }\vert ^{p+1}\hbox {d}x\ge \frac{\delta }{2{\bar{C}}}. \end{aligned}$$

Thus, $u_{\lambda }^{\pm }\ne 0$. From Lemma 2.1, there exists $\alpha _{u_{\lambda }}$, $\beta _{u_{\lambda }}>0$ such that

$$\begin{aligned} {\bar{u}}_{\lambda }:=\alpha _{u_{\lambda }}u_{\lambda }^{+}+\beta _{u_{\lambda }}u_{\lambda }^{-}\in {\mathcal {M}}_{\lambda }. \end{aligned}$$

Now, we show that $\alpha _{u_{\lambda }}$, $\beta _{u_{\lambda }}\le 1$. By (2.15), (2.17), the weak semicontinuity of norm, Fatou’s Lemma and Lemma 1.2, we have

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}u_{\lambda }^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u_{\lambda }^{+}(-\Delta )^{\frac{s}{2}}u_{\lambda }^{-}\hbox {d}x +\int _{{\mathbb {R}}^{3}}V(x){(u_{\lambda }^{+})}^{2}\hbox {d}x\nonumber \\&\quad \quad +\,\lambda \int _{{\mathbb {R}}^{3}}\phi _{u_{\lambda }^{+}}^{t}{(u_{\lambda }^{+})}^{2}\hbox {d}x+\lambda \int _{{\mathbb {R}}^{3}}\phi _{u_{\lambda }^{-}}^{t}{(u_{\lambda }^{+})}^{2}\hbox {d}x \le \int _{{\mathbb {R}}^{3}}f(x, u_{\lambda }^{+}) u_{\lambda }^{+}\hbox {d}x. \end{aligned}$$

(2.20)

From (2.20) and Lemma 2.2, we have $\alpha _{u_{\lambda }}\le 1$. Similarly, $\beta _{u_{\lambda }}\le 1$. The condition $(f_{4})$ implies that $H(u):=uf(x, u)-4F(x, u)$ is a nonnegative function, increasing in $\vert u\vert$, so we have

$$\begin{aligned} m_{\lambda }\le&I_{\lambda }({\bar{u}}_{\lambda })=I_{\lambda }({\bar{u}}_{\lambda })-\frac{1}{4}\langle I'_{\lambda }({\bar{u}}_{\lambda }), {\bar{u}}_{\lambda }\rangle \\&=\frac{1}{4}\Vert {\bar{u}}_{\lambda }\Vert ^{2}+\frac{1}{4}\int _{{\mathbb {R}}^{3}}\Big (f({\bar{u}}_{\lambda }){\bar{u}}_{\lambda }-4F({\bar{u}}_{\lambda })\Big )\hbox {d}x\\&= \frac{1}{4}\Vert \alpha _{u_{\lambda }}u_{\lambda }^{+}\Vert ^{2}+\frac{1}{4}\Vert \beta _{u_{\lambda }}u_{\lambda }^{-}\Vert ^{2}\\&\quad +\,\frac{\alpha _{u_{\lambda }} \beta _{u_{\lambda }}}{2}\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u_{\lambda }^{+}(-\Delta )^{\frac{s}{2}}u_{\lambda }^{-}\hbox {d}x\\&\quad +\,\frac{1}{4}\int _{{\mathbb {R}}^{3}}\Big (f(\alpha _{u_{\lambda }}u_{\lambda }^{+})\alpha _{u_{\lambda }}u_{\lambda }^{+} -4F(x, \alpha _{u_{\lambda }}u_{\lambda }^{+})\Big )\hbox {d}x\\&\quad +\,\frac{1}{4}\int _{{\mathbb {R}}^{3}}\Big (f(x, \beta _{u_{\lambda }}u_{\lambda }^{-})\beta _{u_{\lambda }}u_{\lambda }^{-} -4F(x, \beta _{u_{\lambda }}u_{\lambda }^{-})\Big )\hbox {d}x\\&\le \, \frac{1}{4}\Vert u_{\lambda }^{+}\Vert ^{2}+\frac{1}{4}\Vert u_{\lambda }^{-}\Vert ^{2}+\frac{1}{2}\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}u_{\lambda }^{+}(-\Delta )^{\frac{s}{2}}u_{\lambda }^{-}\hbox {d}x\\&\quad +\,\frac{1}{4}\int _{{\mathbb {R}}^{3}}\Big (f(x, u_{\lambda }^{+})u_{\lambda }^{+} -4F(x, u_{\lambda }^{+})\Big )\hbox {d}x+\frac{1}{4}\int _{{\mathbb {R}}^{3}}\Big (f(x, u_{\lambda }^{-})u_{\lambda }^{-} -4F(x, u_{\lambda }^{-})\Big )\hbox {d}x\\&\le \, \underset{n\rightarrow \infty }{\lim \inf }\Big [I_{\lambda }(u_{n})-\frac{1}{4}\langle I'_{\lambda }(u_{n}), u_{n}\rangle \Big ]=m_{\lambda }. \end{aligned}$$

We then conclude that $\alpha _{u_{\lambda }}=\beta _{u_{\lambda }}=1$. Thus, ${\bar{u}}_{\lambda }=u_{\lambda }$ and $I_{\lambda }(u_{\lambda })=m_{\lambda }$. $\square$

3 Proof of main results

In this section, we are devoted to proving our main results.

Proof of Theorem 1.1

We firstly prove that the minimizer $u_{\lambda }$ for the minimization problem is indeed a sign-changing solution of problem (1.4), that is, $I'_{\lambda }(u_{\lambda })=0$. For it, we will use the quantitative deformation lemma.

It is clear that $I'_{\lambda }(u_{\lambda })u_{\lambda }^{+}=0=I'_{\lambda }(u_{\lambda })u_{\lambda }^{-}$. From Lemma 2.2, for any $(\alpha , \beta )\in {\mathbb {R}}_{+}\times {\mathbb {R}}_{+}$ and $(\alpha , \beta )\ne (1, 1)$,

$$\begin{aligned} I_{\lambda }(\alpha u_{\lambda }^{+}+\beta u_{\lambda }^{-})<I_{\lambda }( u_{\lambda }^{+}+u_{\lambda }^{-})=m_{\lambda }. \end{aligned}$$

If $I'_{\lambda }(u_{\lambda })\ne 0$, then there exist $\delta >0$ and $\kappa >0$ such that

$$\begin{aligned} \Vert I'_{\lambda }(v)\Vert \ge \kappa \quad \text {for all} \,\Vert v-u_{\lambda }\Vert \le 3\delta . \end{aligned}$$

Let $D:=(\frac{1}{2}, \frac{3}{2})\times (\frac{1}{2}, \frac{3}{2})$ and $g(\alpha , \beta ):=\alpha u_{\lambda }^{+}+\beta u_{\lambda }^{-}$. From Lemma 2.3, we also have

$$\begin{aligned} {\bar{m}}_{\lambda }:=\underset{\partial D}{\max } (I_{\lambda }\circ g)<m_{\lambda }. \end{aligned}$$

For $\epsilon :=\min \{(m_{\lambda }-{\bar{m}}_{\lambda })/2, \kappa \delta /8\}$ and $S:=B(u_{\lambda }, \delta )$, there is a deformation $\eta$ such that

(a):

$\eta (1, u)=u$ if $u\not \in I_{\lambda }^{-1}([m_{\lambda }-2\epsilon , m_{\lambda }+2\epsilon ])\cap S_{2\delta }$;

(b):

$\eta (1, I_{\lambda }^{m_{\lambda }+\epsilon }\cap S)\subset I_{\lambda }^{m_{\lambda }-\epsilon }$;

(c):

$I_{\lambda }(\eta (1, u)))\le I_{\lambda }(u)$ for all $u\in H$.

See [24] for more details. It is clear that

$$\begin{aligned} \underset{(\alpha , \beta )\in {\bar{D}}}{\max }I_{\lambda }(\eta (1, g(\alpha , \beta ))))<m_{\lambda }. \end{aligned}$$

Now we prove that $\eta (1, g(D))\cap {\mathcal {M}}_{\lambda }\ne \emptyset$ which contradicts the definition of $m_{\lambda }$. Let us define $h(\alpha , \beta )=\eta (1, g(\alpha , \beta )))$ and

$$\begin{aligned}&\Psi _{0}(\alpha , \beta ):=\Big (I'_{\lambda }(g(\alpha , \beta ))u_{\lambda }^{+}, I'_{\lambda }(g(\alpha , \beta ))u_{\lambda }^{-}\Big )\\&\quad =\Big (I'_{\lambda }(\alpha u_{\lambda }^{+}+\beta u_{\lambda }^{-})u_{\lambda }^{+}, I'_{\lambda }(\alpha u_{\lambda }^{+}+\beta u_{\lambda }^{-})u_{\lambda }^{-}\Big ),\\&\quad \Psi _{1}(\alpha , \beta ):=\Big (\frac{1}{\alpha }I'_{\lambda }(h(\alpha , \beta ))h^{+}(\alpha , \beta ), \frac{1}{\beta }I'_{\lambda }(h(\alpha , \beta ))h^{-}(\alpha , \beta )\Big ). \end{aligned}$$

Lemma 2.1 and the degree theory imply that $\deg (\Psi _{0}, D, 0)=1$. It follows from that $g=h$ on $\partial D$. Consequently, we obtain

$$\begin{aligned} \deg (\Psi _{1}, D, 0)=\deg (\Psi _{0}, D, 0)=1. \end{aligned}$$

Thus, $\Psi _{1}(\alpha _{0}, \beta _{0})=0$ for some $(\alpha _{0}, \beta _{0})\in D$, so that

$$\begin{aligned} \eta (1, g(\alpha _{0}, \beta _{0})))=h(\alpha _{0}, \beta _{0})\in {\mathcal {M}}_{\lambda }, \end{aligned}$$

which is a contradiction. From this, $u_{\lambda }$ is a critical point of $I_{\lambda }$, and moreover, it is a sign-changing solution for problem (1.4).

Now we prove that $u_{\lambda }$ has exactly two nodal domains. By contradiction, we assume that $u_{\lambda }$ has at least three nodal domains $\Omega _{1}$, $\Omega _{2}$, $\Omega _{3}$. Without loss generality, we may assume that $u_{\lambda }\ge 0$ a.e. in $\Omega _{1}$, $u_{\lambda }\le 0$ a.e. in $\Omega _{2}$. Set

$$\begin{aligned} {u_{\lambda _{i}}}:=\chi _{\Omega _{i}}u_{\lambda }, \quad i=1, 2, 3, \end{aligned}$$

where

$$\begin{aligned} \chi _{\Omega _{i}}=\left\{ \begin{array}{ll} 1&{}\quad x\in \Omega _{i},\\ 0&{}\quad x\in {\mathbb {R}}^{3}\setminus \Omega _{i}.\\ \end{array} \right. \end{aligned}$$

So $\text {supp}(u_{\lambda _{1}})\cap \text {supp}(u_{\lambda _{2}})=\emptyset$, ${u_{\lambda _{i}}}\ne 0$ and $\langle I'(u_{\lambda }), {u_{\lambda _{i}}}\rangle =0$ for $i=1, 2, 3$. Assume that $v:=u_{\lambda _{1}}+u_{\lambda _{2}}$, then $v^{+}=u_{\lambda _{1}}$ and $v^{-}=u_{\lambda _{2}}$, i.e., $v^{\pm }\ne 0$. By Lemma 2.1, there is a unique pair $(\alpha _{v}, \beta _{v})$ of positive numbers such that

$$\begin{aligned} \alpha _{v}v^{+}+\beta _{v}v^{+}\in {\mathcal {M}}_{\lambda }, \end{aligned}$$

so we have

$$\begin{aligned} I(\alpha _{v}{u_{\lambda _{1}}}+\beta _{v}{u_{\lambda _{2}}})\ge m_{\lambda }. \end{aligned}$$

From $\langle I'(u_{\lambda }), {u_{\lambda _{i}}}\rangle =0$ for $i=1, 2, 3$, we have

$$\begin{aligned} \langle I'(v), v^{\pm }\rangle <0. \end{aligned}$$

By Lemma 2.2, we know that $(\alpha _{v}, \beta _{v})\in (0, 1]\times (0, 1]$. Since

$$\begin{aligned} 0&=\frac{1}{4}\langle I'_{\lambda }(u_{\lambda }), {u_{\lambda _{3}}}\rangle =\frac{1}{4}\Vert {u_{\lambda _{3}}}\Vert ^{2}+\frac{1}{4} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x\\&\quad +\, \frac{1}{4} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x \\&\quad +\,\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{1}}}{u_{\lambda _{3}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{2}}}{u_{\lambda _{3}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{3}}}{u_{\lambda _{3}}}^{2}\hbox {d}x\\&\quad -\,\frac{1}{4}\int _{{\mathbb {R}}^{3}}f(x, {u_{\lambda _{3}}}){u_{\lambda _{3}}}\hbox {d}x \\&\le \,\frac{1}{4}\Vert {u_{\lambda _{3}}}\Vert ^{2}+\frac{1}{4} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x+ \frac{1}{4} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x \\&\quad +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{1}}}{u_{\lambda _{3}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{2}}}{u_{\lambda _{3}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{3}}}{u_{\lambda _{3}}}^{2}\hbox {d}x-\int _{{\mathbb {R}}^{3}}F(x, {u_{\lambda _{3}}})\hbox {d}x\\&<\, I_{\lambda }({u_{\lambda _{3}}})++\frac{1}{4} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x+ \frac{1}{4} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x\\&\quad +\,\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{1}}}{u_{\lambda _{3}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{u_{\lambda _{2}}}{u_{\lambda _{3}}}^{2}\hbox {d}x. \end{aligned}$$

From $(f_{4})$, we have

$$\begin{aligned} m_{\lambda }&\le I_{\lambda }(\alpha _{v}{u_{\lambda _{1}}}+\beta _{v}{u_{\lambda _{2}}})\\&= I_{\lambda }(\alpha _{v}{u_{\lambda _{1}}}+\beta _{v}{u_{\lambda _{2}}})-\frac{1}{4}\langle I'_{\lambda }(\alpha _{v}{u_{\lambda _{1}}}+\beta _{v}{u_{\lambda _{2}}}), \alpha _{v}{u_{\lambda _{1}}}+\beta _{v}{u_{\lambda _{2}}}\rangle \\&=\frac{\Vert \alpha _{v}{u_{\lambda _{1}}}+\beta _{v} {u_{\lambda _{2}}}\Vert ^{2}}{4}+\int _{{\mathbb {R}}^{3}}\Big (\frac{1}{4}f(x, \alpha _{v}{u_{\lambda _{1}}})\alpha _{v}{u_{\lambda _{1}}}-F(x, \alpha _{v}{u_{\lambda _{1}}})\Big )\hbox {d}x\\&\quad +\int _{{\mathbb {R}}^{3}}\Big (\frac{1}{4}f(x, \beta _{v}{u_{\lambda _{2}}})\beta _{v}{u_{\lambda _{2}}}-F(x, \beta _{v}{u_{\lambda _{2}}})\Big )\hbox {d}x\\&\le \frac{\Vert {u_{\lambda _{1}}}\Vert ^{2}+\Vert {u_{\lambda _{2}}}\Vert ^{2}}{4}+\frac{1}{2}\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}\hbox {d}x\\&\quad +\int _{{\mathbb {R}}^{3}}\Big (\frac{1}{4}f(x,{u_{\lambda _{1}}}){u_{\lambda _{1}}}-F(x,{u_{\lambda _{1}}})\Big )\hbox {d}x+\int _{{\mathbb {R}}^{3}}\Big (\frac{1}{4}f(x, {u_{\lambda _{2}}}){u_{\lambda _{2}}}-F(x, {u_{\lambda _{2}}})\Big )\hbox {d}x\\&=I_{\lambda }({u_{\lambda _{1}}})+I_{\lambda }({u_{\lambda _{2}}}) +\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}\hbox {d}x +\frac{1}{4}\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}\hbox {d}x\\&\quad +\,\frac{1}{4}\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}\hbox {d}x+ \frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{2}}}}{u_{\lambda _{1}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{3}}}}{u_{\lambda _{1}}}^{2}\hbox {d}x\\&\quad +\,\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{1}}}}{u_{\lambda _{2}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{3}}}}{u_{\lambda _{2}}}^{2}\hbox {d}x\\&<\,I_{\lambda }({u_{\lambda _{1}}})+I_{\lambda }({u_{\lambda _{2}}})+I_{\lambda }({u_{\lambda _{3}}})+ \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}\hbox {d}x\\&\quad +\,\int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{1}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x+ \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{2}}}(-\Delta )^{\frac{s}{2}}{u_{\lambda _{3}}}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{2}}}}{u_{\lambda _{1}}}^{2}\hbox {d}x\\&\quad +\,\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{3}}}}{u_{\lambda _{1}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{1}}}}{u_{\lambda _{2}}}^{2}\hbox {d}x +\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{3}}}}{u_{\lambda _{2}}}^{2}\hbox {d}x\\&\quad +\,\frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{1}}}}{u_{\lambda _{3}}}^{2}\hbox {d}x+ \frac{\lambda }{4}\int _{{\mathbb {R}}^{3}}\phi ^{t}_{{u_{\lambda _{2}}}}{u_{\lambda _{3}}}^{2}\hbox {d}x\\&=I_{\lambda }(u_{\lambda })=m_{\lambda }, \end{aligned}$$

which is impossible, so $u_{\lambda }$ has exactly two nodal domains. $\square$

Proof of Theorem 1.2

Similar to the proof of Lemma 2.4, for each $\lambda >0$, we can get a $v_{\lambda }\in {\mathcal {N}}_{\lambda }$ such that $I_{\lambda }(v_{\lambda })=c_{\lambda }>0$, where ${\mathcal {N}}_{\lambda }$ and $c_{\lambda }$ are defined by (1.5) and (1.6), respectively. Moreover, the critical points of $I_{\lambda }$ on ${\mathcal {N}}_{\lambda }$ are the critical points of $I_{\lambda }$ in H. Thus, $v_{\lambda }$ is a ground state solution of problem (1.4).

From Theorem 1.1, we know that problem (1.4) has a least energy sign-changing solution $u_{\lambda }$ which changes sign only once. Suppose that $u_{\lambda }=u_{\lambda }^{+}+u_{\lambda }^{-}$. As the proof of Step 1 in Lemma 2.1, there is a unique $\alpha _{u_{\lambda }^{+}}>0$ such that

$$\begin{aligned} \alpha _{u_{\lambda }^{+}}u_{\lambda }^{+}\in {\mathcal {N}}_{\lambda }. \end{aligned}$$

Similarly, there exists a unique $\beta _{u_{\lambda }^{-}}>0$, such that

$$\begin{aligned} \beta _{u_{\lambda }^{-}}u_{\lambda }^{-}\in {\mathcal {N}}_{\lambda }. \end{aligned}$$

Moreover, Lemma 2.2 implies that $\alpha _{u_{\lambda }^{+}}, \beta _{u_{\lambda }^{-}}\in (0, 1]$. Therefore, by Lemma 2.3, we obtain that

$$\begin{aligned} 2c_{\lambda }\le I_{\lambda }(\alpha _{u_{\lambda }^{+}}u_{\lambda }^{+})+I_{\lambda }(\beta _{u_{\lambda }^{-}}u_{\lambda }^{-}) \le I_{\lambda }(\alpha _{u_{\lambda }^{+}}u_{\lambda }^{+}+\beta _{u_{\lambda }^{-}}u_{\lambda }^{-})\le I_{\lambda }(u_{\lambda }^{+}+u_{\lambda }^{-})=m_{\lambda } \end{aligned}$$

that is $I_{\lambda }(u_{\lambda }) \ge 2c_{\lambda }$. It follows that $c_{\lambda }>0$ which cannot be achieved by a sign-changing function. This completes the proof. $\square$

Now, we prove Theorem 1.3. In the following, we regard $\lambda >0$ as a parameter in problem (1.4). We shall study the convergence property of $u_{\lambda }$ as $\lambda \searrow 0$.

Proof of Theorem 1.3

For any $\lambda >0$, let $u_{\lambda }\in H$ be the least energy sign-changing solution of problem (1.1) obtained in Theorem 1.1, which has exactly two nodal domains.

Step 1 We show that, for any sequence $\{\lambda _{n}\}_n$ with $\lambda _{n} \searrow 0$ as $n\rightarrow \infty$, $\{{u_{\lambda _{n}}}\}_n$ is bounded in H.

Choose a nonzero function $\varphi \in C_{0}^{\infty }({\mathbb {R}}^{3})$ with $\varphi ^{\pm }\ne 0$. By $(f_{3})$ and $(f_{4})$, for any $\lambda \in [0, 1]$, there exists a pair $(\theta _{1}, \theta _{2})\in ({\mathbb {R}}_{+}\times {\mathbb {R}}_{+})$, which does not depend on $\lambda$, such that

$$\begin{aligned} \langle I'_{\lambda }(\theta _{1}\varphi ^{+}+\theta _{2}\varphi ^{-}), \theta _{1}\varphi ^{+}\rangle<0 \quad \text {that}\quad \langle I'_{\lambda }(\theta _{1}\varphi ^{+}+\theta _{2}\varphi ^{-}), \theta _{2}\varphi ^{-}\rangle <0. \end{aligned}$$

Then, in view of Lemmas 2.1 and Lemma 2.2, for any $\lambda \in [0, 1]$, there is a unique pair $(\alpha _{\varphi }(\lambda ), \beta _{\varphi }(\lambda ) )\in (0, 1]\times (0, 1]$ such that ${\bar{\varphi }}:=\alpha _{\varphi }(\lambda )\theta _{1}\varphi ^{+}+\beta _{\varphi }(\lambda )\theta _{2}\varphi ^{-}\in {\mathcal {M}}_{\lambda }$. Thus, for all $\lambda \in [0, 1]$, we have

$$\begin{aligned} I_{\lambda }(u_{\lambda })&\le I_{\lambda }({\bar{\varphi }}) = I_{\lambda }({\bar{\varphi }})-\frac{1}{4}\langle I'_{\lambda }({\bar{\varphi }}), {\bar{\varphi }}\rangle \\&=\frac{\Vert {\bar{\varphi }}\Vert ^{2}}{4}+ \int _{{\mathbb {R}}^{3}}\Big (\frac{1}{4}f(x, {\bar{\varphi }}){\bar{\varphi }}- F(x, {\bar{\varphi }}) \Big )\hbox {d}x\\&\le \frac{\Vert {\bar{\varphi }}\Vert ^{2}}{4}+ \int _{{\mathbb {R}}^{3}}\Big (C_{3}\vert {\bar{\varphi }}\vert ^{2}+ C_{4}\vert {\bar{\varphi }}\vert ^{p+1}\Big )\hbox {d}x\\&\le \frac{\Vert \theta _{1}\varphi ^{+}\Vert ^{2}}{4}+\frac{\Vert \theta _{2}\varphi ^{-}\Vert ^{2}}{4}+ \frac{1}{2} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}({\theta _{1}\varphi ^{+}})(-\Delta )^{\frac{s}{2}}({\theta _{2}\varphi ^{-}})\hbox {d}x\\&\quad +\int _{{\mathbb {R}}^{3}}\Big (C_{3}\vert \theta _{1}\varphi ^{+}\vert ^{2}+ C_{4}\vert \theta _{1}\varphi ^{+}\vert ^{p+1}+C_{3}\vert \theta _{2}\varphi ^{-}\vert ^{2}+ C_{4}\vert \theta _{2}\varphi ^{-}\vert ^{p+1}\Big )\hbox {d}x\\&=C_{0}. \end{aligned}$$

Moreover, for n large enough, we obtain

$$\begin{aligned} C_{0}+1\ge I_{\lambda _{n}}(u_{\lambda _{n}})=I_{\lambda _{n}}(u_{\lambda _{n}})-\frac{1}{4}\langle I'_{\lambda _{n}}(u_{\lambda _{n}}), u_{\lambda _{n}}\rangle \ge \frac{1}{4}\Vert u_{\lambda _{n}}\Vert ^{2}. \end{aligned}$$

So $\{{u_{\lambda }}_{n}\}_n$ is bounded in H.

Step 2 The problem has a sign-changing solution $u_{0}$.

By step 1 and Lemma 1.1, there exists a subsequence of $\{\lambda _{n}\}_n$, up to a subsequence and $u_{0}\in H$ such that

$$\begin{aligned}&u_{\lambda _{n}}\rightharpoonup u_{0}\quad \text {weakly in}\quad H,\nonumber \\&u_{\lambda _{n}}\rightarrow u_{0}\quad \text {strongly in}\, L^{q}({\mathbb {R}}^{3})\quad \text {for}\, q\in [2, 2_{s}^{*}),\nonumber \\&u_{\lambda _{n}}\rightarrow u_{0}\quad \text {a.e. in}\,\, {\mathbb {R}}^{3}. \end{aligned}$$

(3.1)

Since $u_{\lambda _{n}}$ is the least energy sign-changing solution of (1.4) with $\lambda =\lambda _{n}$, then we have

$$\begin{aligned} \int _{{\mathbb {R}}^{3}}\Big ((-\Delta )^{\frac{s}{2}}u_{\lambda _{n}}(-\Delta )^{\frac{s}{2}}v+V(x)u_{\lambda _{n}}v\Big )\hbox {d}x+\lambda _{n}\int _{{\mathbb {R}}^{3}}\phi _{u_{\lambda _{n}}}^{t}u_{\lambda _{n}}v \hbox {d}x=\int _{{\mathbb {R}}^{3}}f(x,u_{\lambda _{n}})v \hbox {d}x. \end{aligned}$$

for all $v\in C_{0}^{\infty }({\mathbb {R}}^{3})$. From (3.1), we get that

$$\begin{aligned} \int _{{\mathbb {R}}^{3}}\Big ((-\Delta )^{\frac{s}{2}}u_{0}(-\Delta )^{\frac{s}{2}}v+V(x)u_{0}v\Big )\hbox {d}x=\int _{{\mathbb {R}}^{3}}f(x,u_{0})v \hbox {d}x, \end{aligned}$$

for all $v\in C_{0}^{\infty }({\mathbb {R}}^{3})$. So $u_{0}$ is a weak solution of (1.7). From a similar argument of the proof in Lemma 2.4, we know that $u_{0}^{\pm }\ne 0$.

Step 3 The problem (1.7) has a least energy sign-changing solution $v_{0}$, and there is a unique pair $(\alpha _{\lambda _{n}}, \beta _{\lambda _{n}} )\in {\mathbb {R}}^{+}\times {\mathbb {R}}^{+}$ such that $\alpha _{\lambda _{n}}{v_{0}}^{+}+ \beta _{\lambda _{n}}{v_{0}}^{-} \in {\mathcal {M}}_{\lambda }$. Moreover, $(\alpha _{\lambda _{n}}, \beta _{\lambda _{n}})\rightarrow (1, 1)$ as $n\rightarrow \infty$.

Via a similar argument in the proof of Theorem 1.1, there is a least energy sign-changing solution $v_{0}$ for problem (1.7) with two nodal domain, so we have

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}{v_{0}}^{+}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{v_{0}}^{+}(-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\hbox {d}x+\int _{{\mathbb {R}}^{3}}V(x){({v_{0}}^{+})}^{2}\hbox {d}x\nonumber \\&\quad =\int _{{\mathbb {R}}^{3}}f(x,{v_{0}}^{+}){v_{0}}^{+}\hbox {d}x, \end{aligned}$$

(3.2)

and

$$\begin{aligned}&\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\vert ^{2} \hbox {d}x + \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{v_{0}}^{+}(-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\hbox {d}x +\int _{{\mathbb {R}}^{3}}V(x){(v_{0}^{-})}^{2}\hbox {d}x\nonumber \\&\quad =\int _{{\mathbb {R}}^{3}}f(x,{v_{0}}^{-}){v_{0}}^{-}\hbox {d}x. \end{aligned}$$

(3.3)

By Lemma 2.1, there exits an unique pair of $(\alpha _{\lambda _{n}}, \beta _{\lambda _{n}} )$ such that $\alpha _{\lambda _{n}}{v_{0}}^{+}+ \beta _{\lambda _{n}}{v_{0}}^{-} \in {\mathcal {M}}_{\lambda }$. So we have

$$\begin{aligned}&\alpha _{\lambda _{n}}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}{v_{0}}^{+}\vert ^{2} \hbox {d}x +\alpha _{\lambda _{n}}\beta _{\lambda _{n}} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{v_{0}}^{+}(-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\hbox {d}x\nonumber \\&\qquad +\,\alpha _{\lambda _{n}}^{2}\int _{{\mathbb {R}}^{3}}V(x){({v_{0}}^{+})}^{2}\hbox {d}x \nonumber \\&\qquad +\,\lambda _{n} \alpha _{\lambda _{n}}^{4}\int _{{\mathbb {R}}^{3}}\phi _{{v_{0}}^{+}}^{t}{({v_{0}}^{+})}^{2}\hbox {d}x+\lambda _{n} \alpha _{\lambda _{n}}^{2}\beta _{\lambda _{n}}^{2}\int _{{\mathbb {R}}^{3}}\phi _{{v_{0}}^{-}}^{t}{({v_{0}}^{+})}^{2}\hbox {d}x\nonumber \\&\quad =\int _{{\mathbb {R}}^{3}}f(x,\alpha _{\lambda _{n}}{v_{0}}^{+})\alpha _{\lambda _{n}} {v_{0}}^{+}\hbox {d}x, \end{aligned}$$

(3.4)

and

$$\begin{aligned}&\beta _{\lambda _{n}}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\vert ^{2} \hbox {d}x +\alpha _{\lambda _{n}}\beta _{\lambda _{n}} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{v_{0}}^{+}(-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\hbox {d}x +\beta _{\lambda _{n}}^{2}\int _{{\mathbb {R}}^{3}}V(x){(v_{0}^{-})}^{2}\hbox {d}x\nonumber \\&\quad \quad +\,\lambda _{n} \beta _{\lambda _{n}}^{4}\int _{{\mathbb {R}}^{3}}\phi _{{v_{0}}^{-}}^{t}{{(v_{0}}^{-})}^{2}\hbox {d}x+\lambda _{n} \alpha _{\lambda _{n}}^{2}\beta _{\lambda _{n}}^{2}\int _{{\mathbb {R}}^{3}}\phi _{{v_{0}}^{+}}^{t}{({v_{0}}^{-})}^{2}\hbox {d}x\nonumber \\&\quad =\int _{{\mathbb {R}}^{3}}f(x,\beta _{\lambda _{n}} {v_{0}}^{-})\beta _{\lambda _{n}}{v_{0}}^{-}\hbox {d}x. \end{aligned}$$

(3.5)

From $(f_{3})$ and $\lambda _{n}\rightarrow 0$ as $n\rightarrow \infty$, we get that the sequences $\{\alpha _{\lambda _{n}}\}$ and $\{\beta _{\lambda _{n}}\}$ are bounded. Assume that $\alpha _{\lambda _{n}}\rightarrow \alpha _{0}$ and $\beta _{\lambda _{n}}\rightarrow \beta _{0}$ as $n\rightarrow \infty$. From (2.16), (3.4) and (3.5), we have

$$\begin{aligned}&\alpha _{0}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}{v_{0}}^{+}\vert ^{2} \hbox {d}x +\alpha _{0}\beta _{0} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{v_{0}}^{+}(-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\hbox {d}x+\alpha _{0}^{2}\int _{{\mathbb {R}}^{3}}V(x){({v_{0}}^{+})}^{2}\hbox {d}x \nonumber \\&\quad \quad =\int _{{\mathbb {R}}^{3}}f(x,\alpha _{0}{v_{0}}^{+})\alpha _{0} {v_{0}}^{+}\hbox {d}x, \end{aligned}$$

(3.6)

and

$$\begin{aligned}&\beta _{0}^{2}\int _{{\mathbb {R}}^{3}}\vert (-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\vert ^{2} \hbox {d}x +\alpha _{0}\beta _{0} \int _{{\mathbb {R}}^{3}}(-\Delta )^{\frac{s}{2}}{v_{0}}^{+}(-\Delta )^{\frac{s}{2}}{v_{0}}^{-}\hbox {d}x +\beta _{0}^{2}\int _{{\mathbb {R}}^{3}}V(x){(v_{0}^{-})}^{2}\hbox {d}x\nonumber \\&\quad \quad =\int _{{\mathbb {R}}^{3}}f(x,\beta _{0} {v_{0}}^{-})\beta _{0}{v_{0}}^{-}\hbox {d}x. \end{aligned}$$

(3.7)

Moreover, by $(f_{3})$ and $(f_{4})$, we know that $\frac{f(x, s)}{\vert s\vert ^{3}}$ is nondecreasing in $\vert s\vert$. So from (3.2), (3.3), (3.6), (3.7), we obtain that $(\alpha _{0}, \beta _{0})=(1, 1)$.

Now, we complete the proof of Theorem 1.3. We only need to show that $u_{0}$ obtained in step 2 is a least energy sign-changing solution of problem (1.7). By Lemma 2.3, we have

$$\begin{aligned} I_{0}(v_{0})\le I_{0}(u_{0})&\le \underset{n\rightarrow \infty }{\lim }I_{\lambda _{n}}(u_{\lambda _{n}}) =\underset{n\rightarrow \infty }{\lim }I_{\lambda _{n}}(u_{\lambda _{n}}^{+}+u_{\lambda _{n}}^{-})\\&\le \underset{n\rightarrow \infty }{\lim }I_{\lambda _{n}}(\alpha _{\lambda _{n}}{v_{0}}^{+}+\beta _{\lambda _{n}}{v_{0}}^{-})\\&=I_{0}(v_{0}). \end{aligned}$$

This show that $u_{0}$ is a least energy sign-changing solution of problem (1.7) which has precisely two nodal domains. We complete the proof of Theorem 1.3. $\square$