Normalized solutions to the fractional Schrödinger equations with combined nonlinearities

Abstract

We study the normalized solutions of the fractional nonlinear Schrödinger equations with combined nonlinearities

$$\begin{aligned} (-\Delta )^s u=\lambda u+\mu |u|^{q-2}u+|u|^{p-2}u \quad \text{ in }~{\mathbb {R}}^N, \end{aligned}$$

and we look for solutions which satisfy prescribed mass

$$\begin{aligned} \int _{\mathbb {R}^N}|u|^2=a^2, \end{aligned}$$

where $N\ge 2,s\in (0,1),\mu \in \mathbb {R}$ and $2<q<p<2_s^*=2N/(N-2s)$. Under different assumptions on $q<p,a>0$ and $\mu \in \mathbb {R}$, we prove some existence and nonexistence results about the normalized solutions. More specifically, in the purely $L^2$-subcritical case, we overcome the lack of compactness by virtue of the monotonicity of the least energy value and obtain the existence of ground state solution for $\mu >0$. While for the defocusing situation $\mu <0$, we prove the nonexistence result by constructing an auxiliary function. We emphasis that the nonexistence result is new even for Laplacian operator. In the purely $L^2$-supercritical case, we introduce a fiber energy functional to obtain the boundedness of the Palais–Smale sequence and get a mountain-pass type solution. In the combined-type cases, we construct different linking structures to obtain the saddle type solutions. Finally, we remark that we prove a uniqueness result for the homogeneous nonlinearity ($\mu =0$), which is based on the Morse index of ground state solutions.

1 Introduction

In this paper, we focus on the fractional Schrödinger equation

$$\begin{aligned} \mathrm {i}\frac{\partial \psi }{\partial t}=\left( -\Delta \right) ^s \psi -f(|\psi |)\psi , \end{aligned}$$

(1.1)

where $0<s<1$, $\mathrm {i}$ denotes the imaginary unit, $\psi =\psi (x,t):\mathbb {R}^N\times (0,+\infty )\rightarrow \mathbb {C},N\ge 2$ and $f(t)=t^{p-2}+\mu t^{q-2},2<p<q<2_s^*:=2N/(N-2s),\mu \in \mathbb {R}$.

The operator $\left( -\Delta \right) ^s$ can be seen as the infinitesimal generators of Lévy stable diffusion processes, see [2] for example. This operator arises in several areas such as physics, biology, chemistry and finance(see [2, 3]). In recent years, the study of nonlinear equations involving a fractional Laplacian has attracted much attention from many mathematicians, we refer the reader to [14,15,16,17,18,19,20,21, 36, 41, 44,45,46, 49,50,52] and the references therein.

When we are looking for standing waves solutions of (1.1), that is solutions of the form $\psi (t,x)=e^{-\mathrm {i}\lambda t}u(x),\lambda \in \mathbb {R}$. The function u then satisfies the elliptic equation

$$\begin{aligned} \left( -\Delta \right) ^s u-\lambda u=\mu |u|^{q-2}u+|u|^{p-2}u \quad \text{ in } \mathbb {R}^N, \end{aligned}$$

(1.2)

where $(-\Delta )^s$ is the fractional Laplacian operator defined as

$$\begin{aligned} (-\Delta )^su(x)=-\frac{1}{2}\int _{\mathbb {R}^N}\frac{u(x+y)+u(x-y)-2u(x)}{|y|^{N+2s}}dy \end{aligned}$$

for all $x\in \mathbb {R}^N$.

A possible choice is to consider that $\lambda \in \mathbb {R}$ is given and to look for solutions $u\in H^s(\mathbb {R}^N)$ corresponding to critical points [42, 54] of the functional

$$\begin{aligned} J(u)=\int _{\mathbb {R}^N}\left( \frac{1}{2}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2 -\frac{\lambda }{2}|u|^2 -\frac{1}{p}|u|^p-\frac{\mu }{q}|u|^q\right) dx, \end{aligned}$$

and of particular interest are the so-called least energy solutions. Namely solutions which minimize J on the set

$$\begin{aligned} \mathcal {N}:=\left\{ u\in H^s(\mathbb {R}^N){\setminus }\{0\}:J'(u)=0\right\} . \end{aligned}$$

This point of view is adopted in the paper [22], see also [43]. Here and hereafter, for $1\le q<\infty$, we denote by $L^q(\mathbb {R}^N)$ the usual Lebesgue space with norm $|u|_q^q:=\int _{\mathbb {R}^N}|u|^q dx$.

Alternatively one can consider the existence of solutions to (1.2) which have a prescribed $L^2$-norm. Since solutions $\psi \in C\left( [0,T);H^s(\mathbb {R}^N)\right)$ to (1.1) conserved their mass along time, i.e. $|\psi (t)|_2=|\psi (0)|_2$ for $t\in [0,T)$(In fact, multiplying (1.1) by the conjugate $\bar{\psi }$ of $\psi$, integrating over $\mathbb {R}^N$, and taking the imaginary part, we get $\frac{d}{dt}|\psi (t)|_2^2=0$.), it is natural, from a physical point view, to search for such solutions.

When $s=1$, i.e. for the Laplacian operator, Jeanjean’s [32] was the first paper to deal with existence of normalized solutions in purely $L^2$-supercritical case. In recent years, many mathematicians have interest in this type of problems, please see [1, 5, 11,12,13, 29, 30, 33, 34, 47, 48, 53, 55] for normalized solutions to scalar equations in the whole space $\mathbb {R}^N$, [6,7,8,9,10, 26, 27, 31] for normalized solutions to systems in $\mathbb {R}^N$, and [23, 28, 37,38,39,40] for normalized solutions to equations or systems in bounded domains. However, there is few literature concerned about the normalized solutions for the fractional Laplacian operator. With regard to the point, we attempt to study this kind of problem in this paper.

In what follows, we study the fractional nonlinear Schrödinger (NLS) equations with combined nonlinearities

$$\begin{aligned} \left( -\Delta \right) ^s u=\lambda u+\mu |u|^{q-2}u+|u|^{p-2}u \quad \text{ in }~\mathbb {R}^N, \end{aligned}$$

(1.3)

and we look for solutions which satisfy prescribed mass

$$\begin{aligned} \int _{\mathbb {R}^N}|u|^2=a^2, \end{aligned}$$

(1.4)

where $N\ge 2,s\in (0,1),\mu \in \mathbb {R}$, $2<q<p<2_s^*=\frac{2N}{N-2s}$ and $a>0$.

We define the energy functional

$$\begin{aligned} E_\mu :H^s(\mathbb {R}^N)\rightarrow \mathbb {R}, \quad E_\mu (u):=\int _{\mathbb {R}^N}\left( \frac{1}{2}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{1}{p}|u|^p-\frac{\mu }{q}|u|^q\right) \end{aligned}$$

(1.5)

with

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\iint _{\mathbb {R}^{2N}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dxdy, \end{aligned}$$

where

$$\begin{aligned} H^s(\mathbb {R}^N):=\left\{ u\in L^2(\mathbb {R}^N),\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<+\infty \right\} \end{aligned}$$

is a Hilbert space with the inner product and the norm

$$\begin{aligned} (u,v)= & {} \int _{\mathbb {R}^N}[\left( -\Delta \right) ^{\frac{s}{2}}u\cdot \left( -\Delta \right) ^{\frac{s}{2}}v+uv],\\ \Vert u\Vert ^2= & {} \int _{\mathbb {R}^N}[|\left( -\Delta \right) ^{\frac{s}{2}}u|^2+u^2]. \end{aligned}$$

And we denote $H_r^s(\mathbb {R}^N)$ by

$$\begin{aligned} H_r^s(\mathbb {R}^N):=\left\{ u\in H^s(\mathbb {R}^N): u(x)=u(|x|),x\in \mathbb {R}^N\right\} . \end{aligned}$$

Then we know the weak solutions of (1.3) are corresponding to critical points of the energy functional $E_\mu$ under the constraint

$$\begin{aligned} S_a:=\left\{ u\in H^s(\mathbb {R}^N):\int _{\mathbb {R}^N}|u|^2=a^2\right\} . \end{aligned}$$

Let

$$\begin{aligned} m_{a,\mu }:=\inf _{S_a}E_\mu , \end{aligned}$$

then we call u a ground state solution if u achieves $m_{a,\mu }$.

First, we consider the homogeneous nonlinearity, i.e. $\mu =0$, then problem (1.3)–(1.4) becomes the equation

$$\begin{aligned} \left( -\Delta \right) ^s u=\lambda u+|u|^{p-2}u \quad \text{ in }~\mathbb {R}^N \end{aligned}$$

(1.6)

under the constraint $S_a$.

We set

$$\begin{aligned} m_{a}:=\inf _{S_a}E_0, \end{aligned}$$

and denote the $L^2$-critical exponent for fractional NLS equations by

$$\begin{aligned} \bar{p}:=2+\frac{4s}{N}. \end{aligned}$$

In fact, for $u\in S_a$ and $\tau \in \mathbb {R}$, we define

$$\begin{aligned} (\tau \star u)(x):=e^{\frac{N}{2}\tau }u(e^\tau x),\quad \text{ for } \text{ a.e. }~x\in \mathbb {R}^N, \end{aligned}$$

then $\tau \star u\in S_a$. By a simple observation(see the following Theorem 1.2, Lemma 3.1 and their proofs for more details ), we know $E_0(\tau \star u)$ is coercive on $S_a$ for $p<\bar{p}$, while $E_0(\tau \star u)$ is not bounded from below on $S_a$ for $p>\bar{p}$. Based on this fact, we call $\bar{p}$ the $L^2$-critical exponent.

To deal with problem (1.6), we introduce the standard model

$$\begin{aligned} \left( -\Delta \right) ^s u+u-|u|^{\alpha -2}u=0 \quad \text{ in } \mathbb {R}^N, \end{aligned}$$

(1.7)

where $2<\alpha <2_s^*$. By Theorem 3.4 of [24], Eq. (1.7) has a unique positive radial ground state solution, denoted by $Q_{N,\alpha }$. In addition, when $\alpha =\bar{p}$, we define

$$\begin{aligned} \bar{a}^2:=\int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^2. \end{aligned}$$

In what follows, we introduce the fractional Gagliardo–Nirenberg–Sobolev (GNS) inequality.

Lemma 1.1

[24] Let $u\in H^s(\mathbb {R}^N)$ and $2<\alpha <2_s^*$, then the inequality

$$\begin{aligned} \int _{\mathbb {R}^N}|u|^\alpha \le C(s,N,\alpha )\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(\alpha -2)}{4s}}\left( \int _{\mathbb {R}^N} |u|^2\right) ^{\frac{\alpha }{2}-\frac{N(\alpha -2)}{4s}} \end{aligned}$$

holds. Moreover, the best constant $C(s,N,\alpha )$ can be achieved by $Q_{N,\alpha }$.

Now, with regard to the existence of ground state solutions to (1.6), we have

Theorem 1.2

Let $\mu =0,2<p<2_s^*$, we have the following results (i)–(iii).

• (i) If $0<p<\bar{p}$, then for any $a>0$, we obtain that

  $$\begin{aligned} m_a=\inf _{S_a}E_0<0, \end{aligned}$$

  and $m_a$ has a unique(up to a translation) positive radial minimizer $k Q_{N,p}(m x)$ with

  $$\begin{aligned} k=\frac{am^{N/2}}{|Q_{N,p}|_2},\quad m^{2s-\frac{(p-2)N}{2}}\left( \frac{a}{|Q_{N,p}|_2}\right) ^{2-p}=1. \end{aligned}$$

  (1.8)

  In particular, $k Q_{N,p}(m x)$ is the only ground state solution of (1.6) with some $\tilde{\lambda }<0$.

• (ii) If $p=\bar{p}$, then

  • (a) for any $0<a<\bar{a}$, we have

    $$\begin{aligned} m_a=\inf _{S_a}E_0=0, \end{aligned}$$

    and problem (1.6) has no solution at all. In particular, the infimum $m_a$ can’t be achieved by any $u\in S_a$, namely, (1.6) has no ground state solution.

  • (b) for $a=\bar{a}$, we have

    $$\begin{aligned} m_a=\inf _{S_a}E_0=0, \end{aligned}$$

    and $m_a$ has a unique(up to a translation) positive radial minimizer $Q_{N,\bar{p}}$. In particular, $Q_{N,\bar{p}}$ is the only ground state solution of (1.6) with some $\tilde{\lambda }<0$.

  • (c) for any $a>\bar{a}$, we get

    $$\begin{aligned} \inf _{S_a}E_0=-\infty . \end{aligned}$$

    Thus, (1.6) has no ground state solution.

• (iii) If $\bar{p}<p<2_s^*$, then for any $a>0$, we get

  $$\begin{aligned} \inf _{S_a}E_0=-\infty . \end{aligned}$$

  Thus, problem (1.6) has no ground state solution. However, (1.6) still admits a positive radial solution $k Q_{N,p}(m x)$, where k, m satisfy (1.8).

At the moment, we briefly outline the proof of Theorem 1.2: to obtain the value of $\inf \limits _{S_a}E_0$, we resort to a fiber map $E_0(\tau \star u)$ and the fractional Gagliardo–Nirenberg–Sobolev (GNS) inequality(see Lemma 1.1). When dealing with the existence and uniqueness of ground state solution, thanks to the homogeneity of the nonlinear term, we can transform (1.6) with $L^2$-mass constraint into (1.7) by a suitable scaling and then make use of the properties of the ground state solution to (1.7).

Next, we consider the purely $L^2$-subcritical case, i.e., $2<q<p<\bar{p},\mu \in \mathbb {R}$.

Theorem 1.3

Let $2<q<p<\bar{p}$, then we get the following results.

• (i) If $\mu >0$, then for any $a>0$

  $$\begin{aligned} m_{a,\mu }:=\inf _{S_a}E_\mu <0, \end{aligned}$$

  and the infimum is achieved by $\hat{u}\in S_a$ with the following properties: $\hat{u}$ is a positive radial function in $\mathbb {R}^N$ and solves (1.3) for some $\hat{\lambda }<0$. In particular, $\hat{u}$ is a ground state solution of (1.3)–(1.4).

• (ii) If $\mu <0$, let $a>0$ and suppose that

  $$\begin{aligned} |\mu |a^{\delta (p,q)}\ge \frac{q}{C(s,N,q)}\left( \frac{C(s,N,p)}{p}\right) ^{\frac{\bar{p}-q}{\bar{p}-p}}2^{\frac{p-q}{\bar{p}-p}} \left( \left( \frac{p-q}{\bar{p}-q}\right) ^{\frac{\bar{p}-q}{\bar{p}-p}}-\left( \frac{p-q}{\bar{p}-q}\right) ^{\frac{p-q}{\bar{p}-p}}\right) \end{aligned}$$

  (1.9)

  with

  $$\begin{aligned} \delta (p,q):=\frac{4s(q-p)}{N(\bar{p}-p)}<0. \end{aligned}$$

  Then

  $$\begin{aligned} m_{a,\mu }=\inf _{S_a}E_\mu =0, \end{aligned}$$

  and the infimum $m_{a,\mu }$ can’t be achieved by any $u\in S_a$. Therefore, problem (1.3)–(1.4) has no ground state solution.

Remark 1.4

The nonexistence result in (ii) of Theorem 1.3 is even new for the Laplacian case, we point out that our method can also apply to the Laplacian operator.

In the proof of existence of a minimizer for $m_{a,\mu }$ in Theorem 1.3, the difficulty lies in the fact that the embedding $H_r^s(\mathbb {R}^N)\hookrightarrow L^2(\mathbb {R}^N)$ is not compact. We will overcome the obstacle by virtue of the monotonicity of $m_{a,\mu }$. To prove the nonexistence result, we smartly construct an auxiliary function and analyse its properties.

In what follows, for the purely $L^2$-supercritical case, namely, $\bar{p}<q<p<2_s^*$, we obtain

Theorem 1.5

Let $\bar{p}<q<p<2_s^*$ and $\mu \in \mathbb {R}$. Then it holds that

$$\begin{aligned} \inf _{S_a}E_\mu =-\infty . \end{aligned}$$

Moreover, if $\mu >0$, then for any $a>0$ Eq. (1.3) has a radial solution $u_a$ for some $\lambda _a<0$.

In the $L^2$-supercritical case, $E_\mu$ is not bounded from below on $S_a$, i.e., $\inf _{S_a}E_\mu =-\infty$. Thus, it is not more possible to search for a minimum of $E_\mu$ on $S_a$. We have to look for a critical point with a minimax characterization. Although $E_\mu$ has a mountain-pass geometry on $S_a$, but unfortunately the boundedness of the obtained Palais–Smale sequence is not yet clear. In this paper we adopt a similar idea in [32] and construct an auxiliary map $I_\mu (u,\tau ):=E_\mu (\tau \star u)$, which on $S_a\times \mathbb {R}$ has the same type of geometric structure as $E_\mu$ on $S_a$. Besides, the Palais–Smale sequence of $I_\mu$ satisfies the additional condition(see Proposition 5.4), which is the key ingredient to obtain the boundedness of the Palais–Smale sequence. We point out that although we take a similar idea in [32], the extra difficulty still occurs due to the nonlocal term.

In the following we give a bifurcation result.

Corollary 1.6

Let $\bar{p}<q<p<2_s^*$ and $\mu >0$. Let $(u_a,\lambda _a)$ be a solution of (1.3) obtained in Theorem 1.5. Then, as $a\rightarrow 0$, we have

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2\rightarrow +\infty \quad \text{ and }~~ \lambda _a\rightarrow -\infty . \end{aligned}$$

Finally, we deal with the combined-type cases $2<q\le \bar{p}=2+\frac{4s}{N}\le p<2_s^*,p\ne q$.

Case (I): $2<q<p=\bar{p}$.

Theorem 1.7

Let $2<q<p=\bar{p}$,we have

• (i) if $0<a<\bar{a}$, then:

  • (a) for every $\mu >0$,

    $$\begin{aligned} m_{a,\mu }:=\inf _{S_a}E_\mu <0, \end{aligned}$$

    and the infimum admits a positive radial minimizer $\tilde{u}\in S_a$, and $\tilde{u}$ solves (1.3) for some $\tilde{\lambda }<0$.

  • (b) for every $\mu <0$,

    $$\begin{aligned} \inf _{S_a}E_\mu =0, \end{aligned}$$

    and problem (1.3)–(1.4) has no solution at all.

• (ii) if $a=\bar{a}$, then:

  • (a) for every $\mu >0$,

    $$\begin{aligned} \inf _{S_a}E_\mu =-\infty . \end{aligned}$$

  • (b) for every $\mu <0$,

    $$\begin{aligned} \inf _{S_a}E_\mu =0, \end{aligned}$$

    and problem (1.3)–(1.4) has no solution at all.

• (iii) if $a>\bar{a}$, then for every $\mu \in \mathbb {R}$

  $$\begin{aligned} \inf _{S_a}E_\mu =-\infty . \end{aligned}$$

We remark that the proof of Theorem 1.7 is based on Theorem 1.2 and the Pohozaev identity.

Case (II): $2<q<\bar{p}<p<2_s^*$.

First, for the focusing subcritical perturbation case, i.e. $\mu >0$, we have:

Theorem 1.8

Let $2<q<\bar{p}<p<2_s^*$, $a,\mu >0$. We also suppose that

$$\begin{aligned} \mu a^{\gamma (p,q)}<\left( \frac{p(\bar{p}-q)}{2C(s,N,p)(p-q)}\right) ^{\frac{\bar{p}-q}{p-\bar{p}}}\left( \frac{q(p-\bar{p})}{2C(s,N,q)(p-q)}\right) \end{aligned}$$

(1.10)

with

$$\begin{aligned} \gamma (p,q)=\left( p-\frac{N(p-2)}{2s}\right) \frac{\bar{p}-q}{p-q}+\left( q-\frac{N(q-2)}{2s}\right) >0. \end{aligned}$$

Then problem (1.3)–(1.4) has two radial solutions, denoted by $\tilde{u}$ and $\hat{u}$. Moreover, $E_\mu (\tilde{u})<0$, $E_\mu (\hat{u})>0$ and $\tilde{u},\hat{u}$ solve (1.3) for suitable $\tilde{\lambda },\hat{\lambda }<0$.

In the proof of Theorem 1.8, we follow the idea of [48] to restricted the functional $E_\mu$ on the Pohozaev set $\mathcal {P}_{a,\mu }$(see Sect. 6)and know that $E_\mu |_{\mathcal {P}_{a,\mu }}$ is bounded from below. Then we can get a local minimizer $\tilde{u}$ for $E_\mu |_{\mathcal {P}_{a,\mu }}$ and construct a minimax characterization for $E_\mu$ to get the second critical point $\hat{u}$. We emphasis that (1.10) has been used to ensure that $\mathcal {P}_{a,\mu }$ is a smooth manifold.

Next we consider the defocusing subcritical perturbation case, i.e. $\mu <0$, we have:

Theorem 1.9

Let $2<q<\bar{p}<p<2_s^*$, $a>0,\mu <0$. We also suppose that

$$\begin{aligned} |\mu | a^{\beta (p,q)}<\left( \frac{2ps}{NC(s,N,p)(p-2)}\right) ^{\frac{\bar{p}-q}{p-\bar{p}}}\left( \frac{q(2_s^*-p)(N-2s)}{2NC(s,N,q)(p-q)}\right) \end{aligned}$$

(1.11)

with

$$\begin{aligned} \beta (p,q)=\left( p-\frac{N(p-2)}{2s}\right) \frac{\bar{p}-q}{p-\bar{p}}+\left( q-\frac{N(q-2)}{2s}\right) >0. \end{aligned}$$

Then problem (1.3)–(1.4) has a radial solution, denoted by $\hat{u}$. Moreover, $E_\mu (\hat{u})>0$ and $\hat{u}$ solve (1.3) for some $\hat{\lambda }<0$.

In the proof of Theorem 1.9, we construct a minimax characterization for $E_\mu$ to get a critical point $\hat{u}$. We emphasis that (1.11) has been used to deduce the compactness of the Palais–Smale sequence obtained by minimax scheme

Case (III): $2<q=\bar{p}<p<2_s^*$.

First, for the focusing critical perturbation case, i.e. $\mu >0$, we have:

Theorem 1.10

Let $2<q=\bar{p}<p<2_s^*$, $a,\mu >0$. We also suppose that

$$\begin{aligned} \mu a^{\frac{4s}{N}}<\frac{\bar{p}}{2C(s,N,\bar{p})}. \end{aligned}$$

(1.12)

Then problem (1.3)–(1.4) has a radial solution, denoted by $\hat{u}$. Moreover, $E_\mu (\hat{u})>0$ and $\hat{u}$ solve (1.3) for some $\hat{\lambda }<0$.

Next we consider the defocusing critical perturbation case, i.e. $\mu <0$, we have:

Theorem 1.11

Let $2<q=\bar{p}<p<2_s^*$, $a>0,\mu <0$. We also suppose that

$$\begin{aligned} |\mu | a^{\frac{4s}{N}}<\frac{\bar{p}(2_s^*-p)(N-2s)}{2NC(s,N,\bar{p})(p-\bar{p})}. \end{aligned}$$

(1.13)

Then problem (1.3)–(1.4) has a radial solution, denoted by $\hat{u}$. Moreover, $E_\mu (\hat{u})>0$ and $\hat{u}$ solve (1.3) for some $\hat{\lambda }<0$.

We remark that the proofs for Theorems 1.10–1.11 are very similar to that of Theorem 1.9.

This paper is organized as follows. In Sect. 2, we give some lemmas which will be used later. We discuss the homogeneous nonlinearity and prove Theorem 1.2 in Sect. 3. In particular, we prove a uniqueness result which is based on the Morse index of ground state solution. Section 4 is devoted to the purely $L^2$-subcritical case. In this case, we overcome the lack of compactness (notice that $H_r^s(\mathbb {R}^N)\hookrightarrow L^2(\mathbb {R}^N)$ is not compact) by virtue of the monotonicity of the least energy value, which can be proved by a similar argument as Lemma 3.2 and Corollary 3.3. And we obtain the ground state solution for $\mu >0$. While for the defocusing situation $\mu <0$, we prove the nonexistence result by smartly constructing an auxiliary function, see Lemma 4.1. We emphasis that the nonexistence result is new even for Laplacian operator. In Sect. 5, we deal with the purely $L^2$-supercritical case and prove Theorem 1.5 and Corollary 1.6. In this case, although the energy functional $E_\mu$ has a mountain-pass geometry on the mass constraint set $S_a$, but unfortunately we can not deduce the boundedness of the Palais–Smale sequence. To overcome the difficulty, we introduce a fiber energy functional to obtain the boundedness of the Palais–Smale sequence and get a mountain-pass type solution, see Propositions 5.3, 5.4 and Lemma 5.5. In the final section, we consider the combined-type cases and prove Theorems 1.7–1.11. In the combined-type cases, we construct different linking structures to obtain the saddle-type solutions, see Lemmas 6.16 and 6.21.

2 Preliminaries

In this section, we will give some lemmas for convenience. First, we give the Pohozaev identity for the fractional Laplacian operator.

Lemma 2.1

[15, Appendix] Let $u\in H^s(\mathbb {R}^N),N\ge 2$ satisfy the equation

$$\begin{aligned} \left( -\Delta \right) ^su=g(u), \end{aligned}$$

then it holds that

$$\begin{aligned} \frac{N-2s}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=N\int _{\mathbb {R}^N} G(u), \end{aligned}$$

where $G(s)=\int _0^s g(t) dt$.

Remark 2.2

For $\alpha =\bar{p}$, we can get

$$\begin{aligned} \bar{a}^2=\int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^2=\left( \frac{\bar{p}}{2C(s,N,\bar{p})}\right) ^{\frac{N}{2s}}. \end{aligned}$$

(2.1)

In fact, by Lemma 1.1, the best constant $C(s,N,\bar{p})$ can be achieved by $Q_{N,\bar{p}}$. In virtue of the Pohozaev identity(see Lemma 2.1)and the Eq. (1.7) for $Q_{N,\bar{p}}$, we know

$$\begin{aligned} \int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^{\bar{p}}=\left( 1+\frac{2N}{4s}\right) \int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^2,\quad \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}Q_{N,\bar{p}}|^2= \frac{2N}{4s}\int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^2. \end{aligned}$$

(2.2)

Substituting these equalities into the fractional GNS inequality, we get (2.1).

Lemma 2.3

[4, Section 9] Let $s\in (0,1)$. For any $u\in H^s(\mathbb {R}^N)$, the following inequality holds

$$\begin{aligned} \iint _{\mathbb {R}^{2N}}\frac{(u^*(x)-u^*(y))^2}{|x-y|^{N+2s}}dxdy\le \iint _{\mathbb {R}^{2N}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dxdy, \end{aligned}$$

where $u^*$ denotes the symmetric radial decreasing rearrangement of u.

Lemma 2.4

[35] Let $N\ge 2$, then $H_r^s(\mathbb {R}^N)$ is compactly embedding into $L^p(\mathbb {R}^N)$ for $p\in (2,2_s^*)$.

Finally, we give a version of linking theorem, see [25, Section 5].

Definition 2.5

Let B be a closed subset of X. We shall say that a class $\mathcal {F}$ of compact subsets of X is homotopy-stable family with extended boundary B if for any set A in $\mathcal {F}$ and any $\eta \in C([0,1]\times X;X)$ satisfying $\eta (t,x)=x$ for all (t, x) in $\left( \left\{ 0\right\} \times X\right) \cup ([0,1]\times B)$ we have that $\eta \left( \left\{ 1\right\} \times A\right) \in \mathcal {F}$.

Lemma 2.6

Let $\varphi$ be a $C^1$-functional on a complete connected $C^1$-Finsler manifold X and consider a homotopy-stable family $\mathcal {F}$ with extended boundary B. Set

$$\begin{aligned} c=c(\varphi ,\mathcal {F})=\inf _{A\in \mathcal {F}}\max _{x\in A}\varphi (x) \end{aligned}$$

and let F be a closed subset of X satisfying

$$\begin{aligned} A\cap F{\setminus } B\ne \emptyset \quad \text{ for } \text{ every } A\in \mathcal {F} \end{aligned}$$

(2.3)

and

$$\begin{aligned} \sup _{x\in B} \varphi (x)\le c\le \inf _{x\in F}\varphi (x). \end{aligned}$$

(2.4)

Then, for any sequence of sets $(A_n)_n$ in $\mathcal {F}$ such that $\lim \limits _{n}\sup _{A_n}\varphi =c$, there exists a sequence $(x_n)_n$ in $X{\setminus } B$ such that

• (i) $\lim _n \varphi (x_n)=c$.

• (ii) $\lim _n\left\| d\varphi (x_n)\right\| =0$.

• (iii) $\lim _n \text{ dist }(x_n,F)=0$.

• (iv) $\lim _n \text{ dist }(x_n,A_n)=0$.

3 Homogeneous nonlinearity ($\mu =0$)

In this section, we deal with the case $\mu =0$ and prove Theorem 1.2.

Lemma 3.1

For any $p\in (2,\bar{p})$ and $a>0$, we have

$$\begin{aligned} -\infty<m_a=\inf _{S_a}E_0<0. \end{aligned}$$

Proof

By the fractional GNS inequality(see Lemma 1.1), we get

$$\begin{aligned} E_0(u)\ge \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}} \end{aligned}$$

(3.1)

for every $u\in S_a$. Since $2<p<\bar{p}$, it implies that $0<\frac{N(p-2)}{4s}<1$ and hence $E_0$ is coercive on $S_a$, which provides that $m_a>-\infty$.

On the other hand, for $u\in S_a$,

$$\begin{aligned} E_0(\tau \star u)= & {} \frac{1}{2}\iint _{\mathbb {R}^{2N}}\frac{e^{N\tau }(u(e^\tau x)-u(e^\tau y))^2}{|x-y|^{N+2s}}dxdy-\frac{1}{p}\int _{\mathbb {R}^N}e^{\frac{N\tau p}{2}}|u(e^\tau x)|^pdx\\= & {} \frac{1}{2}e^{2s\tau }\iint _{\mathbb {R}^{2N}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dxdy-\frac{e^{N\tau (\frac{p}{2}-1)}}{p}\int _{\mathbb {R}^N}|u|^p\\= & {} e^{N\tau (\frac{p}{2}-1)}\left[ \frac{1}{2}e^{\tau (2s-N(\frac{p}{2}-1))}\iint _{\mathbb {R}^{2N}}\frac{(u(x)-u(y))^2}{|x-y|^{N+2s}}dxdy-\frac{1}{p}\int _{\mathbb {R}^N}|u|^p\right] . \end{aligned}$$

Noticing that $p<\bar{p}$, we have $2s-N(p/2-1)>0$ and hence $E_0(\tau \star u)<0$ for every $u\in S_a$ with $\tau \ll -1$. Therefore, we know that $m_a<0$ for any $a>0$. $\square$

In the $L^2$-subcritical case, since $m_a<0$ for any $a>0$, we can give the strict sub-additivity for $m_a$.

Lemma 3.2

Let $p\in (2,\bar{p})$, and $a_1,a_2>0$ be such that $a_1^2+a_2^2=a^2$. Then

$$\begin{aligned} m_a<m_{a_1}+m_{a_2}. \end{aligned}$$

Proof

Let $c>0,\theta >1$ and let $\left\{ u_n\right\} \subseteq S_c$ be a minimizing sequence for $m_c$. Then

$$\begin{aligned} m_{\theta c}\le E_0(\theta u_n)=\frac{1}{2}\theta ^2\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2-\frac{\theta ^p}{p}\int _{\mathbb {R}^N}|u_n|^p<\theta ^2E_0(u_n), \end{aligned}$$

since $\theta >1$ and $p>2$. As a consequence $m_{\theta c}\le \theta ^2 m_c$, with equality if and only if $\int _{\mathbb {R}^N}|u_n|^p\rightarrow 0$ as $n\rightarrow \infty$. But this is not possible, since otherwise we would find

$$\begin{aligned} 0>m_c=\lim _{n\rightarrow \infty }E_0(u_n)\ge \liminf _{n\rightarrow \infty }\frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2\ge 0, \end{aligned}$$

a contradiction, where the first inequality follows from Lemma 3.1. Thus, we have the strict inequality $m_{\theta c}<\theta ^2 m_c$.

Next, we show that $m_a<m_{a_1}+m_{a_2}$. We may assume that $a_1\ge a_2$ and divide into two cases. Case 1: $a_1>a_2$. For this case, we have

$$\begin{aligned} m_a=m_{\frac{a}{a_1}a_1}<\left( \frac{a}{a_1}\right) ^2m_{a_1}=m_{a_1}+\frac{a^2-a_1^2}{a_1^2}m_{a_1}=m_{a_1}+\frac{a_2^2}{a_1^2}m_{\frac{a_1}{a_2}a_2}<m_{a_1}+m_{a_2}. \end{aligned}$$

Case 2: $a_1=a_2$. For this case, we have

$$\begin{aligned} m_a=m_{\sqrt{2}a_1}<2m_{a_1}=m_{a_1}+m_{a_2}. \end{aligned}$$

$\square$

Noticing again that the fact $m_a<0$ for any $a>0$, we immediately obtain

Corollary 3.3

Let $p\in (2,\bar{p})$, then $m_a$ is strictly decreasing in $a\in (0,\infty )$.

Now we are in position to proceed with the proof of Theorem 1.2.

Proof of Theorem 1.2

For (i), let $\{u_n\}\subset S_a$ be a minimizing sequence for $m_a$. By (3.1), we know that $E_0$ is coercive on $S_a$ and deduce that $\{u_n\}$ is bounded in $H^s(\mathbb {R}^N)$. Noting that $E_0$ is even and combining with Lemma 2.3, we can suppose that $u_n's$ are nonnegative and radially symmetric, i.e., $0\le u_n\in H_r^s(\mathbb {R}^N)$. Thus, by Lemma 2.4, we have

$$\begin{aligned} u_n\rightarrow u\quad \text{ weakly } \text{ in } H^s(\mathbb {R}^N),\quad u_n\rightarrow u\quad \text{ strongly } \text{ in } L^p(\mathbb {R}^N),\quad p\in (2,2_s^*), \end{aligned}$$

providing that

$$\begin{aligned} E_0(u)\le \liminf _{n\rightarrow \infty }E_0(u_n)=m_a,\quad |u|_2^2\le a^2. \end{aligned}$$

Since $E_0(u)\le m_a<0$, we know $u\not \equiv 0$. By Corollary 3.3, $m_a$ is strictly decreasing in a, then it must hold that

$$\begin{aligned} E_0(u)=m_a,\quad |u|_2^2= a^2. \end{aligned}$$

Thus, u is a minimizer for $m_a$.

Next we show the uniqueness of the minimizer for $m_a$. Since $S_a$ is a $C^1$ manifold with codimension 1 and u is a minimizer of $E_0$ constrained on $S_a$, we know that the Morse index of u, denoted by m(u), is less than or equal to 1. On the other hand, by the Lagrange multiplier rule, there exists $\lambda \in \mathbb {R}$ such that u satisfies

$$\begin{aligned} \left( -\Delta \right) ^s u=\lambda u+u^{p-1} \quad \text{ in }~\mathbb {R}^N. \end{aligned}$$

Since $u\ge 0,\not \equiv 0$, by the strong maximum principle, we get $u>0$. The linearized operator at u is

$$\begin{aligned} L_\lambda =\left( -\Delta \right) ^s-\lambda -(p-1)u^{p-2}, \end{aligned}$$

together with the equation for u, we easily see that $\left\langle L_\lambda u,u\right\rangle =(2-p)|u|_p^p<0$. Therefore, $m(u)=1$. According to the Pohozaev identity and the equation for u, we obtain

$$\begin{aligned} \left( \frac{1}{p}-\frac{1}{2_s^*}\right) |u|_p^p+ \left( \frac{1}{2}-\frac{1}{2_s^*}\right) \lambda |u|_2^2=0, \end{aligned}$$

which implies $\lambda <0$. Set

$$\begin{aligned} u_{\beta ,\gamma }=\beta u(\gamma x) \end{aligned}$$

with

$$\begin{aligned} \lambda \gamma ^{2s}=-1,\quad \beta ^{2-p}\gamma ^{2s}=1, \end{aligned}$$

(3.2)

then $u_{\beta ,\gamma }$ satisfies (1.7) for $\alpha =p$. Moreover, since $m(u)=1$, it is straightforward to verify that the Morse index of $u_{\beta ,\gamma }$ with respect to the linearized operator

$$\begin{aligned} L_+=\left( -\Delta \right) ^s+1-(p-1)\left( u_{\beta ,\gamma }\right) ^{p-2} \end{aligned}$$

is exactly 1. By [24, Theorem 3.4], it must hold that $u_{\beta ,\gamma }=Q_{N,p}$. Let

$$\begin{aligned} k=\frac{1}{\beta },\quad m=\frac{1}{\gamma }, \end{aligned}$$

notice that $|u_{\beta ,\gamma }|_2=|Q_{N,p}|_2$ and (3.2), we get $u(x)=k Q_{N,p}(m x)$ and (1.8).

For (a) of (ii), by the fractional GNS inequality, we get

$$\begin{aligned} E_0(u)\ge & {} \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{C(s,N,\bar{p})}{\bar{p}}a^{\bar{p}-\frac{N(\bar{p}-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(\bar{p}-2)}{4s}}\\= & {} \frac{1}{2}\left( 1-\left( \frac{a}{\bar{a}}\right) ^{\frac{4s}{N}}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2 \end{aligned}$$

for every $u\in S_a$, here we use (2.1). Thus, it results that $m_a\ge 0$ for any $0<a<\bar{a}$. In addition, $E_0(\tau \star u)\rightarrow 0$ as $\tau \rightarrow -\infty$ for $u\in S_a$. Notice that $\tau \star u\in S_a$ for any $u\in S_a$, we get $m_a=0$.

We assume by contradiction that problem (1.6) has a solution $u\in S_a$, then by the Pohozaev identity and the equation for u, we get

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\frac{2}{\bar{p}}\int _{\mathbb {R}^N}|u|^{\bar{p}}. \end{aligned}$$

In virtue of the fractional GNS inequality and (2.1), we obtain

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\le \left( \frac{a}{\bar{a}}\right) ^{\frac{4s}{N}}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2. \end{aligned}$$

Since being $a<\bar{a}$, it results that u must be a constant, contradicting the fact that $u\in S_a$.

For (b) of (ii), $m_{\bar{a}}=0$ follows from a similar argument as (a) of (ii). By (2.2), we know $E_0(Q_{N,\bar{p}})=0$. Taking a similar argument as the proof of uniqueness in (i), we can obtain the uniqueness of minimizer for $m_{\bar{a}}$.

For (c) of (ii), let

$$\begin{aligned} u_a=\frac{a}{\bar{a}}Q_{N,\bar{p}}, \end{aligned}$$

by (2.2), we get $|u_a|_2^2=a^2$ and $E_0(u_a)<0$. Since $p=\bar{p}$, we have $E_0(\tau \star u_a)=e^{2s\tau }E_0(u_a)$, and hence $E_0(\tau \star u_a)\rightarrow -\infty$ as $\tau \rightarrow +\infty$. Thus, it holds that

$$\begin{aligned} \inf _{S_a}E_0=-\infty . \end{aligned}$$

For (iii), since $p>\bar{p}$, for any $a>0$ and $u\in S_a$, it holds that $E_0(\tau \star u)\rightarrow -\infty$ as $\tau \rightarrow +\infty$. Thus, we get

$$\begin{aligned} \inf _{S_a}E_0=-\infty . \end{aligned}$$

Thanks to the homogeneity of the nonlinear term, for any $a>0$, if we set

$$\begin{aligned} u_a(x)=k Q_{N,p}(m x) \end{aligned}$$

with k, m satisfying (1.8), then $|u_a|_2^2=a^2$ and $u_a$ solves problem (1.6) for some $\lambda <0$. $\square$

4 Purely $L^2$-subcritical case

In this section, we deal with the case $2<q<p<\bar{p}=2+\frac{4s}{N},\mu \in \mathbb {R}$ and prove Theorem 1.3.

Lemma 4.1

Let

$$\begin{aligned} g(t)=\frac{1}{2}t^\beta -At^\gamma +B,\quad t\in [0,\infty ) \end{aligned}$$

with $0<\gamma<\beta <1,A,B>0$, then $g(t)\ge 0$ for any $t\in [0,\infty )$ whenever

$$\begin{aligned} B\ge A^{\frac{\beta }{\beta -\gamma }}2^{\frac{\gamma }{\beta -\gamma }}\left( \left( \frac{\gamma }{\beta }\right) ^{\frac{\gamma }{\beta -\gamma }}- \left( \frac{\gamma }{\beta }\right) ^{\frac{\beta }{\beta -\gamma }}\right) . \end{aligned}$$

Proof

Deviating g(t) with respect to t, we obtain

$$\begin{aligned} g'(t)=t^{\gamma -1}\left( \frac{\beta }{2}t^{\beta -\gamma }-A\gamma \right) . \end{aligned}$$

Set

$$\begin{aligned} t_0=\left( \frac{2A\gamma }{\beta }\right) ^{\frac{1}{\beta -\gamma }}, \end{aligned}$$

then $g'(t)<0$ in $(0,t_0)$ and $g'(t)>0$ in $(t_0,\infty )$. Thus, g(t) has a global minimum at $t_0$. To guarantee that $g(t)\ge 0$ for any $t\in [0,\infty )$, it suffices to show that $g(t_0)\ge 0$, which follows from the fact

$$\begin{aligned} B\ge A^{\frac{\beta }{\beta -\gamma }}2^{\frac{\gamma }{\beta -\gamma }}\left( \left( \frac{\gamma }{\beta }\right) ^{\frac{\gamma }{\beta -\gamma }}- \left( \frac{\gamma }{\beta }\right) ^{\frac{\beta }{\beta -\gamma }}\right) . \end{aligned}$$

$\square$

In what follows, we begin with the proof of Theorem 1.3.

Proof of Theorem 1.3

For (i), we can follows the lines in the proof of (i) of Theorem 1.2. This means that we can prove the analogous versions of Lemma 3.1–3.2 and Corollary 3.3, then we can adopt a similar argument as the proof for the existence of a minimizer of (i) of Theorem 1.2. Here we omit the details.

For (ii), for any $u\in S_a$, by the fractional GNS inequality(see Lemma 1.1), we get

$$\begin{aligned} E_\mu (u)\ge & {} \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}\nonumber \\&-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(q-2)}{4s}}\nonumber \\= & {} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(q-2)}{4s}}\left[ \frac{1}{2} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(\bar{p}-q)}{4s}}\right. \nonumber \\&\left. -\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-q)}{4s}}-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}}\right] . \end{aligned}$$

(4.1)

Set

$$\begin{aligned} \beta= & {} \frac{N(\bar{p}-q)}{4s},\quad \quad \gamma =\frac{N(p-q)}{4s}, \\ A= & {} \frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}},\quad B=-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}}, \end{aligned}$$

then by Lemma 4.1 we know $E_\mu (u)\ge 0$ for any $u\in S_a$ whenever

$$\begin{aligned} B\ge A^{\frac{\beta }{\beta -\gamma }}2^{\frac{\gamma }{\beta -\gamma }}\left( \left( \frac{\gamma }{\beta }\right) ^{\frac{\gamma }{\beta -\gamma }}- \left( \frac{\gamma }{\beta }\right) ^{\frac{\beta }{\beta -\gamma }}\right) , \end{aligned}$$

that is, a and $\mu$ satisfy

$$\begin{aligned} (-\mu )a^{\frac{4s(q-p)}{N(\bar{p}-p)}}\ge \frac{q}{C(s,N,q)}\left( \frac{C(s,N,p)}{p}\right) ^{\frac{\bar{p}-q}{\bar{p}-p}}2^{\frac{p-q}{\bar{p}-p}} \left( \left( \frac{p-q}{\bar{p}-q}\right) ^{\frac{\bar{p}-q}{\bar{p}-p}}-\left( \frac{p-q}{\bar{p}-q}\right) ^{\frac{p-q}{\bar{p}-p}}\right) . \end{aligned}$$

Thus, our assumption (1.9) implies that $m_{a,\mu }\ge 0$. On the other hand, by a direct computation, we see that $E_\mu (\tau \star u)\rightarrow 0$ as $\tau \rightarrow -\infty$ for $u\in S_a$. Notice that $\tau \star u\in S_a$ for any $u\in S_a$, we get $m_{a,\mu }=0$.

Next we show that $m_{a,\mu }=0$ can’t be achieved by any $u\in S_a$. We assume by contradiction that there exists $u_0\in S_a$ such that $E_\mu (u_0)=0$. By (4.1), we can get

$$\begin{aligned} 0= & {} E_\mu (u_0)\ge \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_0|^2-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_0|^2\right) ^{\frac{N(p-2)}{4s}}\\&-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_0|^2\right) ^{\frac{N(q-2)}{4s}}\ge 0, \end{aligned}$$

where the last inequality follows from the assumption (1.9) and Lemma 4.1. Since the equalities for the fractional GNS inequalities at $\alpha =p$ and $\alpha =q$ can’t hold at the same time, the first inequality of the formula above is indeed strict and hence we obtain a contradiction. $\square$

5 Purely $L^2$-supercritical case

In this section, we deal with the case $\bar{p}<q<p<2_s^*,\mu \in \mathbb {R}$ and prove Theorem 1.5.

Setting

$$\begin{aligned} S_{a,r}=S_a\cap H_r^s(\mathbb {R}^N)=\left\{ u\in S_a: u(x)=u(|x|)\right\} , \end{aligned}$$

and the product space $E=H^s(\mathbb {R}^N)\times \mathbb {R}$, we introduce the auxiliary functional $I_\mu :E\rightarrow \mathbb {R}$ by

$$\begin{aligned} I_\mu (u,\tau ):=E_\mu (\tau \star u)=\frac{e^{2s\tau }}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{e^{N\tau (\frac{p}{2}-1)}}{p}\int _{\mathbb {R}^N}|u|^p- \mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|u|^q, \end{aligned}$$

then we easily see that $I_\mu$ is a $C^1$-functional. In addition, we define the Pohozaev set by

$$\begin{aligned} \mathcal {P}_\mu =\left\{ u\in H^s(\mathbb {R}^N):P_\mu (u)=0\right\} \end{aligned}$$

with

$$\begin{aligned} P_\mu (u)=\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{N(p-2)}{2ps}\int _{\mathbb {R}^N}|u|^p-\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|u|^q. \end{aligned}$$

It is well known that any critical points of $E_\mu |_{S_a}$ stay in $\mathcal {P}_\mu$, as a consequence of the Pohozaev identity(see Lemma 2.1).

Lemma 5.1

Let $u\in S_{a,r}$ be arbitrary but fixed. Then we have

• (1) $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2\rightarrow 0$ and $I_\mu (u,\tau )\rightarrow 0$ as $\tau \rightarrow -\infty$;

• (2) $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2\rightarrow +\infty$ and $I_\mu (u,\tau )\rightarrow -\infty$ as $\tau \rightarrow +\infty$.

Proof

A straightforward calculation shows that

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2=\frac{e^{2s\tau }}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2. \end{aligned}$$

In addition, we have

$$\begin{aligned} I_\mu (u,\tau )=\frac{e^{2s\tau }}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{e^{N\tau (\frac{p}{2}-1)}}{p}\int _{\mathbb {R}^N}|u|^p- \mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|u|^q. \end{aligned}$$

Since being $p>q>\bar{p}$, it holds $N(p/2-1)>N(q/2-1)>2s$. Thus, the conclusions (1) and (2) easily follows from these facts above. $\square$

Lemma 5.2

Let $\bar{p}<q<p<2_s^*$ and $\mu >0$. Then there exists $K_a>0$ such that

$$\begin{aligned} 0<\sup _{u\in \mathcal {A}}E_\mu (u)<\inf _{u\in \mathcal {B}}E_\mu (u) \end{aligned}$$

with

$$\begin{aligned} \mathcal {A}=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<K_a\right\} ,\quad \mathcal {B}=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=2K_a\right\} . \end{aligned}$$

Proof

Let $K>0$ be arbitrary but fixed and suppose $u,v\in S_{a,r}$ are such that

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<K \quad \text{ and }\quad \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}v|^2=2K. \end{aligned}$$

Then for K small enough, by the fractional GNS inequality, we have

$$\begin{aligned} E_\mu (v)-E_\mu (u)= & {} \int _{\mathbb {R}^N}\left( \frac{1}{2}|\left( -\Delta \right) ^{\frac{s}{2}}v|^2-\frac{1}{p}|v|^p-\frac{\mu }{q}|v|^q\right) \\&-\int _{\mathbb {R}^N}\left( \frac{1}{2}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{1}{p}|u|^p-\frac{\mu }{q}|u|^q\right) \\\ge & {} \int _{\mathbb {R}^N}\left( \frac{1}{2}|\left( -\Delta \right) ^{\frac{s}{2}}v|^2-\frac{1}{p}|v|^p-\frac{\mu }{q}|v|^q\right) -\frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\\\ge & {} \frac{K}{2}-C_1 \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}v|^2\right) ^{\frac{N(p-2)}{4s}}-C_2\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}v|^2\right) ^{\frac{N(q-2)}{4s}}\\= & {} \frac{K}{2}-C_12^{\frac{N(p-2)}{4s}}K^{\frac{N(p-2)}{4s}}-C_22^{\frac{N(q-2)}{4s}}K^{\frac{N(q-2)}{4s}}\\\ge & {} \frac{K}{4}, \end{aligned}$$

here we use the fact that $\frac{N(p-2)}{4s}>\frac{N(q-2)}{4s}>1$. Clearly also, for $K>0$ sufficiently small: for any $u\in S_{a,r}$ satisfying $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<K$, we have still by the fractional GNS inequality

$$\begin{aligned} E_\mu (u)\ge & {} \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}\\&-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(q-2)}{4s}}>0. \end{aligned}$$

In summary, we can choose suitable sufficiently small constant $K_a>0$ such that

$$\begin{aligned} 0<\sup _{u\in \mathcal {A}}E_\mu (u)<\inf _{u\in \mathcal {B}}E_\mu (u) \end{aligned}$$

with

$$\begin{aligned} \mathcal {A}=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<K_a\right\} ,\quad \mathcal {B}=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=2K_a\right\} . \end{aligned}$$

$\square$

Having established the mountain pass geometry of $I_\mu$ and $E_\mu$, we construct their minimax characterization. For the Laplacian case, the construction has appeared in [32].

Proposition 5.3

Let $\bar{p}<q<p<2_s^*$ and $\mu >0$. There exist $\hat{u},\tilde{u} \in S_{a,r}$ such that

• (1) $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\le K_a$,

• (2) $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\tilde{u}|^2> 2K_a$,

• (3) $E_\mu (\hat{u})>0\ge E_\mu (\tilde{u})$.

Moreover, setting

$$\begin{aligned} \widetilde{\gamma }_a=\inf _{\widetilde{h}\in \widetilde{\Gamma }_a}\max _{t\in [0,1]}I_\mu (\widetilde{h}(t)) \end{aligned}$$

with

$$\begin{aligned} \widetilde{\Gamma }_a=\left\{ \widetilde{h}\in C\left( [0,1],S_{a,r}\times \mathbb {R}\right) :\widetilde{h}(0)=(\hat{u},0), \widetilde{h}(1)=(\tilde{u},0)\right\} , \end{aligned}$$

and

$$\begin{aligned} \gamma _a=\inf _{h\in \Gamma _a}\max _{t\in [0,1]}E_\mu (h(t)) \end{aligned}$$

with

$$\begin{aligned} \Gamma _a=\left\{ h\in C\left( [0,1],S_{a,r}\right) :h(0)=\hat{u}, h(1)=\tilde{u}\right\} , \end{aligned}$$

then we have

$$\begin{aligned} \widetilde{\gamma }_a=\gamma _a\ge \max \{E_\mu (\hat{u}),E_\mu (\tilde{u})\}:= \delta _a>0. \end{aligned}$$

Proof

First note that the existence of $\hat{u},\tilde{u} \in S_{a,r}$ is insured by Lemma 5.1 and 5.2. Next, for any $\widetilde{h}\in \widetilde{\Gamma }_a$, we can write it into

$$\begin{aligned} \widetilde{h}(t)=\left( \widetilde{h}_1(t),\widetilde{h}_2(t)\right) \in S_{a,r}\times \mathbb {R}. \end{aligned}$$

Setting $h(t)=\widetilde{h}_2(t)\star \widetilde{h}_1(t)$, we have $h(t)\in \Gamma _a$ and

$$\begin{aligned} \max _{t\in [0,1]}I_\mu (\widetilde{h}(t))=\max _{t\in [0,1]}E_\mu \left( \widetilde{h}_2(t)\star \widetilde{h}_1(t)\right) =\max _{t\in [0,1]}E_\mu (h(t)), \end{aligned}$$

which implies $\widetilde{\gamma }_a\ge \gamma _a$. On the other hand, for any $h\in \Gamma _a$, if we set $\widetilde{h}(t)=\left( h(t),0\right)$, then we get $\widetilde{h}\in \widetilde{\Gamma }_a$ and

$$\begin{aligned} \max _{t\in [0,1]}I_\mu (\widetilde{h}(t))=\max _{t\in [0,1]}E_\mu (h(t)). \end{aligned}$$

This provides that $\gamma _a\ge \widetilde{\gamma }_a$. Thus, we have $\widetilde{\gamma }_a=\gamma _a$. Finally, $\gamma _a\ge \max \{E_\mu (\hat{u}),E_\mu (\tilde{u})\}$ follows from the definition of $\gamma _a$. $\square$

In what follows, we give the relationship between the Palais–Smale sequence for $I_\mu$ and that of $E_\mu$.

Proposition 5.4

Let $\widetilde{\gamma }_a$ and $\gamma _a$ be defined in Proposition 5.3. Then there exists a sequence $\left\{ (v_n,\tau _n)\right\} \subset S_{a,r}\times \mathbb {R}$ such that for $n\rightarrow \infty$, we have

• (1) $I_\mu (v_n,\tau _n)\rightarrow \widetilde{\gamma }_a$,

• (2) $I'_\mu |_{S_{a,r}\times \mathbb {R}}(v_n,\tau _n)\rightarrow 0$, i.e., it holds that

  $$\begin{aligned} \partial _{\tau }I_\mu (v_n,\tau _n)\rightarrow 0, \end{aligned}$$

  and

  $$\begin{aligned} \left\langle \partial _{u}I_\mu (v_n,\tau _n),\widetilde{\varphi }\right\rangle \rightarrow 0 \end{aligned}$$

  with

  $$\begin{aligned} \widetilde{\varphi }\in T_{v_n}=\left\{ \widetilde{\varphi }\in H^s(\mathbb {R}^N):\int _{\mathbb {R}^N}v_n\widetilde{\varphi }=0\right\} . \end{aligned}$$

In addition, setting $u_n(x)=\tau _n\star v_n(x)$, then for $n\rightarrow \infty$ we get

• (i) $E_\mu (u_n)\rightarrow \gamma _a$,

• (ii) $P_\mu (u_n)\rightarrow 0$,

• (iii) $E'_\mu |_{S_{a,r}}(u_n)\rightarrow 0$, i.e., it holds that

  $$\begin{aligned} \left\langle E'_\mu (u_n),\varphi \right\rangle \rightarrow 0 \end{aligned}$$

  with

  $$\begin{aligned} \varphi \in T_{u_n}=\left\{ \varphi \in H^s(\mathbb {R}^N):\int _{\mathbb {R}^N}u_n\varphi =0\right\} . \end{aligned}$$

Proof

According to the construction of $\widetilde{\gamma }_a$, we know that the conclusions (1) and (2) follow directly from the Ekeland’s Variational Principle. Next we mainly show (i)-(iii).

For (i), it is obvious if we notice that

$$\begin{aligned} E_\mu (u_n)=E_\mu (\tau _n\star v_n)=I_\mu (v_n,\tau _n) \end{aligned}$$

and $\widetilde{\gamma }_a=\gamma _a$.

For (ii), we first have

$$\begin{aligned} \partial _{\tau }I_\mu (v_n,\tau _n)= & {} se^{2s\tau _n}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}v_n|^2-\frac{N(p-2)}{2p}e^{N\tau _n(\frac{p}{2}-1)}\int _{\mathbb {R}^N}|v_n|^p\\&-\mu \frac{N(q-2)}{2q}e^{N\tau _n(\frac{q}{2}-1)}\int _{\mathbb {R}^N}|v_n|^q\\= & {} s\left[ \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau _n\star v_n)|^2-\frac{N(p-2)}{2ps}\int _{\mathbb {R}^N}|\tau _n\star v_n|^p\right. \\&\qquad \left. -\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|\tau _n\star v_n|^q\right] \\= & {} s\left[ \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2-\frac{N(p-2)}{2ps}\int _{\mathbb {R}^N}|u_n|^p -\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|u_n|^q\right] \\= & {} sP_\mu (u_n). \end{aligned}$$

Thus, (ii) is a consequence of $\partial _{\tau }I_\mu (v_n,\tau _n)\rightarrow 0$ as $n\rightarrow \infty$.

For (iii), by the definition of $I_\mu$, we have

$$\begin{aligned} \left\langle \partial _{u}I_\mu (v_n,\tau _n),\widetilde{\varphi }\right\rangle= & {} e^{2s\tau _n}\iint _{\mathbb {R}^{2N}}\frac{(v_n(x)-v_n(y))\left( \widetilde{\varphi }(x)- \widetilde{\varphi }(y)\right) }{|x-y|^{N+2s}}dxdy\\&-e^{N\tau _n\left( \frac{p}{2}-1\right) }\int _{\mathbb {R}^N}|v_n|^{p-2}v_n\widetilde{\varphi }-\mu e^{N\tau _n\left( \frac{q}{2}-1\right) }\int _{\mathbb {R}^N}|v_n|^{q-2}v_n\widetilde{\varphi }, \end{aligned}$$

where

$$\begin{aligned} \widetilde{\varphi }\in T_{v_n}=\left\{ \widetilde{\varphi }\in H^s(\mathbb {R}^N):\int _{\mathbb {R}^N}v_n\widetilde{\varphi }=0\right\} . \end{aligned}$$

On the other hand, for any $\varphi$ with satisfying

$$\begin{aligned} \varphi \in T_{u_n}=\left\{ \varphi \in H^s(\mathbb {R}^N):\int _{\mathbb {R}^N}u_n\varphi =0\right\} , \end{aligned}$$

we have

$$\begin{aligned} \left\langle E'_\mu (u_n),\varphi \right\rangle= & {} \iint _{\mathbb {R}^{2N}}\frac{(u_n(x)-u_n(y))\left( \varphi (x)- \varphi (y)\right) }{|x-y|^{N+2s}}dxdy\\&-\int _{\mathbb {R}^N}|u_n|^{p-2}u_n\varphi -\mu \int _{\mathbb {R}^N}|u_n|^{q-2}u_n\varphi \\= & {} \iint _{\mathbb {R}^{2N}}\frac{e^{\frac{N\tau _n}{2}}(v_n\left( e^{\tau _n}x\right) -v_n\left( e^{\tau _n}y\right) )\left( \varphi (x)- \varphi (y)\right) }{|x-y|^{N+2s}}dxdy\\&-\int _{\mathbb {R}^N}e^{\frac{N\tau _n(p-1)}{2}}|v_n\left( e^{\tau _n}x\right) |^{p-2}v_n\left( e^{\tau _n}x\right) \varphi (x)\\&-\mu \int _{\mathbb {R}^N}e^{\frac{N\tau _n(q-1)}{2}}|v_n\left( e^{\tau _n}x\right) |^{q-2}v_n\left( e^{\tau _n}x\right) \varphi (x)\\= & {} e^{2s\tau _n}\iint _{\mathbb {R}^{2N}}\frac{(v_n(x)-v_n(y))\left( e^{-\frac{N\tau _n}{2}}\varphi \left( e^{-\tau _n}x\right) - e^{-\frac{N\tau _n}{2}}\varphi \left( e^{-\tau _n}y\right) \right) }{|x-y|^{N+2s}}dxdy\\&-e^{N\tau _n\left( \frac{p}{2}-1\right) }\int _{\mathbb {R}^N}|v_n(x)|^{p-2}v_n(x)e^{-\frac{N\tau _n}{2}}\varphi \left( e^{-\tau _n}x\right) \\&-\mu e^{N\tau _n\left( \frac{q}{2}-1\right) }\int _{\mathbb {R}^N}|v_n(x)|^{q-2}v_n(x)e^{-\frac{N\tau _n}{2}}\varphi \left( e^{-\tau _n}x\right) . \end{aligned}$$

Setting

$$\begin{aligned} \widetilde{\varphi }(x)=e^{-\frac{N\tau _n}{2}}\varphi \left( e^{-\tau _n}x\right) , \end{aligned}$$

we get (iii) if we could show that $\widetilde{\varphi }\in T_{v_n}$. In fact, $\widetilde{\varphi }\in T_{v_n}$ follows from the following equalities:

$$\begin{aligned} 0= & {} \int _{\mathbb {R}^N}u_n\varphi =\int _{\mathbb {R}^N}e^{\frac{N\tau _n}{2}}v_n\left( e^{\tau _n}x\right) \varphi (x)\\= & {} \int _{\mathbb {R}^N}v_n(x)e^{-\frac{N\tau _n}{2}}\varphi \left( e^{-\tau _n}x\right) =\int _{\mathbb {R}^N}v_n\widetilde{\varphi }. \end{aligned}$$

$\square$

Lemma 5.5

Let $\bar{p}<q<p<2_s^*,\mu >0$. Let $\left\{ u_n\right\} \subset S_{a,r}$ be a Palais–Smale sequence for $E_\mu |_{S_{a,r}}$ at level $\gamma _a\ne 0$, and suppose in addition that $P_\mu (u_n)\rightarrow 0$ as $n\rightarrow \infty$. Then up to a subsequence $u_n\rightarrow u$ strongly in $H^s(\mathbb {R}^N)$, and $u\in S_{a,r}$ is a radial solution to (1.3) for some $\lambda <0$.

Proof

We divide the proof into four main steps.

Step 1: Boundedness of $\left\{ u_n\right\}$ in $H^s(\mathbb {R}^N)$. As $P_\mu (u_n)\rightarrow 0$, we have

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2=\frac{N(p-2)}{2ps}|u_n|_p^p +\mu \frac{N(q-2)}{2qs}|u_n|_q^q+o(1)\quad \text{ as } n\rightarrow \infty . \end{aligned}$$

(5.1)

Thus, by (5.1) we deduce that

$$\begin{aligned} \frac{1}{p}\left( \frac{N(p-2)}{4s}-1\right) |u_n|_p^p+\frac{\mu }{q}\left( \frac{N(q-2)}{4s}-1\right) |u_n|_q^q+o(1)=E_\mu (u_n)\le \gamma _a+1 \end{aligned}$$

for large n. Since $\bar{p}<q<p<2_s^*$, it implies that $\frac{N(p-2)}{4s}-1>0$ and $\frac{N(q-2)}{4s}-1>0$. Since $\mu >0$, then we can deduce the boundedness of $|u_n|_p$ and $|u_n|_q$. Once again by (5.1) we obtain the boundedness of $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2$.

Step 2: Since $N\ge 2$, the embedding $H_r^s(\mathbb {R}^N)\hookrightarrow L^t(\mathbb {R}^N)$ is compact for $t\in (2,2_s^*)$, and we deduce that there exists $u\in H_r^s(\mathbb {R}^N)$ such that, up to a subsequence, $u_n\rightharpoonup u$ weakly in $H^s(\mathbb {R}^N)$, $u_n\rightarrow u$ strongly in $L^t(\mathbb {R}^N)$ for $t\in (2,2_s^*)$, and a.e. in $\mathbb {R}^N$. Now, since $\left\{ u_n\right\} \subset S_{a,r}$ is a Palais–Smale sequence for $E_\mu |_{S_{a,r}}$, by the Lagrange multipliers rule there exist $\left\{ \lambda _n\right\} \subset \mathbb {R}$ such that

$$\begin{aligned} \left( -\Delta \right) ^su_n-|u_n|^{p-2}u_n-\mu |u_n|^{p-2}u_n=\lambda _n u_n+o(1) \quad \text{ as }~n\rightarrow \infty . \end{aligned}$$

(5.2)

Testing the equation above against $u_n$, we obtain

$$\begin{aligned} \lambda _n a^2=\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2-\int _{\mathbb {R}^N}|u_n|^p -\mu \int _{\mathbb {R}^N}|u_n|^q+o(1), \end{aligned}$$

and the boundedness of $\left\{ u_n\right\}$ in $H^s\cap L^p\cap L^q$ implies that $\left\{ \lambda _n\right\}$ is bounded as well; up to a subsequence $\lambda _n\rightarrow \lambda \in \mathbb {R}$.

Step 3: $\lambda <0$. We first claim that $u\not \equiv 0$. We assume by contradiction that $u\equiv 0$, then $|u_n|_p\rightarrow 0$ and $|u_n|_q\rightarrow 0$. Recalling that $P_\mu (u_n)\rightarrow 0$, we have

$$\begin{aligned} E_\mu (u_n)=\frac{1}{p}\left( \frac{N(p-2)}{4s}-1\right) |u_n|_p^p+\frac{\mu }{q}\left( \frac{N(q-2)}{4s}-1\right) |u_n|_q^q+o(1), \end{aligned}$$

and hence $E_\mu (u_n)\rightarrow 0$, in contradiction with the assumption that $E_\mu (u_n)\rightarrow \gamma _a\ne 0$. Now, since $\lambda _n\rightarrow \lambda$ and $u_n\rightarrow u\ne 0$ weakly in $H^s(\mathbb {R}^N)$, together with (5.2), we know u is a radial solution to (1.3). By the Pohozaev identity, we obtain

$$\begin{aligned} \frac{N-2s}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=N\int _{\mathbb {R}^N}\left( \frac{1}{p}|u|^p+\frac{\mu }{q}|u|^q+\frac{\lambda }{2}|u|^2\right) . \end{aligned}$$

Combining with the Eq. (1.3) for u, we get

$$\begin{aligned} \lambda a^2=\lambda |u|_2^2=\frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2+\mu \frac{2(q-p)}{q(p-2)}|u|_q^q. \end{aligned}$$

(5.3)

Since $\mu >0$, we know $\lambda <0$ by (5.3).

Step 4: $u_n\rightarrow u$ strongly in $H^s(\mathbb {R}^N)$. Testing Eq. (5.1) and (1.3) with $u_n-u$, and subtracting, we obtain

$$\begin{aligned} \left\langle E'_\mu (u_n)-E'_\mu (u),u_n-u\right\rangle -\lambda \int _{\mathbb {R}^N}|u_n-u|^2=o(1). \end{aligned}$$

Using the strong $L^p$ and $L^q$ convergence of $u_n$, we infer that

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(u_n-u)|^2-\lambda \int _{\mathbb {R}^N}|u_n-u|^2=o(1), \end{aligned}$$

which, being $\lambda <0$, establishes the strong convergence in $H^s(\mathbb {R}^N)$. $\square$

Remark 5.6

If we check the proof above carefully, we would find that Lemma 5.5 holds for the case $2<q\le \bar{p}<p<2_s^*$ and $\mu >0$. We only need to modify some details of Step 1 as follows: If $2<q\le \bar{p}<p<2_s^*$, by the Hölder inequality there exists $\theta \in (0,1)$ such that $|u_n|_q\le |u_n|_p^\theta |u_n|_2^{1-\theta }=|u_n|_p^\theta a^{1-\theta }$. Then we have

$$\begin{aligned}&\frac{1}{p}\left( \frac{N(p-2)}{4s}-1\right) |u_n|_p^p+\frac{\mu }{q}\left( \frac{N(q-2)}{4s}-1\right) |u_n|_q^q\\&\quad \ge \frac{1}{p}\left( \frac{N(p-2)}{4s}-1\right) |u_n|_p^p+\frac{\mu }{q}\left( \frac{N(q-2)}{4s}-1\right) |u_n|_p^{q\theta } a^{q(1-\theta )}, \end{aligned}$$

Note that $p>q>q\theta$, we can deduce that $|u_n|_p$ is bounded and hence $|u_n|_q$ is bounded. By (5.1), we obtain the boundedness of $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_n|^2$. The rest of details can be easily modified and hence is omitted.

With these preparations above at hand, we now prove Theorem 1.5 and Corollary 1.6.

Proof of Theorem 1.5 and Corollary 1.6

First, we see that $\inf _{S_a}E_\mu =-\infty$ follows from Lemma 5.1. For $\mu >0$, we can define $\widetilde{\gamma }_a$ and $\gamma _a$ as Proposition 5.3. By Proposition 5.4, we obtain a Palais–Smale sequence $\left\{ u_n\right\} \subset S_{a,r}$ for $E_\mu |_{S_{a,r}}$ at level $\gamma _a>0$, and have $P_\mu (u_n)\rightarrow 0$ as $n\rightarrow \infty$. In virtue of Lemma 5.5, we know $u_n\rightarrow u_a$ strongly in $H^s(\mathbb {R}^N)$, and $u_a\in S_{a,r}$ is a radial solution to (1.3) for some $\lambda _a<0$.

Since $(u_a,\lambda _a)$ is a solution of (1.3), by the Pohozaev identity and the fractional GNS inequality, we have

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2= & {} \frac{N(p-2)}{2ps}\int _{\mathbb {R}^N}|u_a|^p+\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|u_a|^q\\\le & {} C_1 a^{p-\frac{N(p-2)}{2s}}\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2\right) ^{\frac{N(p-2)}{4s}}\\&+\mu C_2a^{q-\frac{N(q-2)}{2s}}\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2\right) ^{\frac{N(q-2)}{4s}}, \end{aligned}$$

where $C_1,C_2$ are constants depending only on N, s, p, q. Recall that $\bar{p}<q<p<2_s^*$, we know $\frac{N(p-2)}{4s}>\frac{N(q-2)}{4s}>1$. Thus, we get

$$\begin{aligned} 1\le C_1 a^{p-\frac{N(p-2)}{2s}}\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2\right) ^{\frac{N}{4s}(p-\bar{p})}+\mu C_2a^{q-\frac{N(q-2)}{2s}}\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2\right) ^{\frac{N}{4s}(q-\bar{p})}, \end{aligned}$$

which implies

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2\rightarrow +\infty \end{aligned}$$

as $a\rightarrow 0$. On the other hand, by the Pohozaev identity, we also have

$$\begin{aligned} \lambda _a a^2=\lambda _a|u_a|_2^2= & {} \frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2+\mu \frac{2(q-p)}{q(p-2)}|u_a|_q^q\\\le & {} \frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2. \end{aligned}$$

Thus, it results that $\lambda _a\rightarrow -\infty$ as $a\rightarrow 0$. $\square$

6 Combined-type cases

In this section, we consider the case $2<q\le \bar{p}=2+\frac{4s}{N}\le p<2_s^*,p\ne q$ and $\mu \in \mathbb {R}$.

6.1 $L^2$-critical leading term

In this subsection, we deal with the case $2<q<p=\bar{p},\mu \in \mathbb {R}$ and prove Theorem 1.7.

Lemma 6.1

Let $2<q<p=\bar{p}$. If $0<a<\bar{a}$, then for any $\mu >0$ we have

$$\begin{aligned} -\infty<m_{a,\mu }=\inf _{S_a}E_\mu <0; \end{aligned}$$

while for any $\mu <0$, we obtain

$$\begin{aligned} \inf _{S_a}E_\mu =0. \end{aligned}$$

Proof

By the fractional GNS inequality and (2.1), we get

$$\begin{aligned} E_\mu (u)\ge & {} \frac{1}{2}\left( 1-\left( \frac{a}{\bar{a}}\right) ^{\frac{4s}{N}}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\\&-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}}\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(q-2)}{4s}} \end{aligned}$$

for any $u\in S_a$. Since $a<\bar{a},N(q-2)/4s<1$, if $\mu >0$, we know that $E_\mu$ is coercive on $S_a$, and $m_{a,\mu }=\inf _{S_a}E_\mu >-\infty$. If $\mu <0$, then it holds $\inf _{S_a}E_\mu \ge 0$. On the other hand, it holds for every $u\in S_a$ that

$$\begin{aligned} E_\mu (\tau \star u)=\frac{e^{2s\tau }}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{e^{2s\tau }}{\bar{p}}\int _{\mathbb {R}^N}|u|^{\bar{p}}- \mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|u|^q, \end{aligned}$$

since $\mu >0$ and $N(q/2-1)<2s$, we have that $E_\mu (\tau \star u)<0$ for every $(\tau ,u)\in \mathbb {R}\times S_a$ with $\tau \ll -1$. Thus, it results that $m_{a,\mu }<0$ for $\mu >0$. For $\mu <0$, we easily see that $E_\mu (\tau \star u)\rightarrow 0$ as $\tau \rightarrow -\infty$, which implies $\inf _{S_a}E_\mu =0$. $\square$

Lemma 6.2

Let $2<q<p=\bar{p}$ and $\mu >0$. Let $a_1,a_2>0$ be such that $a_1^2+a_2^2=a^2<\bar{a}^2$. Then

$$\begin{aligned} m_{a,\mu }<m_{a_1,\mu }+m_{a_2,\mu }. \end{aligned}$$

Proof

We can proceed exactly as in the proof of Lemma 3.2 and omit the details. $\square$

Corollary 6.3

Let $2<q<p=\bar{p}$ and $\mu >0$, then $m_a$ is strictly decreasing in $a\in (0,\bar{a})$.

Proof of Theorem 1.7

For (i)-(a), since having established Lemmas 6.1, 6.2 and Corollary 6.3, we can follow a similar argument as the proof for the existence of a minimizer of (i) of Theorem 1.2. Here we omit the details.

For (i)-(b), $\inf _{S_a}E_\mu =0$ follows from Lemma 6.1. We assume by contradiction that problem (1.3)–(1.4) has a solution $u_a\in S_a$, by the Pohozaev identity we deduce that

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u_a|^2=\frac{2}{\bar{p}}\int _{\mathbb {R}^N}|u_a|^{\bar{p}}+\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|u_a|^q. \end{aligned}$$

Recall that the (ii)-(a) of Theorem 1.2, we have $\inf _{S_a}E_0=0$, and hence we get

$$\begin{aligned} 0>\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|u_a|^q=2E_0(u)\ge 2\inf _{S_a}E_0=0, \end{aligned}$$

a contradiction.

For (ii)-(a), if $a=\bar{a}$ and $p=\bar{p}$, according to the (ii)-(b) of Theorem 1.2, there exists a $Q_{N,\bar{p}}\in S_{\bar{a}}$ such that $E_0(Q_{N,\bar{p}})=0$. For $\tau \in \mathbb {R}$, we have

$$\begin{aligned} E_\mu (\tau \star Q_{N,\bar{p}})=e^{2s\tau }E_0(Q_{N,\bar{p}})-\mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^q=-\mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|Q_{N,\bar{p}}|^q. \end{aligned}$$

Since $\mu >0$ and $q<\bar{p}$, we know $E_\mu (\tau \star Q_{N,\bar{p}})\rightarrow -\infty$ as $\tau \rightarrow +\infty$, which provides that

$$\begin{aligned} \inf _{S_a}E_\mu =-\infty . \end{aligned}$$

For (ii)-(b), the proof follows a similar argument of (i)-(b) in this theorem, we omit the details.

For (iii), let $u_a=\frac{a}{\bar{a}}Q_{N,\bar{p}}$, by a direct computation, we get

$$\begin{aligned} |u_a|_2^2=a^2,\quad \text{ and }~~E_0(u_a)<0. \end{aligned}$$

For $\tau \in \mathbb {R}$, we have

$$\begin{aligned} E_\mu (\tau \star u_a)=e^{2s\tau }E_0(u_a)-\mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|u_a|^q. \end{aligned}$$

Since $q<\bar{p}$, we know that $N(\frac{q}{2}-1)<2s$ and hence $E_\mu (\tau \star u_a)\rightarrow -\infty$ as $\tau \rightarrow +\infty$, which gives our desired result. $\square$

6.2 Supercritical leading term with subcritical perturbation

In this subsection, we deal with the case $2<q<\bar{p}<p<2_s^*,\mu \in \mathbb {R}$ and prove Theorems 1.8 and 1.9. For convenience, we give some notations.

$$\begin{aligned} S_{a,r}= & {} S_a\cap H^s_r(\mathbb {R}^N)=\left\{ u\in S_a: u(x)=u(|x|)\right\} ,\\ \mathcal {P}_\mu= & {} \left\{ u\in H^s(\mathbb {R}^N):P_\mu (u)=0\right\} \end{aligned}$$

with

$$\begin{aligned} P_\mu (u)= & {} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{N(p-2)}{2ps}\int _{\mathbb {R}^N}|u|^p-\mu \frac{N(q-2)}{2qs}\int _{\mathbb {R}^N}|u|^q.\\ \mathcal {P}_{a,\mu }= & {} S_{a,r}\cap \mathcal {P}_\mu =\left\{ u\in S_{a,r}: P_\mu (u)=0\right\} . \end{aligned}$$

For any $u\in S_{a,r}$, we introduce the fiber map

$$\begin{aligned} \Psi _u^\mu (\tau ):=E_\mu (\tau \star u)=\frac{e^{2s\tau }}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{e^{N\tau (\frac{p}{2}-1)}}{p}\int _{\mathbb {R}^N}|u|^p- \mu \frac{e^{N\tau (\frac{q}{2}-1)}}{q}\int _{\mathbb {R}^N}|u|^q, \end{aligned}$$

it is easy to verify that any critical point of $\Psi _u^\mu$ belongs to $\mathcal {P}_{a,\mu }$. Conversely, if $u\in \mathcal {P}_{a,\mu }$, we get $\left( \Psi _u^\mu \right) '(0)=0$. Thus, we consider the decomposition of $\mathcal {P}_{a,\mu }$ into the disjoint union $\mathcal {P}_{a,\mu }=\mathcal {P}_{a,\mu }^+\cup \mathcal {P}_{a,\mu }^0\cup \mathcal {P}_{a,\mu }^-$, where

$$\begin{aligned} \mathcal {P}_{a,\mu }^+&:=\left\{ u\in \mathcal {P}_{a,\mu }:2s^2\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2>\mu q\gamma _q^2|u|_q^q+p\gamma _p^2|u|_p^p\right\} \\&=\left\{ u\in \mathcal {P}_{a,\mu }:\left( \Psi _u^\mu \right) ''(0)>0\right\} \\ \mathcal {P}_{a,\mu }^0&:=\left\{ u\in \mathcal {P}_{a,\mu }:2s^2\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\mu q\gamma _q^2|u|_q^q+p\gamma _p^2|u|_p^p\right\} \\&=\left\{ u\in \mathcal {P}_{a,\mu }:\left( \Psi _u^\mu \right) ''(0)=0\right\} \\ \mathcal {P}_{a,\mu }^-&:=\left\{ u\in \mathcal {P}_{a,\mu }:2s^2\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<\mu q\gamma _q^2|u|_q^q+p\gamma _p^2|u|_p^p\right\} \\&=\left\{ u\in \mathcal {P}_{a,\mu }:\left( \Psi _u^\mu \right) ''(0)<0\right\} \end{aligned}$$

and

$$\begin{aligned} \gamma _q:=\frac{N(q-2)}{2q},\quad \gamma _p:=\frac{N(p-2)}{2p}. \end{aligned}$$

In what follows, we discuss according to the sign of $\mu$.

(a) $\mu >0$. We consider the constrained functional $E_\mu |_{S_{a,r}}$. By the fractional GNS inequality

$$\begin{aligned} E_\mu (u)\ge & {} \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}\nonumber \\&-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(q-2)}{4s}}, \end{aligned}$$

(6.1)

for every $u\in S_{a,r}$. Therefore, to understand the geometry of the functional $E_\mu |_{S_{a,r}}$ it is useful to consider the function $h:\mathbb {R}^+\rightarrow \mathbb {R}$

$$\begin{aligned} h(t):=\frac{1}{2}t-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} t^{\frac{N(p-2)}{4s}}-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}}t^{\frac{N(q-2)}{4s}}. \end{aligned}$$

Since $\mu >0$ and $\frac{N(q-2)}{4s}<1<\frac{N(p-2)}{4s}$, we have that $h(0^+)=0^-$ and $h(+\infty )=-\infty$.

Lemma 6.4

Let $a,\mu >0$ satisfy (1.10), the function h has a local strict minimum at negative level and a global strict maximum at positive level. Moreover, there exist $0<R_0 < R_1$, both depending on a and $\mu$, such that $h(R_0)=0=h(R_1)$ and $h(t)>0$ iff $t\in (R_0, R_1)$.

Proof

Since

$$\begin{aligned} h(t)=t^{\frac{N(q-2)}{4s}}\left[ \frac{1}{2}t^{\frac{N(\bar{p}-q)}{4s}}-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} t^{\frac{N(p-q)}{4s}}-\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}}\right] , \end{aligned}$$

for $t>0$, we have $h(t)>0$ if and only if

$$\begin{aligned} \varphi (t)>\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}},\quad \text{ with } \varphi (t):=\frac{1}{2}t^{\frac{N(\bar{p}-q)}{4s}}-\frac{C(s,N,p)}{p}a^{p-\frac{N(p-2)}{2s}} t^{\frac{N(p-q)}{4s}}. \end{aligned}$$

It is not difficult to check that $\varphi$ has a unique critical point on $(0,\infty )$, which is a global maximum point at positive level, in

$$\begin{aligned} \bar{t}:=C_1a^{\frac{4p-p\bar{p}-4}{p-\bar{p}}},\quad \text{ with } C_1:=\left( \frac{p(\bar{p}-q)}{2C(s,N,p)(p-q)}\right) ^{\frac{4s}{N(p-\bar{p})}}; \end{aligned}$$

the maximum level is

$$\begin{aligned} \varphi (\bar{t})=C_2 a^{\frac{N(4p-p\bar{p}-4)(\bar{p}-q)}{4s(p-\bar{p})}},\quad \text{ with } C_2:=\left( \frac{p(\bar{p}-q)}{2C(s,N,p)(p-q)}\right) ^{\frac{\bar{p}-q}{p-\bar{p}}}\frac{p-\bar{p}}{2(p-q)}. \end{aligned}$$

Therefore, h is positive on an open interval $(R_0, R_1)$ iff $\varphi (\bar{t})>\mu \frac{C(s,N,q)}{q}a^{q-\frac{N(q-2)}{2s}}$, which is ensured by (1.10). It follows immediately that h has a global maximum at positive level in $(R_0, R_1)$ (In fact, a continuous function on a bounded closed interval must admit the maximum value.) Moreover, since $h(0^+)=0^-$, there exists a local minimum point at negative level in $(0,R_0)$. The fact that h has no other critical points can be verified observing that $h'(t)=0$ if and only if

$$\begin{aligned} \psi (t)=\mu \frac{C(s,N,q)\gamma _q}{s}a^{q-\frac{N(q-2)}{2s}},\quad \text{ with }~\psi (t):=t^{\frac{N(\bar{p}-q)}{4s}}-\frac{C(s,N,p)\gamma _p}{s}a^{p-\frac{N(p-2)}{2s}} t^{\frac{N(p-q)}{4s}}. \end{aligned}$$

Clearly $\psi$ has only one critical point, which is a strict maximum, and hence the above equation has at most two solutions, which necessarily are the local minimum and the global maximum of h previously found. $\square$

We now study the structure of the Pohozaev manifold $\mathcal {P}_{a,\mu }$. Recalling the decomposition of $\mathcal {P}_{a,\mu }=\mathcal {P}_{a,\mu }^+\cup \mathcal {P}_{a,\mu }^0\cup \mathcal {P}_{a,\mu }^-$.

Lemma 6.5

$\mathcal {P}_{a,\mu }^0=\emptyset$, and $\mathcal {P}_{a,\mu }$ is a smooth manifold of codimension 2 in $H^s(\mathbb {R}^N)$.

Proof

The proof can follow a similar argument as the proof of [48, Lemma 5.2], here we omit the details. $\square$

The manifold $\mathcal {P}_{a,\mu }$ is then divided into two components $\mathcal {P}_{a,\mu }^+$ and $\mathcal {P}_{a,\mu }^-$, having disjoint closure.

Lemma 6.6

For every $u\in S_{a,r}$, the function $\Psi _u^\mu$ has exactly two critical points $s_u<t_u\in \mathbb {R}$ and two zeros $c_u<d_u\in \mathbb {R}$, with $s_u<c_u<t_u<d_u$. Moreover:

• (1) $s_u\star u\in \mathcal {P}_{a,\mu }^+$, and $t_u\star u\in \mathcal {P}_{a,\mu }^-$, and if $s\star u\in \mathcal {P}_{a,\mu }$, then either $s=s_u$ or $s=t_u$.

• (2) $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2\le R_0$ for every $s\le c_u$, and

  $$\begin{aligned} E_\mu (s_u\star u)=\min \left\{ E_\mu (\tau \star u):\tau \in \mathbb {R} ~\text{ and }~\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2< R_0\right\} <0. \end{aligned}$$

  (6.2)

• (3) We have

  $$\begin{aligned} E_\mu (t_u\star u)=\max \left\{ E_\mu (\tau \star u):\tau \in \mathbb {R} \right\} >0, \end{aligned}$$

  (6.3)

  and $\Psi _u^\mu$ is strictly decreasing and concave on $(t_u,+\infty )$.

• (4) The maps $u\in S_{a,r}\mapsto s_u\in \mathbb {R}$ and $u\in S_{a,r}\mapsto t_u\in \mathbb {R}$ are of class $C^1$.

Proof

Let $u\in S_{a,r}$, by a direct computation, we know $\tau \star u\in \mathcal {P}_{a,\mu }$ if and only if $\left( \Psi _u^\mu \right) '(\tau )=0$. Thus, we first show that $\Psi _u^\mu$ has at least two critical points. To this end, we recall that by (6.1)

$$\begin{aligned} \Psi _u^\mu (\tau )=E_\mu (\tau \star u)\ge h(\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2)=h(e^{2s\tau }\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}} u|^2). \end{aligned}$$

Thus, the $C^2$ functional $\Psi _u^\mu$ is positive on $(C(R_0),C(R_1))$ with

$$\begin{aligned} (C(R_0),C(R_1)):=\left( \frac{1}{2s}\log \left( R_0/\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}} u|^2\right) ,\frac{1}{2s}\log \left( R_1/\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}} u|^2\right) \right) , \end{aligned}$$

and clearly $\Psi _u^\mu (-\infty )=0^-,\Psi _u^\mu (+\infty )=-\infty$. It follows that $\Psi _u^\mu$ has at least two critical points $s_u<t_u$, with $s_u$ local minimum point on $\left( 0,C(R_0)\right)$ at negative level, and $t_u>s_u$ global maximum point at positive level. It is not difficult to check that there are no other critical points. Indeed $\left( \Psi _u^\mu \right) '(\tau )=0$ reads

$$\begin{aligned} \varphi (\tau )=\mu \gamma _q|u|_q^q,\quad \text{ with } \varphi (\tau ):=se^{\frac{N(\bar{p}-q)\tau }{2}}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}} u|^2-\gamma _pe^{\frac{N(p-q)\tau }{2}}|u|_p^p. \end{aligned}$$

(6.4)

But $\varphi$ has a unique maximum point, and hence Eq. (6.4) has at most two solutions.

Collecting together the above considerations, we conclude that $\Psi _u^\mu$ has exactly two critical points: $s_u$, local minimum on $(-\infty ,C(R_0))$ at negative level, and $t_u$, global maximum at positive level, which gives (6.3). To see (6.2), since being $s_u<C(R_0)$, then it holds that

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(s_u\star u)|^2=e^{2ss_u}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}} u|^2<R_0. \end{aligned}$$

In addition, we have $s_u\star u\in \mathcal {P}_{a,\mu }, t_u\star u\in \mathcal {P}_{a,\mu }$ and $\tau \star u\in \mathcal {P}_{a,\mu }$ implies $\tau \in \left\{ s_u,t_u\right\}$. By minimality $\left( \Psi _u^\mu \right) ''(s_u)\ge 0$, and in fact strict inequality must hold, since $\mathcal {P}_{a,\mu }^0=\emptyset$ by Lemma 6.5; namely $s_u\star u\in \mathcal {P}_{a,\mu }^+$. In the same way $t_u\star u\in \mathcal {P}_{a,\mu }^-$.

By monotonicity and recalling the behavior at infinity, $\Psi _u^\mu$ has moreover exactly two zeros $c_u<d_u$, with $s_u<c_u<t_u<d_u$; and, being a $C^2$ function, $\Psi _u^\mu$ has at least two inflection points. Arguing as before, we can easily check that actually $\Psi _u^\mu$ has exactly two inflection points. In particular, $\Psi _u^\mu$ is concave on $[t_u,+\infty )$. It remains to show that $u\mapsto s_u$ and $u\mapsto t_u$ are of class $C^1$; to this end, we apply the implicit function theorem on the $C^1$ function $\Phi (\tau ,u):= \left( \Psi _u^\mu \right) '(\tau )$. We use that $\Phi (\tau ,u)=0$, that $\partial _s\Phi (s_u,u)=\left( \Psi _u^\mu \right) ''(s_u)>0$, and the fact that it is not possible to pass with continuity from $\mathcal {P}_{a,\mu }^+$ to $\mathcal {P}_{a,\mu }^-$ (since $\mathcal {P}_{a,\mu }^0=\emptyset$ ). The same argument proves that $u\mapsto t_u$ is $C^1$. $\square$

From the proof of Lemma 6.6, we see that $s_u<C(R_0)<t_u$ and hence

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(s_u\star u)|^2<R_0< \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(t_u\star u)|^2, \end{aligned}$$

which implies

$$\begin{aligned} \mathcal {P}_{a,\mu }^+\subseteq \left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<R_0\right\} \end{aligned}$$

and

$$\begin{aligned} \mathcal {P}_{a,\mu }^-\subseteq \left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2>R_0\right\} . \end{aligned}$$

For $k>0$, let us set

$$\begin{aligned} \mathcal {A}_k:=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<k\right\} , \end{aligned}$$

and

$$\begin{aligned} M_{a,\mu }:=\inf _{u\in \mathcal {A}_{R_0}}E_\mu (u). \end{aligned}$$

As an immediate corollary, we have:

Corollary 6.7

$\sup _{\mathcal {P}_{a,\mu }^+}E_\mu \le 0\le \inf _{\mathcal {P}_{a,\mu }^-}E_\mu$.

Furthermore:

Lemma 6.8

It results that $M_{a,\mu }\in (-\infty ,0)$, that

$$\begin{aligned} M_{a,\mu }=\inf _{\mathcal {P}_{a,\mu }}E_\mu =\inf _{\mathcal {P}_{a,\mu }^+}E_\mu ,\quad \text{ and } \text{ that }~M_{a,\mu }<\inf _{\overline{\mathcal {A}_{R_0}}{\setminus }\mathcal {A}_{R_0-\rho }}E_\mu \end{aligned}$$

for $\rho >0$ small enough.

Proof

We can adopt a similar argument as the proof in the [48, Lemma 5.5], thus we omit it. $\square$

Theorem 6.9

$M_{a,\mu }$ can be achieved by some $\tilde{u}\in S_{a,r}$. Moreover, $\tilde{u}$ is an interior local minimizer for $E_\mu |_{\mathcal {A}_{R_0}}$, and solves (1.3)–(1.4) for some $\tilde{\lambda }<0$.

Proof

Let us consider a minimizing sequence $\left\{ v_n\right\}$ for $E_\mu |_{\mathcal {A}_{R_0}}$. By Lemma 6.6, there exists a sequence $\{s_{v_n}\}$ such that $s_{v_n}\star v_n\in \mathcal {P}_{a,\mu }^+$ and

$$\begin{aligned} E_\mu (s_{v_n}\star v_n)=\min \left\{ E_\mu (\tau \star u):\tau \in \mathbb {R} ~\text{ and }~\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(\tau \star u)|^2< R_0\right\} <E_\mu (v_n), \end{aligned}$$

where the last inequality follows from $v_n\in \mathcal {A}_{R_0}$. Besides, we also see that

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(s_{v_n}\star v_n)|^2<R_0, \end{aligned}$$

furthermore, we by Lemma 6.8 have

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}(s_{v_n}\star v_n)|^2<R_0-\rho . \end{aligned}$$

Once again by Lemma 6.8, it holds that

$$\begin{aligned} M_{a,\mu }=\inf _{\mathcal {P}_{a,\mu }}E_\mu =\inf _{\mathcal {P}_{a,\mu }^+}E_\mu . \end{aligned}$$

Setting $u_n=s_{v_n}\star v_n$ and using the Ekeland’s variational principle, we may assume that $\left\{ u_n\right\}$ is a Palais–Smale sequence for $E_\mu$ on $S_{a,r}$ and $P_\mu (u_n)=0$. Since being $M_{a,\mu }<0$, then $\left\{ u_n\right\}$ satisfies all the assumptions of Lemma 5.5 (see Remark 5.6) and hence $u_n\rightarrow \tilde{u}$ strongly in $H^s(\mathbb {R}^N)$. Thus, we get $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\tilde{u}|^2<R_0-\rho$ and $\tilde{u}$ is an interior local minimizer for $M_{a,\mu }$. By the Lagrange multiplier rule, $\tilde{u}$ solves (1.3)–(1.4) for some $\tilde{\lambda }\in \mathbb {R}$. The conclusion that $\tilde{\lambda }<0$ can be easily obtained by virtue of the following Pohozaev identity:

$$\begin{aligned} \tilde{\lambda }|\tilde{u}|_2^2=\frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\tilde{u}|^2+\mu \frac{2(q-p)}{q(p-2)}|\tilde{u}|_q^q. \end{aligned}$$

$\square$

We focus now on the existence of a second critical point for $E_\mu |_{S_{a,r}}$. To construct a minimax structure, we give some lemmas. The following two lemmas can be obtained as the proof of [48, Lemma 5.6–5.7].

Lemma 6.10

Suppose that $E_\mu (u)<M_{a,\mu }$. Then the value $t_u$ defined by Lemma 6.6 is negative.

Lemma 6.11

It results that

$$\begin{aligned} \tilde{\sigma }_{a,\mu }=\inf _{u\in \mathcal {P}_{a,\mu }^-}E_\mu (u)>0. \end{aligned}$$

We introduce the minimax class

$$\begin{aligned} \Gamma :=\left\{ \gamma \in C([0,1],S_{a,r}):\gamma (0)\in \mathcal {P}_{a,\mu }^+, E_\mu (\gamma (1))\le 2M_{a,\mu }\right\} , \end{aligned}$$

then $\Gamma \ne \emptyset$. In fact, for every $u\in S_{a,r}$, we have $s_u\star u\in \mathcal {P}_{a,\mu }^+$ by Lemma 6.6, $E_\mu (\tau \star u)\rightarrow -\infty$ as $\tau \rightarrow +\infty$, and $\tau \mapsto \tau \star u$ is continuous. Thus, we can define the minimax value

$$\begin{aligned} \sigma _{a,\mu }:=\inf _{\gamma \in \Gamma }\max _{t\in [0,1]}E_\mu (\gamma (t)). \end{aligned}$$

Theorem 6.12

$\sigma _{a,\mu }>0$ can be achieved by some $\hat{u}\in S_{a,r}$. Moreover, $\hat{u}$ solves (1.3)–(1.4) for some $\hat{\lambda }<0$.

Proof

Step 1: Since we want to use Lemma 2.6, next we verify the conditions of Lemma 2.6 one by one. Let us set

$$\begin{aligned} \mathcal {F}:=\Gamma , A:=\gamma ([0,1]), F:=\mathcal {P}_{a,\mu }^-, ~\text{ and }~ B:=\mathcal {P}_{a,\mu }^+\cup E_\mu ^{2M_{a,\mu }}, \end{aligned}$$

where $E_\mu ^c:=\{u\in S_{a,r}:E_\mu (u)\le c\}$. First, we show that $\mathcal {F}$ is homotopy-stable family with extended boundary B: for any $\gamma \in \Gamma$ and any $\eta \in C\left( [0,1]\times S_{a,r}; S_{a,r}\right)$ satisfying $\eta (t,u)=u,(t,u)\in \left( {0}\times S_{a,r}\right) \cup \left( [0,1]\times B\right)$, we want to get $\eta (1,\gamma (t))\in \Gamma$. In fact, let $\tilde{\gamma }(t)=\eta (1,\gamma (t))$, then $\tilde{\gamma }(0)=\eta (1,\gamma (0))=\gamma (0)\in \mathcal {P}_{a,\mu }^+$. Besides, $\tilde{\gamma }(1)=\eta (1,\gamma (1))=\gamma (1)\in E_\mu ^{2M_{a,\mu }}$. Therefore, we have $\eta (1,\gamma (t))\in \Gamma$.

Next we verify the condition (2.3): By Corollary 6.7 and Lemma 6.11, we know $F\cap B=\emptyset$ and hence $F{\setminus } B=F$. We claim that

$$\begin{aligned} A\cap (F{\setminus } B)=A\cap F=\gamma ([0,1])\cap \mathcal {P}_{a,\mu }^-\ne \emptyset , \quad \forall \gamma \in \Gamma . \end{aligned}$$

(6.5)

Indeed, since $\gamma (0)\in \mathcal {P}_{a,\mu }^+$, we know $s_{\gamma (0)}=0$ (see the definition of $s_u$ in Lemma 6.6) and hence $t_{\gamma (0)}>s_{\gamma (0)}=0$. On the other hand, since $E_\mu (\gamma (1))\le 2M_{a,\mu }<M_{a,\mu }$ (see Lemma 6.8), we by Lemma 6.10 have $t_{\gamma (1)}<0$. By Lemma 6.6, we know $t_{\gamma (\tau )}$ is continuous in $\tau$. It follows that for every $\gamma \in \Gamma$ there exists $\tau _\gamma \in (0,1)$ such that $t_{\gamma (\tau _\gamma )}=0$, that is, $\gamma (\tau _\gamma )\in \mathcal {P}_{a,\mu }^-$, and hence $A\cap (F{\setminus } B)\ne \emptyset$.

Finally we verify the condition (2.4), that is, we need to show

$$\begin{aligned} \inf _{\mathcal {P}_{a,\mu }^-}E_\mu \ge \sigma _{a,\mu }\ge \sup _{\mathcal {P}_{a,\mu }^+\cup E_\mu ^{2M_{a,\mu }}}E_\mu . \end{aligned}$$

By (6.5) for every $\gamma \in \Gamma$

$$\begin{aligned} \max _{t\in [0,1]}E_\mu (\gamma (t))\ge \inf _{\mathcal {P}_{a,\mu }^-}E_\mu , \end{aligned}$$

so that $\sigma _{a,\mu }\ge \tilde{\sigma }_{a,\mu }$. On the other side, if $u\in \mathcal {P}_{a,\mu }^-$, then for $s_1\gg 1$ large enough

$$\begin{aligned} \gamma _u:\tau \in [0,1]\mapsto \left( (1-\tau )s_u+\tau s_1\right) \star u\in S_{a,r} \end{aligned}$$

is a path in $\Gamma$. Since $u\in \mathcal {P}_{a,\mu }^-$, we know $t_u=0$ is a global maximum point for $\Psi _u^\mu$, and deduce that

$$\begin{aligned} E_\mu (u)\ge \max _{t\in [0,1]}E_\mu (\gamma _u(t))\ge \sigma _{a,\mu }, \end{aligned}$$

which implies that $\tilde{\sigma }_{a,\mu }\ge \sigma _{a,\mu }$. Thus, we get $\sigma _{a,\mu }=\tilde{\sigma }_{a,\mu }>0$(see Lemma 6.11). By Corollary 6.7, we know $E_\mu (u)\le 0$ for any $u\in \mathcal {P}_{a,\mu }^+\cup E_\mu ^{2M_{a,\mu }}$ and hence get (2.4).

Step 2: By Step 1, we can use Lemma 2.6 to obtain a Palais–Smale sequence $\left\{ u_n\right\}$ for $E_\mu |_{S_{a,r}}$ at level $\sigma _{a,\mu }>0$ and $\text{ dist }(u_n,\mathcal {P}_{a,\mu }^-)\rightarrow 0$, i.e., $P_\mu (u_n)\rightarrow 0$. By Lemma 5.5 and Remark 5.6, we deduce that up to a subsequence $u_n\rightarrow \hat{u}$ strongly in $H^s(\mathbb {R}^N)$.

Step 3: By the Lagrange multiplier rule, $\hat{u}$ solves (1.3)–(1.4) for some $\hat{\lambda }\in \mathbb {R}$. The conclusion that $\hat{\lambda }<0$ can be easily obtained by virtue of the following Pohozaev identity:

$$\begin{aligned} \hat{\lambda }|\hat{u}|_2^2=\frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2+\mu \frac{2(q-p)}{q(p-2)}|\hat{u}|_q^q. \end{aligned}$$

$\square$

Proof of Theorem 1.8

Theorem 1.8 follows from Theorems 6.9 and 6.12. $\square$

(b) $\mu <0$. For the defocusing case, we consider once again the Pohozaev manifold $\mathcal {P}_{a,\mu }$ and the decomposition $\mathcal {P}_{a,\mu }=\mathcal {P}_{a,\mu }^+\cup \mathcal {P}_{a,\mu }^0\cup \mathcal {P}_{a,\mu }^-$.

Lemma 6.13

$\mathcal {P}_{a,\mu }^0=\emptyset$, and $\mathcal {P}_{a,\mu }$ is a smooth manifold of codimension 2 in $H^s(\mathbb {R}^N)$.

Proof

If $u\in \mathcal {P}_{a,\mu }^0$, then

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\frac{\gamma _p}{s}|u|_p^p+\mu \frac{\gamma _q}{s}|u|_q^q,\quad 2s^2\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=p\gamma _p^2|u|_p^p+\mu q\gamma _q^2|u|_q^q, \end{aligned}$$

which gives

$$\begin{aligned} \mu \gamma _q(2s-q\gamma _q)|u|_q^q=\gamma _p(p\gamma _p-2s)|u|_p^p, \end{aligned}$$

which implies $u\equiv 0$ since $\mu <0$ and $q\gamma _q<2s<p\gamma _p$. This contradicts the fact that $u\in S_{a,r}$. The rest of the proof is very similar to the one of Lemma 6.5, we omit it. $\square$

Lemma 6.14

For every $u\in S_{a,r}$, there exists a unique $t_u\in \mathbb {R}$ such that $t_u\star u\in \mathcal {P}_{a,\mu }$. $t_u$ is the unique critical point of the function $\Psi _u^\mu$, and is a strict maximum point at positive level. Moreover,

• (1) $\mathcal {P}_{a,\mu }=\mathcal {P}_{a,\mu }^-$.

• (2) $\Psi _u^\mu$ is strictly decreasing and concave on $(t_u,\infty )$.

• (3) The map $u\in S_{a,r}\mapsto t_u\in \mathbb {R}$ is of class $C^1$.

• (4) If $P_\mu (u)<0$, then $t_u<0$.

Proof

The proof can argue in the same way as that of [48, Lemma 7.2] and hence omit the details. $\square$

Lemma 6.15

It results that

$$\begin{aligned} M_{a,\mu }:=\inf _{u\in \mathcal {P}_{a,\mu }}E_\mu (u)>0. \end{aligned}$$

Proof

If $u\in \mathcal {P}_{a,\mu }$, we have

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\frac{\gamma _p}{s}|u|_p^p+\mu \frac{\gamma _q}{s}|u|_q^q, \end{aligned}$$

(6.6)

which implies

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\le \frac{\gamma _p}{s}|u|_p^p\le \frac{\gamma _p}{s}C(s,N,p)a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}. \end{aligned}$$

Since $\frac{N(p-2)}{4s}>1$, we deduce that $\inf _{\mathcal {P}_{a,\mu }}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\ge \delta _a>0$. On the other hand, we by (6.6) have

$$\begin{aligned} E_\mu (u)= & {} \frac{1}{2}\left( 1-\frac{2s}{p\gamma _p}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2+\frac{|\mu |}{q}\left( 1-\frac{q\gamma _q}{p\gamma _p}\right) |u|_q^q\\\ge & {} \frac{1}{2}\left( 1-\frac{2s}{p\gamma _p}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2, \end{aligned}$$

and the desired result follows from the inequality above. $\square$

Lemma 6.16

There exists $k_a>0$ sufficiently small such that

$$\begin{aligned} 0<\sup _{\mathcal {A}_{k_a}}E_\mu <M_{a,\mu } \end{aligned}$$

and

$$\begin{aligned} E_\mu (u)>\inf _{\mathcal {A}_{k_a}}E_\mu =0,\quad P_\mu (u)>\inf _{\mathcal {A}_{k_a}}P_\mu =0,\quad \forall u\in \mathcal {A}_{k_a}, \end{aligned}$$

where

$$\begin{aligned} \mathcal {A}_{k_a}:=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<k_a\right\} . \end{aligned}$$

Proof

By the GNS inequality

$$\begin{aligned} E_\mu (u)\ge & {} \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-C_1a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}>0,\nonumber \\ P_\mu (u)\ge & {} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-C_2a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}>0, \end{aligned}$$

(6.7)

if $u\in \mathcal {A}_{k_a}$ with $k_a$ small enough. Thus, $\inf _{\mathcal {A}_{k_a}}E_\mu \ge 0,\inf _{\mathcal {A}_{k_a}}P_\mu \ge 0$. Next we show that $\inf _{\mathcal {A}_{k_a}}P_\mu =0$ can’t be achieved by $u\in S_{a,r}$. In fact, for any $u\in S_{a,r}$, we know $\tau \star u\in \mathcal {A}_{k_a}$ for $\tau \ll -1$ and $P_\mu (\tau \star u)\rightarrow 0$ as $\tau \rightarrow -\infty$. Therefore, $\inf _{\mathcal {A}_{k_a}}P_\mu =0$. If there exists $u\in \mathcal {A}_{k_a}$ such that $P_\mu (u)=\inf _{\mathcal {A}_{k_a}}P_\mu =0$, by (6.7), we know $\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=0$ and hence u must be a constant, contradicting the fact that $u\in S_a$. The similar argument holds for $\inf _{\mathcal {A}_{k_a}}E_\mu$. If necessary replacing k with a smaller quantity, we also have

$$\begin{aligned} E_\mu (u)\le \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2+C_3|\mu |a^{q-\frac{N(q-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(q-2)}{4s}}<M_{a,\mu }. \end{aligned}$$

$\square$

We introduce the minimax class

$$\begin{aligned} \Gamma :=\left\{ \gamma \in C([0,1],S_{a,r}):\gamma (0)\in \overline{\mathcal {A}_{k_a}}, E_\mu (\gamma (1))\le 0\right\} , \end{aligned}$$

then $\Gamma \ne \emptyset$. In fact, for every $u\in S_{a,r}$, there exist $\tau _0\ll -1$ and $\tau _1\gg 1$, such that $\tau _0\star u\in \overline{\mathcal {A}_{k_a}}$ and $E_\mu (\tau _1\star u)<0$, and $\tau \mapsto \tau \star u$ is continuous. Thus, we can define the minimax value

$$\begin{aligned} \sigma _{a,\mu }:=\inf _{\gamma \in \Gamma }\max _{t\in [0,1]}E_\mu (\gamma (t)). \end{aligned}$$

Then we can obtain the proof of Theorem 1.9, that is,

Theorem 6.17

$\sigma _{a,\mu }>0$ can be achieved by some $\hat{u}\in S_{a,r}$. Moreover, $\hat{u}$ solves (1.3)–(1.4) for some $\hat{\lambda }<0$.

Proof

Step 1: Since we want to use Lemma 2.6, next we verify the conditions of Lemma 2.6 one by one. Let us set

$$\begin{aligned} \mathcal {F}:=\Gamma , A:=\gamma ([0,1]), F:=\mathcal {P}_{a,\mu }, ~\text{ and }~ B:=\overline{\mathcal {A}_{k_a}}\cup E_\mu ^0, \end{aligned}$$

where $E_\mu ^c:=\{u\in S_{a,r}:E_\mu (u)\le c\}$. First, we show that $\mathcal {F}$ is homotopy-stable family with extended boundary B: for any $\gamma \in \Gamma$ and any $\eta \in C\left( [0,1]\times S_{a,r}; S_{a,r}\right)$ satisfying $\eta (t,u)=u,(t,u)\in \left( {0}\times S_{a,r}\right) \cup \left( [0,1]\times B\right)$, we want to get $\eta (1,\gamma (t))\in \Gamma$. In fact, let $\tilde{\gamma }(t)=\eta (1,\gamma (t))$, then $\tilde{\gamma }(0)=\eta (1,\gamma (0))=\gamma (0)\in \mathcal {P}_{a,\mu }^+$. Besides, $\tilde{\gamma }(1)=\eta (1,\gamma (1))=\gamma (1)\in E_\mu ^0$. Therefore, we have $\eta (1,\gamma (t))\in \Gamma$.

Next we verify the condition (2.3): By Lemma 6.15 and 6.16, we know $F\cap B=\emptyset$ and hence $F{\setminus } B=F$. We claim that

$$\begin{aligned} A\cap (F{\setminus } B)=A\cap F=\gamma ([0,1])\cap \mathcal {P}_{a,\mu }\ne \emptyset , \quad \forall \gamma \in \Gamma . \end{aligned}$$

(6.8)

Indeed, by Lemma 6.16, we have $P_\mu (\gamma (0))>0$. Since $E_\mu (\gamma (1))\le 0$, we consider the fiber map $\Psi _{\gamma (1)}^\mu$, then we know $t_{\gamma (1)}<0$. By Lemma 6.14, we know $P_\mu (\gamma (1))=\left( \Psi _{\gamma (1)}^\mu \right) '(0)<0$. Thus, by the continuity of $P_\mu (\gamma (t))$, it follows that for every $\gamma \in \Gamma$ there exists $\tau _\gamma \in (0,1)$ such that $P_\mu \left( \gamma (\tau _\gamma )\right) =0$, and hence $A\cap (F{\setminus } B)\ne \emptyset$.

Finally we verify the condition (2.4), that is, we need to show

$$\begin{aligned} \inf _{\mathcal {P}_{a,\mu }}E_\mu \ge \sigma _{a,\mu }\ge \sup _{\overline{\mathcal {A}_{k_a}}\cup E_\mu ^0}E_\mu . \end{aligned}$$

By (6.8) for every $\gamma \in \Gamma$

$$\begin{aligned} \max _{t\in [0,1]}E_\mu (\gamma (t))\ge \inf _{\mathcal {P}_{a,\mu }}E_\mu , \end{aligned}$$

so that $\sigma _{a,\mu }\ge M_{a,\mu }$. On the other side, if $u\in \mathcal {P}_{a,\mu }$, then for $s_0\ll -1$ and $s_1\gg 1$

$$\begin{aligned} \gamma _u:\tau \in [0,1]\mapsto \left( (1-\tau )s_0+\tau s_1\right) \star u\in S_{a,r} \end{aligned}$$

is a path in $\Gamma$. Since $u\in \mathcal {P}_{a,\mu }$, we know $t_u=0$ is a global maximum point for $\Psi _u^\mu$, and deduce that

$$\begin{aligned} E_\mu (u)\ge \max _{t\in [0,1]}E_\mu (\gamma _u(t))\ge \sigma _{a,\mu }, \end{aligned}$$

which implies that $M_{a,\mu }\ge \sigma _{a,\mu }$. Thus, we get $\sigma _{a,\mu }=M_{a,\mu }>0$(see Lemma 6.15). By Lemma 6.16, we know $E_\mu (u)\le M_{a,\mu }$ for any $u\in \overline{\mathcal {A}_{k_a}}\cup E_\mu ^0$ and hence get (2.4).

Step 2: By Step 1, we can use Lemma 2.6 to obtain a Palais–Smale sequence $\left\{ u_n\right\}$ for $E_\mu |_{S_{a,r}}$ at level $\sigma _{a,\mu }>0$ and $\text{ dist }(u_n,\mathcal {P}_{a,\mu })\rightarrow 0$, i.e., $P_\mu (u_n)\rightarrow 0$.

Step 3: The compactness of $\left\{ u_n\right\}$. Since $\left\{ u_n\right\} \subset S_{a,r}$ is a Palais–Smale sequence for $E_\mu |_{S_{a,r}}$, by the Lagrange multipliers rule there exist $\left\{ \lambda _n\right\} \subset \mathbb {R}$ such that

$$\begin{aligned} \left( -\Delta \right) ^su_n-|u_n|^{p-2}u_n-\mu |u_n|^{q-2}u_n=\lambda _n u_n+o(1) \quad \text{ as } n\rightarrow \infty . \end{aligned}$$

We can proceed the proof of the boundedness of $\left\{ u_n\right\}$ and $\left\{ \lambda _n\right\}$ as Step 1 and 2 in Lemma 5.5 and hence omit the details. Thus, we can assume that $u_n\rightharpoonup \hat{u}$ in $H^s(\mathbb {R}^N)$ and $\lambda _n\rightarrow \hat{\lambda }$, which implies that $\hat{u}$ is a weak solution to

$$\begin{aligned} \left( -\Delta \right) ^s\hat{u}-|\hat{u}|^{p-2}\hat{u}-\mu |\hat{u}|^{q-2}\hat{u}=\hat{\lambda } \hat{u}. \end{aligned}$$

By the Pohozaev identity and the equation above for $\hat{u}$, we get

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2=\frac{N(p-2)}{2ps}|\hat{u}|_p^p+\mu \frac{N(q-2)}{2qs}|\hat{u}|_q^q, \end{aligned}$$

(6.9)

and

$$\begin{aligned} \hat{\lambda }|\hat{u}|_2^2=\frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2+\mu \frac{2(q-p)}{q(p-2)}|\hat{u}|_q^q. \end{aligned}$$

(6.10)

Since $\mu <0$, by (6.9) and the fractional GNS inequality, we obtain

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\le \frac{N(p-2)C(s,N,p)}{2ps}a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\right) ^{\frac{N(p-2)}{4s}}. \end{aligned}$$

This gives

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\ge \left( \frac{2ps}{N(p-2)C(s,N,p)}a^{\frac{N(p-2)}{2s}-p}\right) ^\frac{4s}{N(p-\bar{p})}. \end{aligned}$$

(6.11)

Since $q<p<2_s^*$, by (6.10), (6.11) and the fractional GNS inequality, we get

$$\begin{aligned} \hat{\lambda }|\hat{u}|_2^2\le & {} \frac{(N-2s)(p-2_s^*)}{(p-2)N}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\\&+\mu \frac{2(q-p)}{q(p-2)}C(s,N,q)a^{q-\frac{N(q-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\right) ^{\frac{N(q-2)}{4s}}\\\le & {} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\right) ^{\frac{N(q-2)}{4s}}\left[ \frac{(N-2s)(p-2_s^*)}{(p-2)N} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\right) ^{\frac{N(\bar{p}-q)}{4s}}\right. \\&\left. +\mu \frac{2(q-p)}{q(p-2)}C(s,N,q)a^{q-\frac{N(q-2)}{2s}}\right] \\\le & {} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}\hat{u}|^2\right) ^{\frac{N(q-2)}{4s}}\left[ \mu \frac{2(q-p)}{q(p-2)}C(s,N,q)a^{q-\frac{N(q-2)}{2s}}\right. \\&\left. +\frac{(N-2s)(p-2_s^*)}{(p-2)N}\left( \frac{2ps}{N(p-2)C(s,N,p)}a^{\frac{N(p-2)}{2s}-p}\right) ^\frac{\bar{p}-q}{p-\bar{p}}\right] . \end{aligned}$$

Recalling that (1.11), we have $\hat{\lambda }<0$. Then we follow a similar argument as Step 4 in Lemma 5.5 to deduce that $u_n\rightarrow \hat{u}$ in $H^s(\mathbb {R}^N)$. So we know $\sigma _{a,\mu }=E_\mu (\hat{u})$ and $\hat{u}$ solves (1.3) for some $\hat{\lambda }<0$. $\square$

6.3 Supercritical leading term with critical perturbation

In this subsection, we deal with the case $2<q=\bar{p}<p<2_s^*,\mu \in \mathbb {R}$ and prove Theorems 1.10 and 1.11. For convenience, we still use the notations given in Sect. 6.2.

(a) $\mu >0$. For the focusing case, we consider once again the Pohozaev manifold $\mathcal {P}_{a,\mu }$ and the decomposition $\mathcal {P}_{a,\mu }=\mathcal {P}_{a,\mu }^+\cup \mathcal {P}_{a,\mu }^0\cup \mathcal {P}_{a,\mu }^-$.

Lemma 6.18

$\mathcal {P}_{a,\mu }^0=\emptyset$, and $\mathcal {P}_{a,\mu }$ is a smooth manifold of codimension 2 in $H^s(\mathbb {R}^N)$.

Proof

If $u\in \mathcal {P}_{a,\mu }^0$, then

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\frac{\gamma _p}{s}|u|_p^p+\mu \frac{\gamma _{\bar{p}}}{s}|u|_{\bar{p}}^{\bar{p}},\quad 2s^2\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2=\mu {\bar{p}}\gamma _{\bar{p}}^2|u|_{\bar{p}}^{\bar{p}}+p\gamma _p^2|u|_p^p, \end{aligned}$$

which gives

$$\begin{aligned} \mu \gamma _{\bar{p}}(2s-{\bar{p}}\gamma _{\bar{p}})|u|_{\bar{p}}^{\bar{p}}=\gamma _p(p\gamma _p-2s)|u|_p^p, \end{aligned}$$

which implies $u\equiv 0$ since ${\bar{p}}\gamma _{\bar{p}}=2s$. This contradicts the fact that $u\in S_{a,r}$. The rest of the proof is very similar to the one of Lemma 6.5, we omit it. $\square$

Lemma 6.19

For every $u\in S_{a,r}$, there exists a unique $t_u\in \mathbb {R}$ such that $t_u\star u\in \mathcal {P}_{a,\mu }$. $t_u$ is the unique critical point of the function $\Psi _u^\mu$, and is a strict maximum point at positive level. Moreover,

• (1) $\mathcal {P}_{a,\mu }=\mathcal {P}_{a,\mu }^-$.

• (2) $\Psi _u^\mu$ is strictly decreasing and concave on $(t_u,\infty )$.

• (3) The map $u\in S_{a,r}\mapsto t_u\in \mathbb {R}$ is of class $C^1$.

• (4) If $P_\mu (u)<0$, then $t_u<0$.

Proof

Since $q=\bar{p}$ and ${\bar{p}}\gamma _{\bar{p}}=2s$, we have

$$\begin{aligned} \Psi _u^\mu (\tau )=\left( \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{\mu }{\bar{p}}|u|_{\bar{p}}^{\bar{p}}\right) e^{2s\tau }-\frac{1}{p}|u|_p^pe^{p\gamma _p\tau }. \end{aligned}$$

By the fractional GNS inequality and assumption (1.12), we know

$$\begin{aligned} \frac{1}{2}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{\mu }{\bar{p}}|u|_{\bar{p}}^{\bar{p}}\ge \left( \frac{1}{2}-\frac{\mu }{\bar{p}} C(s,N,\bar{p})a^{4s/N}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2>0. \end{aligned}$$

Notice that $p\gamma _p>2s$, then conclusions (1)-(4) easily follow from the properties of the fiber map $\Psi _u^\mu$. $\square$

Lemma 6.20

It results that

$$\begin{aligned} M_{a,\mu }:=\inf _{u\in \mathcal {P}_{a,\mu }}E_\mu (u)>0. \end{aligned}$$

Proof

If $u\in \mathcal {P}_{a,\mu }$, then $P_\mu (u)=0$, and by the fractional GNS inequality

$$\begin{aligned} \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\le & {} \frac{\gamma _p}{s}C(s,N,p)a^{p-\frac{N(p-2)}{2s}} \left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-2)}{4s}}\\&+\frac{2\mu }{\bar{p}} C(s,N,\bar{p})a^{\frac{4s}{N}}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2. \end{aligned}$$

Thus, we get

$$\begin{aligned} \frac{\gamma _p}{s}C(s,N,p)a^{p-\frac{N(p-2)}{2s}}\left( \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2\right) ^{\frac{N(p-\bar{p})}{4s}}\ge 2\left( \frac{1}{2}-\frac{\mu }{\bar{p}} C(s,N,\bar{p})a^{\frac{4s}{N}}\right) , \end{aligned}$$

which provides that $\inf _{\mathcal {P}_{a,\mu }}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2>0$, here we use assumption (1.12). At this point, using again $P_\mu (u)=0$, we note that for any $u\in \mathcal {P}_{a,\mu }$

$$\begin{aligned} E_\mu (u)= & {} \frac{1}{2}\left( 1-\frac{2s}{p\gamma _p}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2-\frac{\mu }{\bar{p}}\left( 1-\frac{2}{p\gamma _p}\right) |u|_{\bar{p}}^{\bar{p}}\\\ge & {} \left( 1-\frac{2s}{p\gamma _p}\right) \left( \frac{1}{2}-\frac{\mu }{\bar{p}} C(s,N,\bar{p})a^{\frac{4s}{N}}\right) \int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2, \end{aligned}$$

and the desired result follows by $\inf _{\mathcal {P}_{a,\mu }}\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2>0$. $\square$

Lemma 6.21

There exists $k_a>0$ sufficiently small such that

$$\begin{aligned} 0<\sup _{\mathcal {A}_{k_a}}E_\mu <M_{a,\mu } \end{aligned}$$

and

$$\begin{aligned} E_\mu (u)>\inf _{\mathcal {A}_{k_a}}E_\mu =0,\quad P_\mu (u)>\inf _{\mathcal {A}_{k_a}}P_\mu =0,\quad \forall u\in \mathcal {A}_{k_a}, \end{aligned}$$

where

$$\begin{aligned} \mathcal {A}_{k_a}:=\left\{ u\in S_{a,r}:\int _{\mathbb {R}^N}|\left( -\Delta \right) ^{\frac{s}{2}}u|^2<k_a\right\} . \end{aligned}$$

Proof

The proof is similar to that of Lemma 6.16 and is omitted. $\square$

Proof of Theorem 1.10

We can proceed exactly as in the proof of Theorem 6.12, using Lemmas 6.19,6.20 and 6.21 instead of Lemmas 6.14,6.15 and 6.16, respectively. Thus, we omit the details. $\square$

(b): $\mu <0$. For the defocusing critical perturbation, we prove Theorem 1.11.

Proof of Theorem 1.11

We can proceed exactly as in the proof of Theorem 1.9 with minor changes, so we omit it. $\square$