Positive Solutions for a Kirchhoff-Type Equation with Critical and Supercritical Nonlinear Terms

Abstract

We consider a Kirchhoff-type equation with critical and supercritical nonlinear terms in a ball. By providing a method of decomposition of energy functional and subtle analysis, we show that every Palais–Smale sequence at a level below a certain energy threshold admits a subsequence that converges strongly to a nontrivial critical point of the variational functional.

1 Introduction and Main Results

We consider the following Kirchhoff-type equation with critical growth:

$$\begin{aligned} {\left\{ \begin{array}{ll} M\left( \displaystyle \int _{B_r}|\nabla u|^2dx\right) \Delta u+u^5+\frac{|x|^\beta }{1+|x|^\beta }u^{p-1}=0, ~~\mathrm {in}~B_r, \\ u=0, ~~~\mathrm {on}~ \partial B_r, \end{array}\right. } \end{aligned}$$

(1.1)

where $B_r\subset {\mathbb {R}}^3$ is an open ball centered at the origin, $0<\beta<1, 6<p<6+2\beta$. Denoting ${\widehat{M}}(s)=\int _0^sM(t)dt$, we make the following assumptions:

$(M_1)$:

$M\in C^1({\mathbb {R}}\mathbb {^+},{\mathbb {R}}^+)$, $M(s)\ge a>0$, a is a constant, M(s) is increasing in s;

$(M_2)$:

$2M(s)\ge sM'(s)$ for each $s>0$, and $\lim _{s\rightarrow +\infty }\frac{M(s)}{s^2}=0$;

$(M_3)$:

for $s>0,$ ${\widehat{M}}(s)-\frac{1}{3}sM(s)\ge \frac{2}{3}as$ and $\frac{1}{s}(\frac{1}{2}{\widehat{M}}(s)-\frac{1}{6}sM(s))$ is nondecreasing in s.

A typical example of M is given by $M(t)=a+bt$ for $t\in {\mathbb {R}}^+$, where $a>0,b\ge 0$. Problem (1.1) is often referred as being nonlocal because the presence of the term $M(\int _{B_r}|\nabla u|^2dx)$ implies that problem (1.1) is no longer a pointwise identity. This phenomenon causes some mathematical difficulties which make the study of such a class of problem particularly interesting. Moreover, such a problem has physical motivation. Indeed, the Kirchhoff equation arises in nonlinear vibrations, namely

$$\begin{aligned} {\left\{ \begin{array}{ll} u_{\text {tt}}-M\left( \displaystyle \int _{\Omega }|\nabla u|^2dx\right) \Delta u=f(x,u), &{}\mathrm{{in}}\quad {\Omega }\times (0,T'), \\ u=0, &{}{\text {on}}\ \ \partial \Omega \times (0,T'),\\ u(x,0)=u_0(x), ~u_t(x,0)=u_1(x), \end{array}\right. } \end{aligned}$$

where $T'>0.$ Such a hyperbolic equation is related to the stationary analogue of the following equation:

$$\begin{aligned} u_\mathrm{tt}-\left( a+b\int _{\Omega }|\nabla u|^2dx\right) \Delta u=h(x,u), \end{aligned}$$

where u denotes the displacement, h is the external force, and a is the initial tension, while b is related to the intrinsic properties of the string (such as Young’s modulus). Such nonlocal elliptic problems like problem (1.1) have received a lot of attention, and some important and interesting results have been established by using the variational methods; see for example ([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]) and the references therein. In particular, Figueiredo in [1] studied the following semilinear equation with critical growth:

$$\begin{aligned} {\left\{ \begin{array}{ll} M\left( \displaystyle \int _{\Omega }|\nabla u|^2dx\right) \Delta u+|u|^4u +\lambda f(x,u)=0, &{}\mathrm{in}\ \ \Omega , \\ u=0, &{} \mathrm {on}\ \ \partial \Omega , \end{array}\right. } \end{aligned}$$

(1.2)

where M satisfies the following conditions:

$(A_1)$:

The function M is increasing;

$(A_2)$:

There exists $M_0>0$ such that $M(t)\ge M_0=M(0)$ , for all $t\ge 0.$ Under suitable conditions about f and the above assumptions, applying an appropriated truncated argument, the author proved that there exists a threshold value (here the threshold value is just $(\frac{1}{\theta }-\frac{1}{6})(M_0S)^{\frac{3}{2}}$, where $4<\theta <6$ and S is the best constant for the Sobolev embedding $H_{0}^{1}(\Omega )\hookrightarrow L^{6}(\Omega )$), only below this threshold value the functional associated with the problem (1.2) satisfies the Palais–Smale condition. Moreover, the author obtained that there exists $\lambda _*>0$ such that for all $\lambda \ge \lambda _*$, problem (1.2) admits a positive solution $u_\lambda$ with $\lim _{\lambda \rightarrow \infty }\Vert u_\lambda \Vert =0$.

After that, Wang et al. in [2] extended the above equation to the following p-Kirchhoff-type equation:

$$\begin{aligned} {\left\{ \begin{array}{ll} M\left( \displaystyle \int _{\Omega }|\nabla u|^pdx\right) \Delta _p u+|u|^{p^*-2}u +\lambda f(x,u)=0, &{} \mathrm {in}\ \ \Omega , \\ u=0, &{} \mathrm {on}\ \ \partial \Omega . \end{array}\right. } \end{aligned}$$

The authors made the following assumptions $M(0)=0$, and

$(B_1)$:

there exists $\theta \in (1,p^*/p)$($p^{*}=\frac{Np}{N-p}$) such that $tM(t)\le \theta {\widetilde{M}}(t):=\theta \int _0^tM(s)ds$ for all $t>0$;

$(B_2)$:

for any $\tau >0$ there exists $\kappa =\kappa (\tau )>0$ such that $M(t)\ge \kappa$ for all $t\ge \tau$;

$(B_3)$:

there exists a constant $c>0$ such that $M(t)\ge ct^{\theta -1}$ for all $t\in [0,1]$. The authors also obtained the same result in [1] by using the Mountain pass theorem.

Now, returning to problem (1.1), it is clear that problem (1.1) has a variational structure. We understand critical points of the associated energy functional acting on the Sobolev space H:

$$\begin{aligned} I(u)=\frac{1}{2}{\widehat{M}}\left( \int _{B_r}|\nabla u|^2dx\right) -\frac{1}{6}\int _{B_r}|u|^6dx-\frac{1}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }|u|^pdx, \end{aligned}$$

where $H:=H_{0, \mathrm rad}^{1}(B_r)$ is the first-order Sobolev space of radial functions equipped with the inner product and norm

$$\begin{aligned} \langle u,v\rangle =\int _{B_r}\nabla u\nabla vdx,~~\Vert u\Vert =\langle u,v\rangle ^{\frac{1}{2}}. \end{aligned}$$

We say that the functional I satisfies the Palais–Smale ((PS) for short) condition if any (PS) sequence $\{u_n\}\subset H$, that is a sequence satisfying

$$\begin{aligned} \{I(u_n)\}~\mathrm {bounded~and}~I'(u_n)\rightarrow 0~\mathrm {in}~H^{-1}~\mathrm {as}~n\rightarrow +\infty , \end{aligned}$$

admits a convergent subsequence. If $\{u_n\}\subset H$ is a bounded (PS) sequence of I, $\{u_n\}$ has a profile decomposition with containing finitely bubbles (see Lemma 2.4)

$$\begin{aligned} u_{n}=u+\sum _{k\in \Lambda }g_{n,k}V_N+\gamma _{n}, \end{aligned}$$

where $V_N, \gamma _{n}\in D^{1,2}({\mathbb {R}}^3)$ which is the completion of $C_0^\infty ({\mathbb {R}}^3)$ with respect to the norm

$$\begin{aligned} \Vert \varphi \Vert _{D}=\left( \int _{{\mathbb {R}}^3}|\nabla \varphi |^2dx\right) ^{\frac{1}{2}}. \end{aligned}$$

$\Lambda$ is finite, say $\Lambda = {1,2,...,N}$ ($\Lambda$ may be empty and $N=0$). In particular, if $N=0$, then $u_n\rightarrow u$ strongly in H. In order to prove the result, we need to establish the exact threshold value, and only below this threshold value the functional I satisfies the (PS) condition.

Motivated by the above works, we study the existence of solutions for problem (1.1). Here, in order to overcome the lack of compactness induced by the presence of the Kirchhoff term and critical exponent, some delicate estimates are exploited which are totally different from those used in the papers mentioned above. Unlike [1, 2], due to the enough large constraint condition on the parameter $\lambda$, which plays a very important role, it causes us not to estimate the threshold value of the energy functional. Therefore, assume that M is an abstract function and the parameter $\lambda$ is not restricted, in this case, it is very difficult to determine the threshold value of the energy functional. However, in this paper, by a concentration compactness analysis on the Palais–Smale sequence, we establish the threshold value to consequently prove the existence of positive solutions. Thus, the method used in those articles cannot be repeated here because we are working with M which is an abstract function and the parameter $\lambda \equiv 1$.

Now our main result is as follows:

Theorem 1.1

Assume that $0<\beta<1, 6<p<6+2\beta$ and $(M_1), (M_2), (M_3)$ hold, then problem (1.1) has a positive solution u. Moreover

$$\begin{aligned} \lim _{r\rightarrow \infty }u(r)=0. \end{aligned}$$

Remark 1.2

Under the assumptions of Theorem 1.2, the following problem

$$\begin{aligned} \left\{ \begin{array}{l@{\qquad }l} M\left( \displaystyle \int _{B_r}|\nabla u|^2dx\right) \Delta u+u^5+|x|^\beta u^{p-1}=0, &{}~~\mathrm {in}~B_r,\\ u=0, &{}~~\mathrm {on}~ \partial B_r, \end{array}\right. \end{aligned}$$

admits a positive solution u with $\lim _{r\rightarrow \infty }u(r)=0$.

This paper is organized as follows. In Sect. 2, we give the compactness analysis and establish the Palais–Smale condition. In Sect. 3, we demonstrate the threshold value and conclude Theorem 1.2. In the proof, we use a same character C to denote several positive constants.

2 Concentration Compactness Analysis

In this section, we make concentration compactness analysis on the (PS) sequence of the functional I. The results will be used to deduce the system of coupled equations satisfied by the weak limit function of a (PS) sequence and the bubbles.

Lemma 2.1

Assume that $0<\beta<1, 6<p<6+2\beta$. If $\{u_n\}$ is bounded and $u_n\rightharpoonup u$ in H. Then,

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }|u_n|^pdx=\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }|u|^pdx. \end{aligned}$$

(2.1)

Proof

From Lemma 2.2 in [24], we have

$$\begin{aligned} |u_n(x)|\le C\Vert u_n\Vert \frac{1}{|x|^{\frac{1}{2}}},~~~\mathrm {a.e}.~x\in B_r. \end{aligned}$$

By the boundedness of $\{u_n\}$, for $6<p<6+2\beta$, it follows that

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \int _{|x|\le 1}\frac{|x|^\beta }{1+|x|^\beta }|u_n|^pdx&{}\le &{}\displaystyle C\int _{|x|\le 1}\frac{|x|^\beta }{1+|x|^\beta }\frac{dx}{|x|^{\frac{p}{2}}}\\ &{}\le &{}\displaystyle C\int _{|x|\le 1}\frac{dx}{|x|^{\frac{p}{2}-\beta }}\\ &{}=&{}\displaystyle C\int _0^1\frac{dt}{t^{\frac{p}{2}-\beta -2}}\\ &{}<&{}\displaystyle +\infty ~~~~(\mathrm {as}~p<6+2\beta ) \end{array} \right. \end{aligned}$$

(2.2)

and

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \int _{|x|\ge 1}\frac{|x|^\beta }{1+|x|^\beta }|u_n|^pdx&{}\le &{}\displaystyle C\int _{|x|\ge 1}\frac{|x|^\beta }{1+|x|^\beta }\frac{dx}{|x|^{\frac{p}{2}}}\\ &{}\le &{}\displaystyle C\int _{|x|\ge 1}\frac{dx}{|x|^{\frac{p}{2}}}\\ &{}=&{}\displaystyle C\int _{0}^{+\infty }\frac{dt}{t^{\frac{p}{2}-2}}\\ &{}<&{}\displaystyle +\infty . ~~~~(\mathrm {as}~p>6) \end{array} \right. \end{aligned}$$

(2.3)

Consequently, it leads to (2.1) by means of the Lebesgue dominated convergence theorem. The proof is complete.$\square$

Lemma 2.2

Any Palais–Smale sequence of I is bounded in H.

Proof

Let $\{u_n\}\subset H$ be such that

$$\begin{aligned} I'(u_n)\rightarrow 0,~~ I(u_n)\rightarrow c~ \mathrm {as} ~n\rightarrow \infty . \end{aligned}$$

(2.4)

Combining with $(M_{3})$, one has

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle I(u_n)-\frac{1}{6}\langle I'(u_n),u_n\rangle &{}\ge &{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u_n\Vert ^2)-\frac{1}{6}M(\Vert u_n\Vert ^2)\Vert u_n\Vert ^2\\ &{}&{}\displaystyle +\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }|u_n|^pdx\\ &{}\ge &{}\displaystyle \frac{a}{3}\Vert u_n\Vert ^2. \end{array} \right. \end{aligned}$$

Therefore, $\{u_n\}$ is bounded in H. The proof is complete.$\square$

To make the concentration compactness analysis, we introduce the dilation group ${\mathfrak {D}}$ in ${\mathbb {R}}^3$

$$\begin{aligned} {\mathfrak {D}}=\left\{ g_{\sigma ,y}|g_{\sigma ,y}u(\cdot )=\sigma ^{\frac{1}{2}} u(\sigma (\cdot -y)),~~y\in {\mathbb {R}}^3,~\sigma \in {\mathbb {R}}^+.\right\} \end{aligned}$$

The dilation g in ${\mathfrak {D}}$ is an isometry in both $L^{6}({\mathbb {R}}^3)$ and $D^{1,2}({\mathbb {R}}^3)$.

Let $\{u_n\}\subset H$ be a Palais–Smale sequence of the functional I. By Lemma 2.2, $\{u_n\}$ is bounded in H. According to Theorem 3.1 and Corollary 3.2 in [26](see also [25]), $\{u_n\}$ has a profile decomposition

$$\begin{aligned} u_{n}=u+\sum _{k\in \Lambda }g_{n,k}U_{k}+\gamma _{n}, \end{aligned}$$

(2.5)

where $u\in H$, $U_k\in D^{1,2}({\mathbb {R}}^{3})$, $g_{n,k}=g_{\sigma _{n,k},y_{n,k}}\in {\mathfrak {D}}$, $\sigma _{n,k}>0$, $y_{n,k}\in {\overline{B}}_r$, $\gamma _n\in D^{1,2}({\mathbb {R}}^{3})$, $\Lambda$ is an index set, satisfy:

• (1) $u_{n}\rightharpoonup u$ in H, $g^{-1}_{n,k}u_{n}\rightharpoonup U_{k}$ in $D^{1,2}({\mathbb {R}}^{3})$, as $n\rightarrow \infty$, $k\in \Lambda$;

• (2) $g_{n,k}\rightharpoonup 0$ in $[D^{1,2}({\mathbb {R}}^{3})]^*,~g^{-1}_{n,k}g_{n,l}\rightharpoonup 0$ in $[D^{1,2}({\mathbb {R}}^{3})]^*$ as $n\rightarrow \infty , k,l\in \Lambda , k\ne l$;

• (3) $\Vert u_n\Vert _{D}^2=\Vert u\Vert ^2_{D}+\sum _{k\in \Lambda }\Vert U_{k}\Vert ^2_{D}+\Vert \gamma _n\Vert ^2_{D}+o(1)$, as $n\rightarrow \infty$. From property (3) of (2.5), by the method of the Br$\acute{\mathrm {e}}$zis–Lieb’s lemma and [25], one has

• (4) $\gamma _{n}\rightarrow 0$ in $L^6({\mathbb {R}}^3)$ and

$$\begin{aligned} \int _{B_r}u_n^6dx=\int _{B_r}u^6dx+\sum _{k\in \Lambda }\int _{{\mathbb {R}}^3}|U_{k}|^6dx+o(1), ~\mathrm {as}~n\rightarrow \infty . \end{aligned}$$

Here for a sequence $\{g_n\}$ of ${\mathfrak {D}}$, we say $g_n\rightharpoonup 0$ in $[D^{1,2}({\mathbb {R}}^{3})]^*$, if for all $v\in D^{1,2}({\mathbb {R}}^{3})$, $g_nv\rightharpoonup 0$ in $D^{1,2}({\mathbb {R}}^{3})$. Moreover, since $\{u_n\}$ is bounded in H, we have $\sigma _{n,k}\rightarrow \infty$ as $n\rightarrow \infty$, $k\in \Lambda$.

We deduce the system of coupled equations satisfied by the weak limit function u and the bubbles $U_k, k\in \Lambda$. Inspired by Lemma 2.1 in [30], we obtain the following conclusion.

Lemma 2.3

Let $\{u_n\}$ be a Palais–Smale sequence of I, $A_n\triangleq \int _{B_r}|\nabla u_n|^2dx\rightarrow A$ as $n\rightarrow \infty$.

• (1) Assume $u_n\rightharpoonup u$ in H, then u satisfies the equation:

  $$\begin{aligned} M(A)\int _{B_r}\nabla u\nabla \varphi dx=\int _{B_r}u^5\varphi dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^{p-1}\varphi dx,~for~\varphi \in H.\qquad \end{aligned}$$

  (2.6)

• (2) Let $g_n=g_{\sigma _n, y_n}\in {\mathfrak {D}}$, $\sigma _n\rightarrow \infty$ as $n\rightarrow \infty$ and $y_n\in {\overline{B}}_r$. Assume ${\widetilde{u}}_n=g_n^{-1}u_n\rightharpoonup U\ne 0$ in $D^{1,2}({\mathbb {R}}^{3})$. Then, U satisfies the equation:

  $$\begin{aligned} M(A)\int _{{\mathbb {R}}^3}\nabla U\nabla \phi dx=\int _{{\mathbb {R}}^3}U^5\phi dx,~for~\phi \in D^{1,2}({\mathbb {R}}^{3}). \end{aligned}$$

  (2.7)

Proof

• (1) Since $\{u_n\}$ is (PS) sequence of I, by (2.1), we have

  $$\begin{aligned}&M\left( \int _{B_r}|\nabla u_n|^2dx\right) \int _{B_r}\nabla u_n\nabla \varphi dx\nonumber \\&-\int _{B_r}u_n^5\varphi dx -\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_n^{p-1}\varphi dx=o(1) \end{aligned}$$

  (2.8)

  for $\varphi \in H$. Letting $n\rightarrow \infty$, which deduce that (2.6) holds.

• (2) Denote

  $$\begin{aligned} d_n=\sigma _n\mathrm {dist}(y_n, \partial B_r). \end{aligned}$$

  We first assume $d_n\rightarrow \infty$. Let $\varphi$ be a smooth function in $\varphi \in C_0^\infty ({\mathbb {R}}^3)$ and $\psi =g_n\varphi =\sigma _n^{\frac{1}{2}}\varphi (\sigma _n(\cdot -x_n))$. For n large enough $\psi \in C_0^\infty (B_r)$. Taking $\psi$ as a test function in (2.8), we have

  $$\begin{aligned}&M\left( \int _{B_r}|\nabla u_n|^2dx\right) \int _{B_r}\nabla u_n\nabla \psi dx \nonumber \\&\qquad = \int _{B_r}u_n^5\psi dx +\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta } u_n^{p-1}\psi dx +o(1). \end{aligned}$$

  (2.9)

  Making a variable change, we get

  $$\begin{aligned} y=\sigma _n(x-y_n). \end{aligned}$$

  Set ${\widetilde{u}}_n=g_n^{-1}u_n$. In view of

  $$\begin{aligned} g_n^{-1}u_n=\sigma _n^{-\frac{1}{2}}u_n(\sigma _n^{-1}(\cdot +x_n)), \end{aligned}$$

  we see that

  $$\begin{aligned} \nabla g_n^{-1}u_n=\sigma _n^{-\frac{1}{2}}\frac{1}{\sigma _n}\nabla u_n. \end{aligned}$$

  Consequently, we obtain

  $$\begin{aligned} \displaystyle \int _{{\mathbb {R}}^3}\nabla u_n\nabla (g_n\varphi )dx= & {} \displaystyle \int _{\Omega _n}\sigma _n\sigma ^{\frac{1}{2}}\nabla g_n^{-1}u_n\sigma _n^{\frac{1}{2}}\sigma _n\nabla \varphi \frac{1}{\sigma _n^3}dy~~(\mathrm {let}~\sigma _n(x-x_n)=y)\\= & {} \displaystyle \int _{\Omega _n}\nabla g_n^{-1}u_n\nabla \varphi dx\\= & {} \displaystyle \int _{\Omega _n}\nabla {\widetilde{u}}_n\nabla \varphi dx,\\ \displaystyle \int _{{\mathbb {R}}^3}u_n^5g_n\varphi dx= & {} \displaystyle \int _{\Omega _n}\sigma _n^{\frac{5}{2}}(\sigma _n^{-\frac{1}{2}}u_n(\frac{y}{\sigma _n}+x_n))^5g_n\varphi (x) \frac{1}{\sigma _n^3}dy\\= & {} \displaystyle \int _{\Omega _n}\sigma _n^{\frac{5}{2}}(\sigma _n^{-\frac{1}{2}}u_n(\frac{y}{\sigma _n}+x_n))^5\sigma _n^{\frac{1}{2}}\varphi (y)\frac{1}{\sigma _n^3}dy\\= & {} \displaystyle \int _{\Omega _n}{\widetilde{u}}_n^5\varphi dx \end{aligned}$$

  where $\Omega _n=\{y|y\in {\mathbb {R}}^3, x=\sigma _n^{-1}y+y_n\in B_r\},$

  $$\begin{aligned} \int _{{\mathbb {R}}^3}\frac{|x|^\beta }{1+|x|^\beta }u_n^{p-1}g_n\varphi dx=\sigma _n^{-\frac{6-p}{2}}\int _{\Omega _n}\frac{|x|^\beta }{1+|x|^\beta }{\widetilde{u}}_n^{p-1}\varphi dx. \end{aligned}$$

  Consequently, (2.9) becomes

  $$\begin{aligned}&M\left( \int _{B_r}|\nabla u_n|^2dx\right) \int _{\Omega _n}\nabla {\widetilde{u}}_n\nabla \varphi dx \nonumber \\&\qquad = \int _{\Omega _n}{\widetilde{u}}_n^5\varphi dx+\sigma _n^{-\frac{6-p}{2}} \int _{\Omega _n}\frac{|x|^\beta }{1+|x|^\beta }{\widetilde{u}}_n^{p-1}\varphi dx +o(1), \end{aligned}$$

  (2.10)

Since ${\widetilde{u}}_n=g_n^{-1}u_n\rightharpoonup U$ and $\sigma _n\rightarrow \infty$ as $n\rightarrow \infty$, taking the limit $n\rightarrow \infty$ in (2.10), we obtain

$$\begin{aligned} M(A)\int _{{\mathbb {R}}^3}\nabla U\nabla \varphi dx-\int _{{\mathbb {R}}^3}U^5\varphi dx=0,~\mathrm {for}~\varphi \in C_0^\infty ({\mathbb {R}}^3). \end{aligned}$$

By a density argument, we obtain

$$\begin{aligned} M(A)\int _{{\mathbb {R}}^3}\nabla U\nabla Vdx-\int _{{\mathbb {R}}^3}U^5V dx=0 \end{aligned}$$

for $V\in D^{1,2}(\mathbb {{R}}^{3})$.

Finally, in the case $d_n=\sigma _n\mathrm {dist}(y_n, \partial B_r)\rightarrow d<+\infty$. Without loss of generality, we assume $d=0$. In this case, we can prove that U satisfies $U=0$ in ${\mathbb {R}}^3\backslash {\mathbb {R}}^3_+$ and

$$\begin{aligned} M(A)\int _{{\mathbb {R}}_+^3}\nabla U\nabla Vdx=\int _{{\mathbb {R}}_+^3}U^5Vdx,~\mathrm {for}~V\in D^{1,2}({\mathbb {R}}^3), V=0~\mathrm {in}~{\mathbb {R}}^3\backslash {\mathbb {R}}^3_+. \end{aligned}$$

By the uniqueness theory in [27] for positive solutions of the equation

$$\begin{aligned} {\left\{ \begin{array}{ll} \Delta u+u^{2^*-1}=0, ~~\mathrm {in}~ {\mathbb {R}}^N, \\ u>0, ~~\mathrm {in}~ {\mathbb {R}}^N,\\ u(x)\rightarrow 0, ~~\mathrm {as}~ |x|\rightarrow \infty , \end{array}\right. } \end{aligned}$$

(2.11)

$U\equiv 0$ in ${\mathbb {R}}^3$, which is a contradiction. The proof is complete.$\square$

We continue the concentration compactness analysis on Palais–Smale sequences.

Lemma 2.4

Let $\{u_n\}$ be a Palais–Smale sequence of I. Assume the profile decomposition (2.5) holds, namely

$$\begin{aligned} u_{n}=u+\sum _{k\in \Lambda }g_{n,k}U_{k}+\gamma _{n}. \end{aligned}$$

Then,

• (1) the index set $\Lambda$ is finite, say $\Lambda =\{1,2,...,N\}$ ($\Lambda$ may be empty and $N=0$).

• (2) There exist $V_N\in D^{1,2}({\mathbb {R}}^{3})$ and $g_n\in {\mathfrak {D}}$, $k=1,2,...,N$ such that

($2_a$):

$U_k=g_kV_N$, $k=1,2...,N$ and the profile decomposition (2.5) reduces to

$$\begin{aligned} u_{n}=u+\sum _{k=1}^{N}g_{n,k}V_N+\gamma _{n}. \end{aligned}$$

(2.12)

($2_b$):

u and $V_N$ satisfy the system

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle M(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)\int _{B_r}\nabla u\nabla \varphi dx=\displaystyle \int _{B_r}(u^5+\frac{|x|^\beta }{1+|x|^\beta }u^{p-1})\varphi dx, \varphi \in H,\\ \displaystyle M(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)\int _{{\mathbb {R}}^3}\nabla V_N\nabla \phi dx=\displaystyle \int _{{\mathbb {R}}^3}V_N^5 \phi dx, \phi \in D^{1,2}({\mathbb {R}}^{3}). \end{array}\right. } \end{aligned}$$

($2_c$):

There hold that

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle \int _{B_r}|\nabla u_n|^2dx=\displaystyle \int _{B_r}|\nabla u|^2dx+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx+o(1), \\ \displaystyle \int _{B_r}u_n^6dx=\displaystyle \int _{B_r}u^6dx+N\int _{{\mathbb {R}}^3}V_N^6dx+o(1),~as~n\rightarrow \infty . \end{array}\right. } \end{aligned}$$

Proof

• (1) By Lemma 2.3, we have the system

  $$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle M(A)\int _{B_r}\nabla u\nabla \varphi dx=\displaystyle \int _{B_r}u^5\varphi dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^{p-1}\varphi dx, \varphi \in H,\\ \displaystyle M(A)\int _{{\mathbb {R}}^3}\nabla U_k\nabla \phi dx=\displaystyle \int _{{\mathbb {R}}^3}U_k^5 \phi dx, \phi \in D^{1,2}({\mathbb {R}}^{3}),~k\in \Lambda , \end{array}\right. } \end{aligned}$$

  (2.13)

  where $A=\lim _{n\rightarrow \infty }\int _{B_r}|\nabla u_n|^2dx$. Taking $\phi =U_k$ as test function in the second equation of (2.13), we have

  $$\begin{aligned} \left. \begin{array}{rcl} \displaystyle a\int _{{\mathbb {R}}^3}|\nabla U_k|^2dx&{}\le &{}\displaystyle m(A)\int _{{\mathbb {R}}^3}|\nabla U_k|^2dx=\int _{{\mathbb {R}}^3}U_k^6dx\\ &{}\le &{}\displaystyle S^{-3}\left( \int _{{\mathbb {R}}^3}|\nabla U_k|^{2}dx\right) ^3, \end{array} \right. \end{aligned}$$

  where S is the Sobolev constant for the embedding $D^{1,2}({\mathbb {R}}^{3})\hookrightarrow L^6({\mathbb {R}}^3)$. We deduce that

  $$\begin{aligned} \int _{{\mathbb {R}}^3}|\nabla U_k|^2dx\ge \sqrt{aS^3}. \end{aligned}$$

  (2.14)

  By the property (3) of the profile decomposition (2.5), $\Lambda$ is a finite set, say $\Lambda =\{1,2,...,N\}$.

• (2) By the second equation of (2.13) and the uniqueness theory [27] of the positive solutions of the equation (2.11), there exist $V_k\in D^{1,2}({\mathbb {R}}^{3})$, $g_k\in {\mathfrak {D}}$, $k=1,2,...,N$ such that $U_k=g_kV_N$ and $V_N$ satisfies

  $$\begin{aligned} M(A)\int _{{\mathbb {R}}^3}\nabla V_N\nabla \phi dx=\int _{{\mathbb {R}}^3}V_N^5\phi dx, \phi \in D^{1,2}({\mathbb {R}}^{3}). \end{aligned}$$

  Replacing $g_{n,k}$ by $g_{n,k}g_k$ in the profile decomposition in (2.5), we obtain

  $$\begin{aligned} u_{n}=u+\sum _{k=1}^{N}g_{n,k}V_N+\gamma _{n}. \end{aligned}$$

  Noting that $u_n$ satisfies the inequality

  $$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_n^{p-1}vdx&{}=&{}\displaystyle M\left( \int _{B_r}|\nabla u_n|^2dx\right) \int _{B_r}\nabla u_n\nabla vdx\\ &{}&{}\displaystyle -\int _{B_r}u_n^5vdx+o(1) \end{array} \right. \end{aligned}$$

  for $v\in H$. Taking $v=u_n$ as test function in the above inequality, it yields

  $$\begin{aligned}&M\left( \int _{B_r}|\nabla u_n|^2dx\right) \int _{B_r}|\nabla u_n|^2dx\nonumber \\&\quad -\int _{B_r}u_n^6dx-\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_n^{p-1}dx=o(1). \end{aligned}$$

  (2.15)

  By (2.13)

  $$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle M(A)\int _{B_r}|\nabla u|^2dx=\displaystyle \int _{B_r}u^6 dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^{p}dx, \\ \displaystyle M(A)\int _{{\mathbb {R}}^3}|\nabla U_k|^2dx=\displaystyle \int _{{\mathbb {R}}^3}U_k^6dx, ~k\in \Lambda \end{array}\right. } \end{aligned}$$

  (2.16)

  and the property (4) of the profile decomposition (2.5)

  $$\begin{aligned} \int _{B_r}u_n^6dx=\int _{B_r}u^6dx+\sum _{k=1}^N\int _{{\mathbb {R}}^3}U_k^6dx+o(1). \end{aligned}$$

  (2.17)

  Notice that

  $$\begin{aligned} \int _{B_r}|\nabla u_n|^2dx\rightarrow A,~~\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_n^{p}dx\rightarrow \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^{p}dx \end{aligned}$$

  as $n\rightarrow \infty$. It follows from (2.15), (2.16) and (2.17) that

  $$\begin{aligned} \int _{B_r}|\nabla u_n|^2dx=\int _{B_r}|\nabla u|^2dx+\sum _{k=1}^N\int _{{\mathbb {R}}^3}|\nabla U_k|^2dx+o(1). \end{aligned}$$

  (2.18)

  Finally, since $g_k\in {\mathfrak {D}}$, $k=1,2,...,N$ are isometry in both $L^6({\mathbb {R}}^3)$ and $D^{1,2}({\mathbb {R}}^{3})$, we have

  $$\begin{aligned} \int _{{\mathbb {R}}^3}|\nabla U_k|^2dx=\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx, \end{aligned}$$

  and

  $$\begin{aligned} \int _{{\mathbb {R}}^3}V_k^6dx=\int _{{\mathbb {R}}^3}V_N^6dx, \end{aligned}$$

  where $k=1,2,...,N$. Hence, (2.17) and (2.18) can be rewritten as, respectively.

  $$\begin{aligned} \int _{B_r}|\nabla u_n|^2dx=\int _{B_r}|\nabla u|^2dx+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx+o(1), \end{aligned}$$

  and

  $$\begin{aligned} \int _{B_r}u_n^6dx=\int _{B_r}u^6dx+N\int _{{\mathbb {R}}^3}V_N^6dx+o(1) \end{aligned}$$

  as $n\rightarrow \infty$. In particular,

  $$\begin{aligned} A=\lim _{n\rightarrow \infty }\int _{B_r}|\nabla u_n|^2dx=\int _{B_r}|\nabla u|^2dx+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx, \end{aligned}$$

  and $u, V_N$ satisfy the system in ($2_b$). The proof is complete.$\square$

3 Threshold Value and the Proof of Theorem 1.2

In this section, we determine the threshold value, below which the functional I satisfies the $(PS)_c$ condition, and then show that this level is less than the threshold value. Consequently, we prove the existence of the mountain pass-type solution.

Assume that $\{u_n\}$ is a $(PS)_c$ sequence of I and the profile decomposition (2.12) holds, namely

$$\begin{aligned} u_{n}=u+\sum _{k=1}^{N}g_{n,k}V_N+\gamma _{n}. \end{aligned}$$

By Lemma 2.4, we have

$$\begin{aligned} \displaystyle \lim _{n\rightarrow \infty }I(u_{n})= & {} \displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)-\frac{1}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx\nonumber \\&\displaystyle - \frac{1}{6}\left( \int _{B_r}u^6dx+N\int _{{\mathbb {R}}^3}V_N^{6}dx\right) \nonumber \\= & {} \displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)-\left( \frac{1}{p}-\frac{1}{6}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx\nonumber \\&\displaystyle - \frac{1}{6}M(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)\left( \Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx\right) ,\nonumber \\ \end{aligned}$$

(3.1)

where we have used the fact that,

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle M(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)\int _{B_r}\Vert u\Vert ^2=\displaystyle \int _{B_r}u^6dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx,\\ \displaystyle M(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx=\displaystyle \int _{{\mathbb {R}}^3}V_N^6dx. \end{array}\right. } \end{aligned}$$

Noting that

$$\begin{aligned}&\displaystyle S^{-3}\left( \int _{{\mathbb {R}}^3}|\nabla V_N|^2dx\right) ^3=\displaystyle \int _{{\mathbb {R}}^3}V_N^6dx\nonumber \\&\qquad =\displaystyle M\left( \Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla V_N|^2dx. \end{aligned}$$

(3.2)

Using the following lemma, we can solve equation (3.2) for $\displaystyle \int _{{\mathbb {R}}^3}|\nabla V_N|^2dx$.

Lemma 3.1

Give $s\ge 0$, the equation $M(s+Nt)=S^{-3}t^2$ has a unique positive solution $t:={\mathcal {F}}_N(s)$. The function ${\mathcal {F}}_N$ is continuously differentiable. Moreover, ${\mathcal {F}}_N(s)\ge {\mathcal {F}}_1(0):=T$, where T is the unique positive solution of the equation $M(t)=S^{-3}t^2$.

Proof

By the assumption $(M_2)$, the function

$$\begin{aligned} g(t,s)=\frac{M(s+Nt)}{t^2}=\frac{M(s+Nt)}{(s+Nt)^2}\frac{(s+Nt)^2}{t^2} \end{aligned}$$

is strictly decreasing in t, and

$$\begin{aligned} \lim _{t\rightarrow +\infty }g(t,s)=0,~~~\lim _{t\rightarrow 0^+}g(t,s)=+\infty . \end{aligned}$$

Hence, there exists a unique $t>0$, denoted by ${\mathcal {F}}_N(s)$, satisfies the equation $g(t,s)=S^{-3}$, that is,

$$\begin{aligned} M(s+Nt)=S^{-3}t^2. \end{aligned}$$

Since M is a continuously differentiable function and

$$\begin{aligned} \frac{\partial }{\partial t}g(t,s)=\frac{1}{t^3}(NtM'(s+Nt)-2M(s+Nt))<0, \end{aligned}$$

so is the function $t={\mathcal {F}}_N(s)$ by the implicit function theorem. Finally, by the assumption $(M_1)$ for $t={\mathcal {F}}_N(s)$

$$\begin{aligned} \frac{M(s+{\mathcal {F}}_N(s))}{{\mathcal {F}}^2_N(s)}=S^{-3}=g(t)=\frac{M(s+Nt)}{t^2}\ge \frac{M(t)}{t^2}, \end{aligned}$$

and by assumption $(M_3)$, ${\mathcal {F}}_N(s)=t\ge T={\mathcal {F}}_1(0)$. The proof is complete.$\square$

As a result of (3.2) and Lemma 3.1, we obtain

$$\begin{aligned} \int _{{\mathbb {R}}^3}|\nabla V_N|^2dx={\mathcal {F}}_N(\Vert u\Vert ^2) \end{aligned}$$

(3.3)

and rewrite Formula (3.1) as

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \lim _{n\rightarrow \infty }I(u_{n})&{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N\int _{{\mathbb {R}}^3}|\nabla V_N|^2dx)-\frac{1}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx\\ &{}&{}\displaystyle - \frac{1}{6}\left( \int _{B_r}u^6dx+N\int _{{\mathbb {R}}^3}V_N^6dx\right) \\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx~~(\mathrm {by}~(3.3))\\ &{}&{}\displaystyle - \frac{1}{6}\left( \int _{B_r}u^6dx+NS^{-3}{\mathcal {F}}^3_N(\Vert u\Vert ^2)\right) \\ &{}=&{}\displaystyle I_N(u) \end{array} \right. \nonumber \\ \end{aligned}$$

(3.4)

where

$$\begin{aligned} \int _{{\mathbb {R}}^3}V_N^6dx=S^{-3}\left( \int _{{\mathbb {R}}^3}|\nabla V_N|^2dx\right) ^3=S^{-3}{\mathcal {F}}^3_N(\Vert u\Vert ^2). \end{aligned}$$

Also, we rewrite equation (in Lemma 2.4 ($2_b$)) satisfied by u as

$$\begin{aligned} M(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))\int _{B_r}\nabla u\nabla \varphi dx=\int _{B_r}u^5\varphi dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^{p-1}\varphi dx\nonumber \\ \end{aligned}$$

(3.5)

for $\varphi \in H$. Define

$$\begin{aligned} \Sigma _N= & {} \{u|u\in H, ~u~ \mathrm {satisfies that the equation} (3.5)\}, \\ \mu _N= & {} \inf \{I_N(u)|u\in \Sigma _N\}. \end{aligned}$$

The following lemma gives the lower bound for $\mu _N$.

Lemma 3.2

There exists a constant $C_1$ (independent of N) such that $\mu _N\ge ND$, where $D=\frac{1}{2}{\widehat{M}}(T)-\frac{1}{6}M(T)T$ and $T=\displaystyle \int _{{\mathbb {R}}^3}|\nabla V_N|^2dx.$

Proof

Let $u\in \Sigma _N$, it follows that

$$\begin{aligned} M(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))\Vert u\Vert ^2=\int _{B_r}u^6dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx. \end{aligned}$$

It follows from (3.2) and (3.3) that

$$\begin{aligned} M(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))=S^{-3}{\mathcal {F}}^2_N(\Vert u\Vert ^2). \end{aligned}$$

At this moment, by (3.4), we have

$$\begin{aligned} I_N(u)= & {} \displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx\nonumber \\&\displaystyle - \frac{1}{6}\left( \int _{B_r}u^6dx+NS^{-3}{\mathcal {F}}_N^3(\Vert u\Vert ^2)\right) \nonumber \\= & {} \displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{6}M(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))\Vert u\Vert ^2\nonumber \\&\displaystyle +\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx-\frac{1}{6}NS^{-3}{\mathcal {F}}_N^3(\Vert u\Vert ^2))\nonumber \\= & {} \displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))+\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx\nonumber \\&\displaystyle - \frac{1}{6}M(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2))(\Vert u\Vert ^2+N{\mathcal {F}}_N(\Vert u\Vert ^2)). \end{aligned}$$

(3.6)

Let

$$\begin{aligned} h(s)=\frac{1}{2}{\widehat{M}}(s)-\frac{1}{6}M(s)s. \end{aligned}$$

By the assumption $(M_3)$, one has

$$\begin{aligned} h(a+b)\ge h(a)+h(b),~~\mathrm {for}~a,b\in {\mathbb {R}}^+. \end{aligned}$$

(3.7)

In fact

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle h(a+b)&{}=&{}\displaystyle a\cdot \frac{h(a+b)}{a+b}+b\cdot \frac{h(a+b)}{a+b}\\ &{}\ge &{}\displaystyle a\cdot \frac{h(a)}{a}+b\cdot \frac{h(a)}{b}\\ &{}=&{}\displaystyle h(a)+h(b). \end{array} \right. \end{aligned}$$

By (3.7), (3.6) can be reduced to

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle I_N(u)&{}\ge &{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2)+\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u^pdx- \frac{1}{6}M(\Vert u\Vert ^2)\Vert u\Vert ^2\\ &{}&{}\displaystyle +\frac{1}{2}{\widehat{M}}(N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{6}M\left( N{\mathcal {F}}_N(\Vert u\Vert ^2)\right) N{\mathcal {F}}_N(\Vert u\Vert ^2)\\ &{}\triangleq &{} J(u)+G_N(u), \end{array} \right. \end{aligned}$$

(3.8)

where

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle J(u)= & {} \displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2)+\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta } u^pdx-\frac{1}{6}M(\Vert u\Vert ^2)\Vert u\Vert ^2, \end{array} \right. \end{aligned}$$

and

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle G_N(u)= & {} \displaystyle \frac{1}{2}{\widehat{M}}(N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{6}M(N{\mathcal {F}}_N(\Vert u\Vert ^2))N{\mathcal {F}}_N(\Vert u\Vert ^2). \end{array} \right. \end{aligned}$$

Since $p>6$, using the assumption $(M_3)$ again, we get

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle J(u)&{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2)+\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta } u^pdx-\frac{1}{6}M(\Vert u\Vert ^2)\Vert u\Vert ^2\\ &{}\ge &{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u\Vert ^2)-\frac{1}{6}M(\Vert u\Vert ^2)\Vert u\Vert ^2\\ &{}\ge &{}\displaystyle \frac{1}{3}a\Vert u\Vert ^2\\[4pt] &{}\ge &{}0. \end{array} \right. \end{aligned}$$

(3.9)

If $N\ge 1$, by Lemma 3.1, one obtains

$$\begin{aligned} N{\mathcal {F}}_N(\Vert u\Vert ^2)\ge NT\ge T. \end{aligned}$$

Hence, by the assumption $(M_3)$ again,

$$\begin{aligned} \left. \begin{array}{rcl} &{}&{}\displaystyle \frac{\frac{1}{2}{\widehat{M}}(N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{6}M\left( N{\mathcal {F}}_N(\Vert u\Vert ^2)\right) N{\mathcal {F}}_N(\Vert u\Vert ^2)}{N{\mathcal {F}}_N(\Vert u\Vert ^2)}\\[4pt] &{}\ge &{}\displaystyle \frac{\frac{1}{2}{\widehat{M}}(NT)-\frac{1}{6}M(NT)T}{NT}\\[4pt] &{}\ge &{}\displaystyle \frac{\frac{1}{2}{\widehat{M}}(T)-\frac{1}{6}M(T)T}{T}. \end{array} \right. \end{aligned}$$

As a result,

$$\begin{aligned} \left. \begin{array}{rcl} G_N(u)&{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(N{\mathcal {F}}_N(\Vert u\Vert ^2))-\frac{1}{6}M\left( N{\mathcal {F}}_N(\Vert u\Vert ^2)\right) N{\mathcal {F}}_N(\Vert u\Vert ^2)\\ &{}\ge &{}\displaystyle \left[ \frac{1}{2}{\widehat{M}}(T)-\frac{1}{6}M(T)T\right] \frac{N{\mathcal {F}}_N(\Vert u\Vert ^2)}{T}\\ &{}\ge &{}\displaystyle N\left[ \frac{1}{2}{\widehat{M}}(T)-\frac{1}{6}M(T)T\right] \\ &{}\triangleq &{}ND. \end{array} \right. \end{aligned}$$

(3.10)

Therefore, the estimate for $\mu _N$ follows from relations (3.8), (3.9) and (3.10), that is,

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \mu _N&{}=&{}\displaystyle \inf \{I_N(u)|u\in \Sigma _N\}\\ &{}\ge &{}\displaystyle \inf _{u\in \Sigma _N}J(u)+\inf _{u\in \Sigma _N} G_N(u)\\ &{}\ge &{} ND. \end{array} \right. \end{aligned}$$

The proof is complete.$\square$

Lemma 3.3

$\mu _1$ is achieved at the point 0 and $\mu _1=D$, where D is defined in Lemma 3.2.

Proof

Since $0\in \Sigma _1$, and

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle I_1(0)&{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}({\mathcal {F}}_1(0))-\frac{1}{6}S^{-3}{\mathcal {F}}^3_1(0)\\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}({\mathcal {F}}_1(0))-\frac{1}{6}M({\mathcal {F}}_1(0)){\mathcal {F}}_1(0)\\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(T)-\frac{1}{6}M(T)T\\ &{}=&{}D. \end{array} \right. \end{aligned}$$

(3.11)

By Lemma 3.2, for any $u\in \Sigma _1$, we know that

$$\begin{aligned} \left. \begin{array}{rcl} I_1(u)&{}\ge &{}J(u)+G_1(u)\\ &{}\ge &{}\displaystyle G_1(u)\\ &{}\ge &{}D. \end{array} \right. \end{aligned}$$

Hence, by the definition of $\mu _1$ and (3.11), which is achieved at 0 and $\mu _1=D$, the proof is complete.$\square$

Lemma 3.4

Under the assumptions of Theorem 1.1, the functional I satisfies the $(PS)_c$ condition provided $c<D$, where D is defined in Lemma 3.2.

Proof

Let $\{u_n\}\subset H$ be such that

$$\begin{aligned} I(u_n)\rightarrow c,~~I'(u_n)\rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty$. By Lemma 2.4, $\{u_n\}$ has the profile decomposition (2.12)

$$\begin{aligned} u_n=u+\sum _{k=1}^Ng_{n,k}V_N+\gamma _n. \end{aligned}$$

By Lemma 3.2, we deduce that

$$\begin{aligned} D>c=\lim _{n\rightarrow \infty }I(u_n)=I_N(u)\ge J(u)+G_N\ge G_N(u)\ge \mu _N\ge ND. \end{aligned}$$

which implies that $N=0$. Consequently, $u_n=u+o(1)$ in H as $n\rightarrow \infty$. That is,

$$\begin{aligned} \int _{B_r}|\nabla u_n|^2dx\rightarrow \int _{B_r}|\nabla u|^2dx,~~\int _{B_r}u_n^6dx\rightarrow \int _{B_r}u^6dx \end{aligned}$$

as $n\rightarrow 0$. The proof is complete.$\square$

In the following, we estimate the threshold value of I. Denote

$$\begin{aligned} U(x)=\frac{3^{\frac{1}{4}}}{(1+|x|^2)^{\frac{1}{2}}},~~U_\varepsilon (x)=\frac{3^{\frac{1}{4}}\varepsilon ^{\frac{1}{2}}}{(\varepsilon ^2+|x|^2)^{\frac{1}{2}}},~x\in {\mathbb {R}}^3,~\varepsilon >0. \end{aligned}$$

(3.12)

U (and $U_\varepsilon$) satisfies the limit equation

$$\begin{aligned} \Delta U+U^5=0,~~U>0~\mathrm {in}~{\mathbb {R}}^3. \end{aligned}$$

Choose $\eta \in C_0^\infty (B_1(x_0), [0,1])$ where $B_1(x_0)\subset B_r$ such that $\eta (x)=1$ near $x=x_0$. Denote $\varphi _\varepsilon =U_\varepsilon \eta$.

Lemma 3.5

Assume that $(M_1)$ and $(M_2)$ hold. Then, $\sup _{t\ge 0}I(t\varphi _\varepsilon )<D$ for sufficient small $\varepsilon >0$, where D is defined in Lemma 3.2.

Proof

From Lemma 1.1 in [28], we know

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle \int _{B_r}|\nabla \varphi _\varepsilon |^2dx=\displaystyle \int _{{\mathbb {R}}^3}|\nabla U|^2dx+O(\varepsilon )=S^{\frac{3}{2}}+O(\varepsilon ),\\ \displaystyle \int _{B_r}\varphi _\varepsilon ^{6}dx=\displaystyle \int _{{\mathbb {R}}^3}U^{6}dx+O(\varepsilon ^3)=S^{\frac{3}{2}}+O(\varepsilon ^3). \end{array}\right. } \end{aligned}$$

Therefore, we deduce that

$$\begin{aligned} \int _{B_r}|\nabla \varphi _\varepsilon |^2dx\rightarrow \int _{{\mathbb {R}}^3}|\nabla U|^2dx~~\mathrm {as}~\varepsilon \rightarrow 0. \end{aligned}$$

Consequently, according to the continuity of ${\widehat{M}}$, we obtain

$$\begin{aligned} {\widehat{M}}\left( \int _{B_r}|\nabla \varphi _\varepsilon |^2dx\right) ={\widehat{M}}\left( \int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) +O(\varepsilon ). \end{aligned}$$

By the definition of $\varphi _\varepsilon$, we infer that

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }\varphi _\varepsilon ^pdx&{}=&{}\displaystyle C\int _{|x|\le 1}\frac{|x|^\beta }{1+|x|^\beta }\frac{\varepsilon ^{\frac{p}{2}}}{(\varepsilon ^2+|x|^2)^{\frac{p}{2}}}dx\\ &{}\ge &{}\displaystyle C\int _{|x|\le 1}\frac{|x|^\beta }{2}\frac{\varepsilon ^{\frac{p}{2}}}{(\varepsilon ^2+|x|^2)^{\frac{p}{2}}}dx\\ &{}=&{}\displaystyle C\varepsilon ^{\beta +3-\frac{p}{2}}\int _0^{1/\varepsilon }\frac{t^{2+\beta }}{(1+t^2)^{\frac{p}{2}}}dt\\ &{}=&{}\displaystyle C\varepsilon ^{\beta +3-\frac{p}{2}}\int _0^1\frac{t^{2+\beta }}{(1+t^2)^{\frac{p}{2}}}dt+C\varepsilon ^{\beta +3-\frac{p}{2}}\int _0^{1/\varepsilon }\frac{t^{2+\beta }}{(1+t^2)^{\frac{p}{2}}}dt\\ &{}=&{}\displaystyle C\varepsilon ^{\beta +3-\frac{p}{2}}\int _0^1\frac{t^{2+\beta }}{(1+t^2)^{\frac{p}{2}}}dt\\ &{}\ge &{}\displaystyle C\varepsilon ^{\beta +3-\frac{p}{2}}\int _0^1t^{2+\beta }dt\\ &{}=&{}\displaystyle C\varepsilon ^{\beta +3-\frac{p}{2}} \end{array} \right. \end{aligned}$$

for some $C>0$. Since $M(t)=o(t^2)$, ${\widehat{M}}(t)=o(t^3)$ as $t\rightarrow +\infty$, $I(t\varphi _\varepsilon )\rightarrow -\infty$ as $t\rightarrow \infty$. We can assume there exist $0<t_1<t_2$ such that

$$\begin{aligned} \sup _{t\ge 0}I(t\varphi _\varepsilon )=\sup _{t\in [t_1,t_2]}I(t\varphi _\varepsilon ). \end{aligned}$$

From the above information, there holds

$$\begin{aligned} \left. \begin{array}{rcl} I(t\varphi _\varepsilon )&{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{t^6}{6}\int _{{\mathbb {R}}^3}U^6dx-\frac{t^p}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }\varphi _\varepsilon ^pdx+O(\varepsilon )\\ &{}\le &{}\displaystyle \frac{1}{2}{\widehat{M}}\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{t^6}{6}\int _{{\mathbb {R}}^3}U^6dx-\frac{t_1^p}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }\varphi _\varepsilon ^pdx+O(\varepsilon )\\ &{}\le &{}\displaystyle \frac{1}{2}{\widehat{M}}\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{t^6}{6}\int _{{\mathbb {R}}^3}U^6dx-C\varepsilon ^{\beta +3-\frac{p}{2}}+C\varepsilon . \end{array} \right. \nonumber \\ \end{aligned}$$

(3.13)

Define

$$\begin{aligned} g(t)=\frac{1}{2}{\widehat{M}}\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{t^6}{6}\int _{{\mathbb {R}}^3}U^6dx. \end{aligned}$$

Then,

$$\begin{aligned} \left. \begin{array}{rcl} g'(t)&{}=&{}\displaystyle tM\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx-t^5\int _{{\mathbb {R}}^3}U^6dx\\ &{}=&{}\displaystyle tM\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx-t^5S^{-3}\left( \int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) ^3\\ &{}=&{}\displaystyle t\int _{{\mathbb {R}}^3}|\nabla U|^2dx\left[ M\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -S^{-3}\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) ^2\right] . \end{array} \right. \end{aligned}$$

Let $t_0>0$ be the unique positive zero, according to $g'(t_0)=0$, one has

$$\begin{aligned} M\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) =S^{-3}\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) ^2. \end{aligned}$$

(3.14)

By Lemma 3.1 and (3.14), we have

$$\begin{aligned} t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx={\mathcal {F}}_1(0). \end{aligned}$$

Furthermore, we have

$$\begin{aligned} \left. \begin{array}{rcl} g(t_0) &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{t_0^6}{6}\int _{{\mathbb {R}}^3}U^6dx\\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{1}{6}S^{-3}\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) ^3\\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}\left( {\mathcal {F}}_1(0)\right) -\frac{1}{6}S^{-3}{\mathcal {F}}_1^3(0)\\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}\left( {\mathcal {F}}_1(0)\right) -\frac{1}{6}M({\mathcal {F}}_1(0)){\mathcal {F}}_1(0)\\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(T)-\frac{1}{6}M(T)T\\ &{}=&{}\mu _1=D. \end{array} \right. \end{aligned}$$

(3.15)

Moreover, we have

$$\begin{aligned} \left. \begin{array}{rcl} g''(t)&{}=&{}\displaystyle M\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx\\ &{}&{}\displaystyle +2t^2M'\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \left( \int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) ^2-5t^4\int _{{\mathbb {R}}^3}U^6dx. \end{array} \right. \end{aligned}$$

Noting that

$$\begin{aligned} 0=g'(t_0)=t_0M\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx-t_0^5\int _{{\mathbb {R}}^3}U^6dx, \end{aligned}$$

and hence by the assumption $(M_1)$, $(M_2)$, we have

$$\begin{aligned} \left. \begin{array}{rcl} g''(t_0)&{}=&{}\displaystyle -4M\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx\\ &{}&{}\displaystyle +\,2t_0^2M'\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \left( \int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) ^2\\ &{}\le &{}\displaystyle -\,2t_0^2M'\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx\\ &{}<&{}0 \end{array} \right. \end{aligned}$$

provided $M'\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) >0$. In case

$$\begin{aligned} M'\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) =0, \end{aligned}$$

then

$$\begin{aligned} g''(t_0)=-4M\left( t_0^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) \int _{{\mathbb {R}}^3}|\nabla U|^2dx. \end{aligned}$$

Again, we obtain $g''(t_0)<0$. Since $t_0$ is the unique stationary point of g and $g''(t_0)<0$, which implies that g achieves its maximum at $t_0$, namely

$$\begin{aligned} g(t)\le g(t_0),~~\mathrm {for}~t\in [t_1, t_2]. \end{aligned}$$

Therefore, by (3.13) and (3.15), we have for $t\in [t_1,t_2]$

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle I(t\varphi _\varepsilon ) &{}\le &{}\displaystyle \frac{1}{2}{\widehat{M}}\left( t^2\int _{{\mathbb {R}}^3}|\nabla U|^2dx\right) -\frac{t^6}{6}\int _{{\mathbb {R}}^3}U^6dx-C\varepsilon ^{\beta +3-\frac{p}{2}}+C\varepsilon \\ &{}\le &{}\displaystyle g(t_0)+C\varepsilon -C\varepsilon ^{\beta +3-\frac{p}{2}}\\ &{}<&{}\displaystyle D \end{array} \right. \end{aligned}$$

for some $C>0$ and $\varepsilon >0$ is small enough. This leads us to the proof.$\square$

Proof of Theorem 1.1

It is easy to verify that I has a mountain pass geometry. Indeed, by condition $(M_1)$, we have

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle I(u) &{}\ge &{}\displaystyle \frac{a}{2}\Vert u\Vert ^2-\frac{1}{6}\int _{B_r}|u|^6dx-\frac{1}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }|u|^pdx\\ &{}\ge &{}\displaystyle \frac{a}{2}\Vert u\Vert ^2-C_1\Vert u\Vert ^6-C_2\Vert u\Vert ^p~~(\mathrm {by}~(2.2), (2.3)), \end{array} \right. \end{aligned}$$

where $C_1, C_2>0$. Thus, there exist $\alpha , \rho >0$ and $e\in H$ with $\Vert e\Vert >\rho$ such that $I(u)>\alpha$ for all $\Vert u\Vert =\rho$, and $I(e)<0$ (since $\lim _{t\rightarrow +\infty }I(tu)\rightarrow -\infty$). Applying the mountain pass lemma [29], there is a sequence $\{u_n\}\subset H$ such that

$$\begin{aligned} I(u_n)\rightarrow c>0~~~\mathrm {and}~~~I'(u_n)\rightarrow 0, \end{aligned}$$

where

$$\begin{aligned} c=\inf _{\gamma \in \Gamma }\max _{t\in [0,1]}I(\gamma (t)), \end{aligned}$$

and

$$\begin{aligned} \Gamma =\left\{ \gamma \in C([0, 1], H): \gamma (0)=0, \gamma (1)=e\right\} . \end{aligned}$$

From Lemmas 3.4 and 3.5, $\{u_n\}$ has a convergent subsequence (still denoted by $\{u_n\}$) and there exists $u_*\in H$ such that $u_n\rightarrow u_*$ in H. Moreover, $u_*$ is a solution of problem (1.1) and

$$\begin{aligned} 0<c=\lim _{n\rightarrow \infty }I(u_n)=I(u_*)<D. \end{aligned}$$

Therefore, we infer that $u_*\not \equiv 0$. By the strong maximum principle, we obtain $u_*>0$ in $B_r$.

By the Pohozaev equality, there holds

$$\begin{aligned} \left. \begin{array}{rcl} \displaystyle \frac{1}{2}M(\Vert u_*\Vert ^2)\Vert u_*\Vert ^2 &{}=&{}\displaystyle \frac{1}{2}\int _{B_r}u_*^6dx+\frac{3}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_*^pdx\\ &{}&{}\displaystyle -\frac{1}{2}M(\Vert u_*\Vert ^2)\int _{\partial B_r}(\frac{\partial u_*}{\partial n})^2\cdot x\cdot nd\delta . \end{array} \right. \end{aligned}$$

Noting that $x\cdot n\ge 0$, we have

$$\begin{aligned} \frac{1}{2}M(\Vert u_*\Vert ^2)\Vert u_*\Vert ^2\le \frac{1}{2}\int _{B_r}u_*^6dx+\frac{3}{p}\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_*^pdx. \end{aligned}$$

(3.16)

Since $u_*$ is a positive solution of problem (1.1), it follows that

$$\begin{aligned} M(\Vert u_*\Vert ^2)\Vert u_*\Vert ^2=\int _{B_r}u_*^6dx+\int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_*^pdx. \end{aligned}$$

(3.17)

It follows from (3.16) and (3.17) that

$$\begin{aligned} \left( \frac{1}{2}-\frac{3}{p}\right) M(\Vert u_*\Vert ^2)\Vert u_*\Vert ^2\le \left( \frac{1}{2}-\frac{3}{p}\right) \int _{B_r}u_*^6dx. \end{aligned}$$

By condition $(M_1)$, we deduce that

$$\begin{aligned} a\Vert u_*\Vert ^2\le M(\Vert u_*\Vert ^2)\Vert u_*\Vert ^2\le \int _{B_r}u_*^6dx\le S^3\Vert u_*\Vert ^6, \end{aligned}$$

that is,

$$\begin{aligned} \Vert u_*\Vert ^2\ge \sqrt{aS^3}. \end{aligned}$$

Besides, by (3.17) and condition $(M_1)$

$$\begin{aligned} \left. \begin{array}{rcl} D>c&{}=&{}\displaystyle I(u_*)-\frac{1}{6}\langle I'(u_*),u_*\rangle \\ &{}=&{}\displaystyle \frac{1}{2}{\widehat{M}}(\Vert u_*\Vert ^2)-\frac{1}{6}M(\Vert u_*\Vert ^2)\Vert u_*\Vert ^2\\ &{}&{}\displaystyle +\left( \frac{1}{6}-\frac{1}{p}\right) \int _{B_r}\frac{|x|^\beta }{1+|x|^\beta }u_*^pdx\\ &{}\ge &{}\displaystyle \frac{a}{4}\Vert u_*\Vert ^2, \end{array} \right. \end{aligned}$$

which implies that

$$\begin{aligned} \sqrt{aS^3}\le \Vert u_*\Vert ^2\le \frac{4D}{a}<+\infty . \end{aligned}$$

Therefore,

$$\begin{aligned} 0\le \lim _{r\rightarrow \infty }u_*(r)\le \lim _{r\rightarrow \infty }\frac{C\Vert u_*\Vert }{\sqrt{r}}=0, \end{aligned}$$

that is, $\lim _{r\rightarrow \infty }u_*(r)=0$. The proof is complete.$\square$