1 Introduction and Main Result

In this article, we focus on the following critical Kirchhoff-type equations with strong singularity, which involve the fractional Laplacian:

$$\begin{aligned} \left\{ \begin{array}{ll} M(\iint _{\mathbb R^{2N}} \frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2s}} \textrm{d}x{d}y) \left( -\Delta \right) ^su=f(x)u^{-\gamma }-h(x)u^{2_{s}^{*}-1},&{}\text {in}~\Omega ,\\ u>0,&{}\text {in}~\Omega ,\\ u=0,&{}\text {in}~\mathbb {R}^{N}\backslash \Omega , \end{array} \right. \end{aligned}$$
(1.1)

where \(\Omega \) is a smooth bounded domain in \(\mathbb {R}^{N}\), \(N>2s\) with \(s \in \left( 0,1\right) \), \(2_{s}^{*}=2N/(N-2s)\) is the fractional critical Sobolev exponent, h(x) is a nonnegative function and \(h(x)\in L^{\infty }(\Omega )\), \(\gamma >1\) and \(f\in L^{1}(\Omega )\) is positive almost everywhere in \(\Omega \), \(M(t)=a+bt^{m-1}\), \(a,b\ge 0\), \(a+b> 0\), \(m\geqslant 1\), and \(\left( -\Delta \right) ^{s}\) is the fractional Laplace operator defined by, up to normalization factors,

$$\begin{aligned} \left( -\Delta \right) ^{s}\phi (x)=2\lim _{\tau \rightarrow 0^{+}} \int _{\mathbb {R}^{N}\backslash B_{\tau }(x)} \frac{\phi (x)-\phi (y)}{|x-y|^{N+2s}}\textrm{d}y , x \in \mathbb {R}^{N} , \end{aligned}$$
(1.2)

along any \(\phi \in C_{0}^{\infty }(\Omega )\), where \(B_{\tau }(x)\) is the ball centered at \(x \in \mathbb {R}^{N} \) with the radius \(\tau >0\).

First of all, let us recall some references about Laplacian equations involving singular term. On the one hand, the real applications of singular problems like (1.1) with \(\gamma \in (0, +\infty )\) are closely related to some physical models such as non-Newtonian fluids, pseudo-plastic flows, biological pattern formation, chemical heterogeneous catalysts, and electrically conducting materials, see [22] and the references therein for more details. In particular, we mention some advances of p-Laplacian equations involving singular terms as follows.

$$\begin{aligned} \left\{ \begin{array}{cc} -\Delta _{p}u=\lambda k(x)u^{-\gamma }+\rho u^{q}, &{}\text {in} \Omega \\ u>0,&{}\text {in}~\Omega \\ u=0,&{}\text {in}~\partial \Omega . \end{array} \right. \end{aligned}$$
(1.3)

In the setting of \(0<\gamma <1\), when \(p=2\), \(\rho =0\) and \(k(x)\equiv 1\), the author obtained in [9] that (1.3) has a unique weak solution. Sun et al. first studied on multiplicity of positive solutions of singular elliptic equations in [34]. In [14], Fiscella proved that problem (1.3) involving critical nonlinearity has two different solutions. Furthermore, the authors in [26] studied Kirchhoff equation with Hardy potential and \(u^{-\gamma }\). On the other hand, for \(p=2\), \(\rho \equiv k(x)=1\) and \(1<q\le 2^{*}=2N/(N-2)\), the multiplicity of weak solutions was established by applying Nehari manifold method and sub- and supersolutions technique in [16, 18]. When \(\gamma =1\), Wang and Yan [35] obtained the unique positive solution by using a minimization argument as \(1<q<2^{*}-1\). With regard to \(\gamma >1\), when \(p=2, \lambda \equiv 1\) and \(\rho =0\), Lazer and McKenna in [20] obtained the uniqueness result of problem (1.3). The authors in [17] considered the similar form of (1.3) based on a local minimization argument. As a result, they solved the singularity case \(\gamma \ge 1\).

Meanwhile, the Kirchhoff model simulates several physical and biological systems, and we can refer to [19] for further details. With respect to the Kirchhoff term, we presume that \(M: \mathbb R_{0}^{+}\rightarrow \mathbb R_{0}^{+}\) is a continuous function, and we consider a specific model of M,

$$\begin{aligned} M(t)=a+bt^{m-1} , a,b\geqslant 0, a+b>0, m\geqslant 1 . \end{aligned}$$
(1.4)

If M is this type, \(M(t) \geqslant C >0\) for all \(t \in \mathbb R_{0}^{+}\) and some constant C, problem (1.1) is known as non-degenerate and this happens, for instance, if \(a>0\) and b \(\geqslant \) 0 in the case of (1.4), whereas if \(a=0\) and \(b>0\), that is, if \(M(0)=0\) but \(M(t)>0\) for all \(t \in \mathbb R^{+}\), Kirchhoff equation similar to (1.1) is commonly known as degenerate. For some recent literature on Kirchhoff-type problems in the non-degenerate case, see [7, 11, 15, 35, 38], and in the degenerate case, see [2, 3, 6, 39]. It is worth mentioning that the mathematical model was initiated by Kirchhoff in [19], which is relevant with problem (1.1), as shown below:

$$\begin{aligned} \rho u_{tt}-\left( a+b\left( \int _{0}^{L} u_{x}^{2}dx\right) ^{m-1}\right) u_{xx}=0. \end{aligned}$$
(1.5)

In the above equation, a, b, L, \(\rho \) are constants, and \(M(\int _{0}^{L}u_{x}^{2}dx):=a+b(\int _{0}^{L}u_{x}^{2}dx)^{m-1}\) describes the change of tension caused by the increase of string length during vibration. In this sense, the actual meaning of \(M(0) = 0\) could be understood that the fundamental tension of the string is zero.

A Kirchhoff-type version related to problem (1.1) is shown below:

$$\begin{aligned} \left\{ \begin{array}{ll} -M\left( \int _{\Omega }|\nabla u(x)|^{2}\right) \Delta u=\lambda u^{-\gamma }+u^{q}&{}\text {in}~ \Omega ,\\ u>0,&{}\text {in}~ \Omega ,\\ u=0,&{}\text {in} ~\partial \Omega . \end{array} \right. \end{aligned}$$
(1.6)

When \(M\equiv 1\), the author in [18] proved the existence of two weak solutions for (1.6) by variational methods and the method of sub- and supersolutions. Meanwhile, under the influence of Nehari method, the author in [16] also got two weak solutions for (1.6). In the Kirchhoff context, we refer to [21, 24, 25] for various Kirchhoff-type Laplacian problems with the different conditions of M similar to (1.4) involving a singular term of type \(u^{-\gamma }\). More specifically, in [21], presuming \(a>0\) in (1.4), Lei et al. testified the existence of two weak solutions for a Kirchhoff problem like (1.6) with the aim of variational methods and combining perturbation. In [24], the authors focused on a singular Kirchhoff problem with a subcritical term by employing the Nehari method. In particular, by using the method of minimizing parameters, Liao et al. in [25] confirmed that a singular Kirchhoff problem involving negative critical nonlinearity has a unique weak solution.

In recent years, researchers are particularly interested in non-local operator equations, especially the related problems with fractional Laplace operators. This kind of problem comes from some different kinds of practical problems, such as financial market problems, phase transformation problems, abnormal diffusion problems, semipermeable membrane problems, and minimal surface problems. For more details about the background of fractional Laplacian, one can be referred to [1, 10, 12, 23]. Therefore, the models with fractional Laplace operator are important to the study of modern natural phenomena, especially its nonlinear equation. More detailed information about the introduction of fractional Laplace operators and the fractional Sobolev spaces is referred to [29] and the recent monograph [27].

In the above context, the following singular Kirchhoff problem driven by the fractional Laplacian with critical growth has been discussed by Nehari manifold method recently:

$$\begin{aligned} \left\{ \begin{array}{ll} M(\iint _{\mathbb R^{2N}} \frac{|u(x)-u(y)|^{2}}{|x-y|^{N+2s}} \textrm{d}x{d}y) \left( -\Delta \right) ^su=\lambda f(x)u^{-\gamma }+g(x)u^{p},&{}\text {in}~\Omega \\ u>0,&{}\text {in}~\Omega \\ u=0,&{}\text {in}~\mathbb {R}^{N}\backslash \Omega , \end{array} \right. \end{aligned}$$
(1.7)

where \(\gamma \in (0,1)\). When \(f, g\equiv 1\) and \(\lambda \) satisfies various specific conditions, Fiscella in [14] testified that (1.7) has at least two distinct solutions for (1.7) with \(p=2_{s}^{*}-1\) under the degenerate case via the mountain pass theorem together with truncation method. When \(f\in L^{2_{s}^{*}/(2_{s}^{*}+\gamma -1)}(\Omega )\) and \(g\in L^{\infty }(\Omega )\) is sign-changing function, Fiscella and Mishra in [15] obtained the existence of two distinct solutions to (1.7) with \(p=2_{s}^{*}-1\) in the non-degenerate case by using the Nehari manifold technique. As to the study of this type of problem, we just suggest a few literature, for instance, see [2, 8] and the references therein for more results. Concerning the other singular problems of Kirchhoff type, we just sketch some recent literature in the following. Wang et al. in [36] proved the existence and multiplicity of solutions for a class of fractional Kirchhoff problems with Choquard-type nonlinearity. Under appropriate conditions, they obtained two weak solutions by using Nehari manifold method and Hardy–Littlewood–Sobolev inequality. Song and Xiang in [32] proved that there are two positive solutions to the singular nonlinear weighted fractional p-Laplace problem by the fractional Caffarelli–Kohn–Nirenberg inequalities and Nehari manifold method. Last but not the least, the authors in [28] studied a class of fractional Kirchhoff problems with \(\gamma < 1\) and \(N=sp\) involving exponential nonlinearity. Accordingly, they concluded that the equation has at least two weak solutions in the degenerate case via Nehari manifold techniques.

Inspired by the aforementioned works, especially by [3, 4, 14, 15, 31, 35], we are particularly interested in whether the problem (1.1) with \(\gamma >1\) and \(q=2_{s}^{*}-1\) has a unique weak solution under certain assumption. The most fundamental difficulties lie in the non-integrability of singular term and the lack of compactness due to the critical term. For this, we give appropriate constraints to restore the integrability of \(u^{-\gamma }\). Besides, a novelty is that we use the Brézis–Lieb lemma to restore the compactness of the energy functional in Sect.  4. Finally, it may be interesting to consider that if Kirchhoff term M satisfies more general conditions (cf. [30]), can problem (1.1) be solved? This problem would be further studied in our next work.

Now, we are in a position to state our major result as follows.

Theorem 1.1

Let \(\gamma >1\) and \(f\in L^{1}(\Omega )\) is positive a.e. in \(\Omega \). Then, problem (1.1) has a unique positive solution \(u_{0}\in G_{0}\) if and only if there is \(u_{0}\in G_{0}\) such that

$$\begin{aligned} \int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx<+\infty . \end{aligned}$$
(1.8)

Remark 1.1

Compared with the main result obtained in [37], there are three different points: (i) Our nonlinearity involves the critical exponent; (ii) we extend the case of \(M(t)=a+bt\) to the one of \(M(t)=a+bt^{m-1}\); and (iii) we consider N dimension instead of three dimension. Of course, we would like to stress the significance of the first item since we need to get over a serious difficulty due to the lack of compactness, and hence, much delicate analysis techniques are involved.

The rest of this article is organized as follows. The variational formulation of problem (1.1) and the Nehari manifold structure are described in Sect. 2. In Sect.  3, we analyze the fiber mapping and state the relevant lemmas will be used subsequently. In Sect. 4, the compactness of the energy functional is demonstrated. In Sect. 5, Theorem 1.1 is formally proved.

2 Variational Framework

Throughout the paper, we mainly research problem (1.1) on the fractional Sobolev space. The fractional Sobolev space is defined by

$$\begin{aligned} G&=\Bigg \{\Psi \Big | \Psi : \mathbb {R}^{N} \rightarrow \mathbb {R}\,\,\text {is measureable},\,\,\Psi \big |_{\Omega }\in L^{2}\left( \Omega \right) \,\,\text {and} \nonumber \\&\quad \iint _{E}\frac{|\Psi (x)-\Psi (y)|^{2}}{|x-y|^{N+2s}}dxdy<\infty \Bigg \}, \end{aligned}$$
(2.1)

where \(E=\left( \mathbb {R}^{N} \times \mathbb {R}^{N} \right) {\setminus } \left( C\Omega \times C\Omega \right) \) with \(C\Omega =\mathbb {R}^{N} {\setminus } \Omega \). The norm of the space G is endowed as follows:

$$\begin{aligned} ||\Psi ||_{G}=||\Psi ||_{L^{2}\left( \Omega \right) }+\left( \iint _{E}\frac{|\Psi (x)-\Psi (y)|^{2}}{|x-y|^{N+2s}}dxdy\right) ^{\frac{1}{2}}. \end{aligned}$$
(2.2)

Furthermore, we shall use \(G_{0}\) to represent the linear subspace of G, as shown below:

$$\begin{aligned} G_{0}:=\left\{ \Psi \in G:\Psi =0 \,\,\text {a.e. in}\,\,\mathbb R^{N}\setminus \Omega \right\} . \end{aligned}$$

Then, we consider the following norm on \(G_{0}\):

$$\begin{aligned} ||\Psi ||_{G_{0}}:=\left( \iint _{E}\frac{|\Psi (x)-\Psi (y)|^{2}}{|x-y|^{N+2s}}dxdy\right) ^{\frac{1}{2}}, \end{aligned}$$
(2.3)

which is equivalent to the usual one defined in (2.2) (see [33, Lemma 6]). By Lemma 7 in [33], we recall that \((G_{0},\Vert .\Vert _{G_{0}})\) is a Hilbert space, which can be endowed with the scalar product:

$$\begin{aligned} \langle \zeta ,\eta \rangle :=\langle \zeta ,\eta \rangle _{G_{0}}=\iint _{E}\frac{(\zeta (x)-\zeta (y))(\eta (x)-\eta (y))}{|x-y|^{N+2s}}dxdy,\,\,\text {for}\,\, \zeta ,\eta \in G_{0}. \end{aligned}$$
(2.4)

In addition, we recall the embedding \(G_{0}\hookrightarrow L^{r}(\Omega )\) which is continuous and compact for \(r \in [1,2_{s}^{*})\), see [33, Lemma 8]). There is an optimal constant \(S_{s}>0\) such that

$$\begin{aligned} S_{s}=\inf _{\Psi \in G_{0}\backslash \left\{ 0\right\} }S_{s}(\Psi )=\frac{\iint _{\mathbb R^{N}\times \mathbb R^{N}}\frac{|\Psi (x)-\Psi (y)|^{2}}{|x-y|^{N+2s}}dxdy}{\left( \int _{\mathbb R^{N}}|\Psi (x)|^{2^{*}_{s}}dx\right) ^{2/2^{*}_{s}}}. \end{aligned}$$
(2.5)

For simplicity, for any \(r \in [1,\infty ]\), we will denote \(\Vert .\Vert _{G_{0}}\) and \(\Vert .\Vert _{L^{r}(\Omega )}\) by \(\Vert .\Vert \) and \(\Vert .\Vert _{r}\), respectively. To get the existence and uniqueness result for problem (1.1), we will using variational methods to achieve this goal. It is standard to verify that problem (1.1) has a variational structure and the energy functional \(\mathcal {I}\): \(G_{0} \rightarrow \mathbb R\) is given by

$$\begin{aligned} \mathcal {I}(u)=\frac{a}{2}||u||^{2}+\frac{b}{2m}||u||^{2m}+\frac{1}{2_{s}^{*}}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-\frac{1}{1-\gamma }\int _{\Omega }f(x)|u|^{1-\gamma }dx. \end{aligned}$$
(2.6)

The energy functional \(\mathcal {I}\) fails to be Fr\(\acute{e}\)chet differentiable because of the singular term. To solve problem (1.1), two constrained sets are defined as follows:

$$\begin{aligned} \mathbb X=\left\{ u \in G_{0}:a||u||^{2}+b\Vert u\Vert ^{2m}+\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u|^{1-\gamma }dx \ge 0 \right\} ,\\ \mathbb X_{0}=\left\{ u \in G_{0}:a||u||^{2}+b\Vert u\Vert ^{2m}+\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u|^{1-\gamma }dx = 0 \right\} . \end{aligned}$$

If \(u\in G_{0}\) is a weak solution of problem (1.1), it means that there holds

$$\begin{aligned} a\langle u,\phi \rangle +b\Vert u\Vert ^{2m-2}\langle u,\phi \rangle +\int _{\Omega } h(x)|u|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u|^{-\gamma } \phi dx=0,\forall \phi \in G_{0}. \end{aligned}$$

3 Preliminary Results

In this section, we first give some auxiliary lemmas which will be used later.

Lemma 3.1

Assume that \(u_{n} \rightarrow u\) in \(G_{0}\), then

$$\begin{aligned} \lim _{n \rightarrow +\infty }\int _{\Omega }h(x)|u_{n}|^{2_{s}^{*}}dx=\int _{\Omega }h(x)|u|^{2_{s}^{*}}dx. \end{aligned}$$
(3.1)

Proof

Let \(\left\{ u_{n}\right\} \subset G_{0}\) and \(u_{n}\rightarrow u\) in \(G_{0}\). Due to \(h \in L^{\infty }(\Omega )\) and \(u_{n}\rightarrow u\), we infer that there exist \(C_{1}>0\) and \(C_{2}>0\) such that \(|h(x)|\le C_{1}\) a.e. in \(\Omega \) and \(\Vert u_{n}\Vert \le C_{2}\). Set \(f_{n}(x)=h(x)^{\frac{1}{2_{s}^{*}}}u_{n}\), \(f(x)=h(x)^{\frac{1}{2_{s}^{*}}}u\), then

$$\begin{aligned} \begin{aligned} \left( \int _{\Omega }|f_{n}(x)|^{2_{s}^{*}}dx\right) ^{\frac{1}{2_{s}^{*}}} =\left( \int _{\Omega }h(x)|u_{n}|^{2_{s}^{*}}dx \right) ^{\frac{1}{2_{s}^{*}}} \le {C_{1}}^{\frac{1}{2_{s}^{*}}} \left( \int _{\Omega }|u_{n}|^{2_{s}^{*}}dx\right) ^{\frac{1}{2_{s}^{*}}} \le {C_{1}}^{\frac{1}{2_{s}^{*}}} S_{s}^{-\frac{1}{2}}C_{2}. \end{aligned}\nonumber \\ \end{aligned}$$
(3.2)

Therefore, \(f_{n}\) is bounded in \(L^{2_{s}^{*}}(\Omega )\). Obviously, \(f_{n} \rightarrow f\) a.e. in \(\Omega \) as \(n \rightarrow \infty \). Furthermore,

$$\begin{aligned} \int _{\Omega }|f_{n}(x)-f(x)|^{2_{s}^{*}}dx= & {} \int _{\Omega }h(x)|u_{n}-u|^{2_{s}^{*}}dx \le C_{1} \int _{\Omega }|u_{n}-u|^{2_{s^{*}}}dx\nonumber \\\le & {} C_{1} \Vert u_{n}-u\Vert _{2_{s}^{*}}^{2} \rightarrow 0, \end{aligned}$$
(3.3)

as \(n\rightarrow \infty \). By the well-known Brézis–Lieb lemma, we have

$$\begin{aligned} \lim _{n\rightarrow +\infty }\left( \int _{\Omega }|f_{n}(x)|^{2_{s}^{*}}dx\right) ^{\frac{1}{2_{s}^{*}}}-\lim _{n\rightarrow +\infty } \left( \int _{\Omega }|f_{n}(x)-f(x)|^{2_{s}^{*}}dx\right) ^{\frac{1}{2_{s}^{*}}} =\left( \int _{\Omega }|f(x)|^{2_{s}^{*}}dx\right) ^{2_{s}^{*}}. \end{aligned}$$

On account of (3.3), we obtain (3.1). \(\square \)

Lemma 3.2

Let \(\gamma \in (1,\infty )\) and (1.8) hold true. Then, the following conclusions hold:

  1. (i)

    For any \(u \in G_{0}\), there exist unique \(t_{min}=t_{min}(u)>0\), such that \(t_{min}u \in \mathbb X_{0}\) and \(tu \in \mathbb X\) for any \(t>t_{min}\), \(\mathbb X\) and \(\mathbb X_{0}\) are non-empty. \(\mathcal {I}(t_{min}u)=\inf _{t>0}\mathcal {I}(tu)\).

  2. (ii)

    \(\mathbb X\) is an unbounded closed set in \(G_{0}\).

Proof

\(\left( i \right) \) Due to the presence of the singular term, we know that \(\mathcal {I}\) is not \(C^{1}\). For any \(u \in G_{0}\) and \(t>0\), we introduce the fibering function \(\psi _{u}(t):\mathbb R^{+} \rightarrow \mathbb R\) defined by

$$\begin{aligned} \psi _{u}(t)=\mathcal {I}(tu) \end{aligned}$$

which gives

$$\begin{aligned} \psi _{u}(t)=\frac{at^{2}}{2}||u||^{2}+\frac{bt^{2m}}{2m}||u||^{2m}+\frac{t^{2_{s}^{*}}}{2_{s}^{*}}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-\frac{t^{1-\gamma }}{1-\gamma }\int _{\Omega }f(x)|u|^{1-\gamma }dx. \end{aligned}$$

Thanks to \(\gamma >1\), we can get \(\lim _{t \rightarrow 0^{+}}\psi _{u}(t)=+\infty \) and \(\lim _{t \rightarrow +\infty }\psi _{u}(t)=+\infty \). In particular, a simple calculation shows that

$$\begin{aligned} \psi _{u}^{'}(t)=at||u||^{2}+bt^{2m-1}||u||^{2m}+t^{2_{s}^{*}-1}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-t^{-\gamma }\int _{\Omega }f(x)|u|^{1-\gamma }dx \end{aligned}$$

and

$$\begin{aligned} \psi _{u}^{''}(t)= & {} a||u||^{2}+(2m-1)bt^{2m-2}||u||^{2m}+(2_{s}^{*}-1)t^{2_{s}^{*}-2}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx\\{} & {} +\gamma t^{-\gamma -1}\int _{\Omega }f(x)|u|^{1-\gamma }dx. \end{aligned}$$

As a result, we can deduce that \(\psi _{u}^{'}(t)\) is increasing owing to \(\psi _{u}^{''}(t)>0 \) which holds for all \(t>0\) and \(\lim _{t \rightarrow 0^{+}} \psi _{u}^{'}(t)=-\infty ,\lim _{t \rightarrow +\infty }\psi _{u}^{'}(t)=+\infty \). According to the zero point theorem, we can get a unique \(t_{\min }>0\) such that \(\psi _{u}^{'}(t_{\min })=0\) and \(\psi _{u}^{'}(t)>0\) for any \(t>t_{\min }\). Since \(\psi _{u}^{'}(t_{\min })=0\), we have

$$\begin{aligned} at_{\min }||u||^{2}+bt^{2m-1}_{\min }||u||^{2m}+t^{2_{s}^{*}-1}_{\min }\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-t^{-\gamma }_{\min }\int _{\Omega }f(x)|u|^{1-\gamma }dx=0. \end{aligned}$$

Multiplying \(t_{\min }\) on both sides of the equation, one has

$$\begin{aligned} \begin{aligned} 0&=at_{\min }^{2}||u||^{2}+bt^{2m}_{\min }||u||^{2m}+t^{2_{s}^{*}}_{\min }\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-t^{1-\gamma }_{\min }\int _{\Omega }f(x)|u|^{1-\gamma }dx\\&=a||t_{\min }u||^{2}+b||t_{\min }u||^{2m}+\int _{\Omega } h(x)|t_{\min }u|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|t_{\min }u|^{1-\gamma }dx. \end{aligned} \end{aligned}$$
(3.4)

Consequently, \(t_{\min }u \in \mathbb X_{0}\). Similarly, \(tu \in \mathbb X\) for any \(t>t_{\min }\). This means that \(t_{\min }\) is an unique minimizing point of \(\psi _{u}(t)\), and \(\mathbb X\) and \(\mathbb X_{0}\) are non-empty.

  1. (ii)

    Now, we begin to prove \(\mathbb X\) is an unbounded closed set in \(G_{0}\). Firstly, we prove that \(\mathbb X\) is unbounded. We assume \(\mathbb X\) is bounded; that is, for all \(u \in \mathbb X\), there is \( \Lambda >0\), such that \(\Vert u\Vert \le \Lambda \). We know that \(tu \in \mathbb X\) for any \( t>t_{min}\). Therefore, \(\Vert tu\Vert \le \Lambda \). We can take t as \(\max \left\{ t_{\min }+1,\frac{\Lambda +1}{\Vert u\Vert }\right\} \), and then,

$$\begin{aligned} t\Vert u\Vert =\max \left\{ (t_{0}+1)\Vert u\Vert ,{\Lambda +1}\right\} >\Lambda . \end{aligned}$$

This is wrong. Hence, \(\mathbb X\) is unbounded.

Secondly, we should demonstrate \(\mathbb X\) is a closed set. Set \(\left\{ u_{n}\right\} \subset \mathbb X\) with \(\int _{\Omega }f(x)|u_{n}|^{1-\gamma }dx<+\infty \) and \(u_{n} \rightarrow u\) in \(G_{0}\). Our purpose is to prove that

$$\begin{aligned} a\Vert u\Vert ^{2}+b\Vert u\Vert ^{2m}+\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u|^{1-\gamma }dx\ge 0. \end{aligned}$$

In the beginning, we know

$$\begin{aligned} a\Vert u_{n}\Vert ^{2}+b\Vert u_{n}\Vert ^{2m}+\int _{\Omega } h(x)|u_{n}|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u_{n}|^{1-\gamma }dx\ge 0. \end{aligned}$$

Because \(|||u_{n}||-||u|||\le {||u_{n}-u||_{G_{0}}}\rightarrow 0\) as \(n\rightarrow \infty \), we can get

$$\begin{aligned} \lim _{n \rightarrow \infty }\Vert u_{n}\Vert ^{2}=\Vert u\Vert ^{2},\lim _{n \rightarrow \infty }\Vert u_{n}\Vert ^{2m}=\Vert u\Vert ^{2m}. \end{aligned}$$
(3.5)

According to Lemma 3.1, we get (3.1), owing to \(\mathcal {I}(|u|)=\mathcal {I}(u)\) and Fatou’s lemma, we have

$$\begin{aligned} \liminf _{n \rightarrow \infty }\int _{\Omega }f(x)|u_{n}|^{1-\gamma }dx \ge \int _{\Omega } \liminf _{n \rightarrow \infty }f(x)|u_{n}|^{1-\gamma }dx=\int _{\Omega }f(x)|u|^{1-\gamma }dx. \end{aligned}$$
(3.6)

Letting \(n \rightarrow \infty \), from the above mentioned, the following conclusion can be drawn:

$$\begin{aligned} a\Vert u\Vert ^{2}+b\Vert u\Vert ^{2m}+\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx \ge \int _{\Omega }f(x)|u|^{1-\gamma }dx. \end{aligned}$$

This proves the desired conclusion. \(\square \)

Lemma 3.3

Let \(u \in \mathbb X_{0}\) with \(u\ge 0\), \(\gamma >1\) and \(f \in L^{1}(\Omega )\). Then, there exist \(\epsilon >0\) and the continuous function \(\zeta :B_{\epsilon }(0) \rightarrow \mathbb R^{+}\) such that

$$\begin{aligned} \zeta (q)>0, \zeta (0)=1, \zeta (q)(u+q) \in \mathbb X_{0}\end{aligned}$$

for any \(q \in B_{\epsilon }(0)\), where \(B_{\epsilon }(0)=\left\{ q \in G_{0}:\Vert q\Vert < \epsilon \right\} \).

Proof

For all \(u \in \mathbb X_{0}\), define \(F:G_{0} \times \mathbb R^{+} \rightarrow \mathbb R\) as follows

$$\begin{aligned} F(q,r)= & {} r^{1+\gamma }a\Vert u+q\Vert ^{2}+r^{2m-1+\gamma }b\Vert u+q\Vert ^{2m}+r^{2_{s}^{*}-1+\gamma }\int _{\Omega }h(x)|u+q|^{2_{s}^{*}}dx\\{} & {} -\int _{\Omega }f(x)|u+q|^{1-\gamma }dx. \end{aligned}$$

Through simple calculation, we get

$$\begin{aligned} \frac{\partial F}{\partial r}= & {} a(1+\gamma )r^{\gamma }\Vert u+q\Vert ^{2}+b(2m-1+\gamma )r^{2m-2+\gamma }\Vert u+q\Vert ^{2m}\\{} & {} +(2_{s}^{*}-1+\gamma )r^{2_{s}^{*}-2+\gamma }\int _{\Omega }h(x)|u+q|^{2_{s}^{*}}dx. \end{aligned}$$

On account of \(u\in \mathbb X_{0}\), we can obtain

$$\begin{aligned} F(0,1)=a\Vert u\Vert ^{2}+b\Vert u\Vert ^{2m}-\int _{\Omega }f(x)|u|^{1-\gamma }dx+\int _{\Omega }h(x)|u|^{2_{s}^{*}}dx=0 \end{aligned}$$
(3.7)

and

$$\begin{aligned} \frac{\partial F}{\partial r}(0,1)=a(1+\gamma )\Vert u\Vert ^{2}+b(2m-1+\gamma )\Vert u\Vert ^{2m}+(2_{s}^{*}-1+\gamma )\int _{\Omega }h(x)|u|^{2_{s}^{*}}dx>0. \end{aligned}$$
(3.8)

By the implicit function theorem, we know there exists \(\epsilon >0\) such that for any \(q \in G_{0}\) with \(\Vert q\Vert <\epsilon \), the equation \(F(q,r)=0\) has a unique continuous solution \(r=\zeta (q)>0\). We can see that \(\zeta (0)=1\) from (3.7). So \(F(q,\zeta (q))=0\) for any \(q \in G_{0}\) with \(\Vert q\Vert <\epsilon \).

$$\begin{aligned} F(q,r)= & {} F(q,\zeta (q))\nonumber \\= & {} \zeta ^{1+\gamma }(q)a\Vert u+q\Vert ^{2}+\zeta ^{2m-1+\gamma }(q)b\Vert u+q\Vert ^{2m}\nonumber \\{} & {} +\zeta ^{2_{s}^{*}-1+\gamma }(q)\int _{\Omega }h(x)|u+q|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u+q|^{1-\gamma }dx\nonumber \\= & {} \big [a\Vert \zeta (q)(u+q)\Vert ^{2}+b\Vert \zeta (q)(u+q)\Vert ^{2m}\nonumber \\{} & {} +\int _{\Omega }h(x)|\zeta (q)(u+q)|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|\zeta (q)(u+q)|^{1-\gamma }dx\big ]/\zeta ^{1-\gamma }(q)\nonumber \\= & {} 0, \end{aligned}$$
(3.9)

that is \(\zeta (q)(u+q) \in \mathbb X_{0}\) for any \(q \in G_{0}\) with \(\Vert q\Vert <\epsilon .\) \(\square \)

4 A Compactness Result

Lemma 4.1

Suppose \(\gamma \in (1,+\infty )\) and (1.8) holds. There is \(u_{0} \in \mathbb X_{0}\) satisfying \(\mathcal {I}(u_{0})=c\). The bounded sequence \(\left\{ u_{n}\right\} \) possesses a subsequence strongly convergent to \(u_{0}\) in \(G_{0}\).

Proof

By Lemma 3.2, we have learned that \(\mathbb X\) is an unbounded closed set in \(G_{0}\). Since \(\gamma >1\), for any \(u\in \mathbb X\), we discover

$$\begin{aligned} \mathcal {I}(u)= & {} \frac{a}{2}||u||^{2}+\frac{b}{2m}||u||^{2m}+\frac{1}{2_{s}^{*}}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx-\frac{1}{1-\gamma }\int _{\Omega }f(x)|u|^{1-\gamma }dx\\\ge & {} \frac{a}{2}\Vert u\Vert ^{2}, \end{aligned}$$

which implies \(\mathcal {I}(u) \rightarrow \infty \) as \(\Vert u\Vert \rightarrow \infty \). So \(\mathcal {I}\) is coercive, and hence, the energy functional \(\mathcal {I}\) is bounded from below in \(\mathbb X\).

Moreover, for any \(\left\{ u_{n}\right\} \subset \mathbb X\) and \(u_{n}\rightarrow u\), in terms of (3.1), (3.5) and (3.6) again, we have

$$\begin{aligned} \liminf _{n \rightarrow \infty }\mathcal {I}(u_{n})= & {} \liminf _{n \rightarrow \infty }\left( \frac{a}{2}||u_{n}||^{2}+\frac{b}{2m}||u_{n}||^{2m}+\frac{1}{2_{s}^{*}}\int _{\Omega } h(x)|u_{n}|^{2_{s}^{*}}dx\right. \\{} & {} \left. -\frac{1}{1-\gamma }\int _{\Omega }f(x)|u_{n}|^{1-\gamma }dx\right) \\\ge & {} \frac{a}{2}||u||^{2}+\frac{b}{2m}||u||^{2m}+\frac{1}{2_{s}^{*}}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx\\{} & {} -\frac{1}{1-\gamma }\int _{\Omega }\liminf _{n \rightarrow \infty }f(x)|u_{n}|^{1-\gamma }dx\\\ge & {} \frac{a}{2}||u||^{2}+\frac{b}{2m}||u||^{2m}+\frac{1}{2_{s}^{*}}\int _{\Omega } h(x)|u|^{2_{s}^{*}}dx\\{} & {} -\frac{1}{1-\gamma }\int _{\Omega }f(x)|u|^{1-\gamma }dx \ge \mathcal {I}(u), \end{aligned}$$

To sum up, \(\mathbb X\) is an unbounded closed set, and \(\mathcal {I}\) is bounded from below and be lower semicontinuous functional in \(\mathbb X\). Hence, \(c=\inf _{u\in \mathbb X}\mathcal {I}(u)\) can be defined. Then, Ekeland’s variational principle (see Theorem 1 in [13]) can be applied to find the minimum point of energy functional. Consequently, there is a minimum sequence \(\left\{ u_{n}\right\} \subset \mathbb X\) meeting the following properties:

$$\begin{aligned} (i)\ \mathcal {I}(u_{n})<c+\frac{1}{n} , (ii)\ \mathcal {I}(u_{n})\le \mathcal {I}(u)+\frac{1}{n}\Vert u_{n}-u\Vert ,\ \forall u \in \mathbb X. \end{aligned}$$
(4.1)

We assume

$$\begin{aligned} \int _{\Omega }f(x)|u_{n}|^{1-\gamma }dx<+\infty . \end{aligned}$$

On account of \(\mathcal {I}(u)=\mathcal {I}(|u|)\), we presume \(u_{n}(x)\ge 0\) a.e. in \(\Omega \). Evidently, \(\left\{ u_{n}\right\} \) is bounded in \(G_{0}\). Otherwise, let us assume \(\left\{ u_{n}\right\} \) is unbounded, we can choose a subsequence \({u_{i}}\) such that \(u_{i} \rightarrow \infty \) as \(i \rightarrow \infty \), but \(\mathcal {I}(u_{i})\) satisfies (i) in (4.1), and it is contradictory. Because \(G_{0}\) is reflexive, the sequence \(\left\{ u_{n}\right\} \) possesses a subsequence weakly convergent to \(u_{0}\) in \(G_{0}\). As to subsequence, we denote it by \(\left\{ u_{k}\right\} \), such that \(u_{k} \rightharpoonup u_{0}\) in \(G_{0}\). For \(1\le r<2_{s}^{*}\), the embedding \(G_{0}\hookrightarrow L^{r}(\Omega )\) is compact and continuous and when \(r=2_{s}^{*}\), the embedding is continuous. In summary, by using Lemma 8 in [33] and Theorem 4.9 in [5], there exists \(u_{0}\in G_{0}\) with \(u_{0}\ge 0\) such that as \(n\rightarrow \infty \),

$$\begin{aligned} \begin{array}{cc} u_k\rightharpoonup u_0 \text { in}\ G_0,\\ u_k\rightharpoonup u_0 \text { in}\ L^{2_{s}^{*}},\\ u_k \rightarrow u_0 \text { a.e. in}\ \ \Omega , \\ u_k \rightarrow u_0 \text { in}\ L^r(\Omega )\ \text {for}\ 2 \le r <2^*_s . \end{array} \end{aligned}$$
(4.2)

If \(u_{0}=0\), \(\int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx<\infty \) fails, thus \(u_{0}>0\) a.e. in \(\Omega \). Since \(u_{k}\rightharpoonup u_{0}\) as \(k\rightarrow \infty \) in \(G_{0}\). According to the weakly lower semicontinuity of the norm, we can conclude that

$$\begin{aligned} \liminf _{k\rightarrow \infty }\Vert u_{k}\Vert ^{2}\ge \Vert u_{0}\Vert ^{2}. \end{aligned}$$
(4.3)

Through simple deformation, we can also get

$$\begin{aligned} \liminf _{k\rightarrow \infty }\Vert u_{k}\Vert ^{2m}\ge \Vert u_{0}\Vert ^{2m}. \end{aligned}$$
(4.4)

Moreover, similar to (3.6), we have

$$\begin{aligned} \begin{aligned} \liminf _{n\rightarrow \infty }\left( -\frac{1}{1-\gamma }\int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx\right) \ge -\frac{1}{1-\gamma }\int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx. \end{aligned} \end{aligned}$$
(4.5)

Now, we start to use (i) in Lemma 3.2, and there exists a unique positive constant \(t_{\min }(u_{0})\) such that \(I(t_{\min }(u_{0})u_{0})=\min _{t>0}I(tu_{0})\) and \(t_{\min }(u_{0})u_{0}\in \mathbb X_{0}.\) Since \(\left\{ u_{k}\right\} \) is bounded and \(u_{k} \rightarrow u_{0}\) a.e. in \(\Omega \), according to (3.1), (4.2), (4.3), (4.4), (4.5) and Brézis–Lieb lemma, we have

$$\begin{aligned} c= & {} \liminf _{n \rightarrow \infty }\mathcal {I}(u_{k})\nonumber \\= & {} \liminf _{n\rightarrow \infty } \left[ \frac{a\Vert u_k\Vert ^2}{2}+\frac{b\Vert u_k\Vert ^{2m}}{2m} -\frac{1}{1-\gamma } \int _{\Omega }f(x)|u_k|^{1-\gamma } \textrm{d}x\right] \nonumber \\{} & {} +\frac{1}{2_{s}^{*}} \int _{\Omega }h(x)|u_0|^{2_{s}^{*}} \textrm{d}x+\lim _{n\rightarrow \infty }\frac{1}{2_{s}^{*}}\int _{\Omega }h(x)\mid u_{k}-u_{0}\mid ^{2_{s}^{*}}\textrm{d}x \nonumber \\\ge & {} a\frac{\Vert u_0\Vert ^2}{2} +\frac{b\Vert u_0\Vert ^{2m}}{2m} -\frac{1}{1-\gamma } \int _{\Omega }f(x)|u_0|^{1-\gamma } \textrm{d}x +\frac{1}{2_{s}^{*}} \int _{\Omega }h(x)|u_0|^{2_{s}^{*}} \textrm{d}x \nonumber \\= & {} \mathcal {I}(u_0)\ge \mathcal {I}(t_{\min }(u_0)u_0)\ge \inf _{u\in \mathbb X_{0}}\mathcal {I}(u)\ge c, \end{aligned}$$
(4.6)

which implies that

$$\begin{aligned} \mathcal {I}(u_{0})=c, u_{0}\in \mathbb X_{0}. \end{aligned}$$

That is, \(t_{\min }(u_{0})=1 \). From (4.6), we can acquire

$$\begin{aligned}{} & {} \liminf _{n \rightarrow \infty }\left( \frac{a}{2}||u_{k}||^{2}+\frac{b}{2m}||u_{k}||^{2m}+\frac{1}{2_{s}^{*}}\int _{\Omega } h(x)|u_{k}|^{2_{s}^{*}}dx -\frac{1}{1-\gamma }\int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx\right) \\{} & {} \quad =a\frac{\Vert u_0\Vert ^2}{2} +\frac{b\Vert u_0\Vert ^{2m}}{2m} -\frac{1}{1-\gamma } \int _{\Omega }f(x)|u_0|^{1-\gamma } \textrm{d}x +\frac{1}{2_{s}^{*}} \int _{\Omega }h(x)|u_0|^{2_{s}^{*}} \textrm{d}x. \end{aligned}$$

If \(a>0\), \(b\ge 0\), then we have \(\Vert u_{k}\Vert ^{2}\rightarrow \Vert u_{0}\Vert ^{2}\) as \(k\rightarrow \infty \). If \(a=0\), \(b>0\), then we have \(\Vert u_{k}\Vert ^{2m}\rightarrow \Vert u_{0}\Vert ^{2m}\), which immediately implies that \(\Vert u_{k}\Vert ^{2}\rightarrow \Vert u_{0}\Vert ^{2}\). Hence for any \( \epsilon >0\), there exists k large enough, and it holds

$$|\Vert u_{k}\Vert ^{2}-\Vert u_{0}\Vert ^{2}|=|\big (\Vert u_{k}\Vert +\Vert u_{0}\Vert \big )\big (\Vert u_{k}\Vert -\Vert u_{0}\Vert \big )|<\epsilon ,$$

Since \(\left\{ u_{n}\right\} \) is bounded in \(G_{0}\), we get that \(\lim _{k\rightarrow \infty }\Vert u_{k}\Vert =\Vert u_{0}\Vert \). According to \(G_{0}\) which is a Hilbert space, \(u_{k}\rightharpoonup u_{0}\) and \(\lim _{k\rightarrow \infty }\Vert u_{k}\Vert =\Vert u_{0}\Vert \). Then, we can have \(u_{k} \rightarrow u_{0}\). \(\square \)

5 Proof of Theorem 1.1

In this section, we begin to certify Theorem 1.1. More precisely, we will prove the necessity and sufficiency of the theorem, respectively.

First we prove the necessity: For \(\gamma >1\) and \(f\in L^{1}(\Omega )\) which is positive a.e. in \(\Omega \), there is a unique positive solution \(u_{0} \in G_{0}\) such that

$$\begin{aligned} \int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx<+\infty . \end{aligned}$$

Since \(u_{0}\) is a unique positive solution, we know

$$\begin{aligned}{} & {} \int _{\Omega }f(x)|u_{0}|^{-\gamma } \psi dx=a\langle u_{0},\psi \rangle +b\Vert u_{0}\Vert ^{2m-2}\langle u_{0},\psi \rangle \\{} & {} \quad +\int _{\Omega } h(x)|u_{0}|^{2_{s}^{*}-1}\psi dx,\forall \psi \in G_{0}. \end{aligned}$$

Obviously, we just consider that \(\int _{\Omega } h(x)|u_{0}|^{2_{s}^{*}-1}\psi dx\) is well defined. This point can be observed from the following estimates:

$$\begin{aligned} \begin{aligned} \int _{\Omega } h(x)|u_{0}|^{2_{s}^{*}-1}\psi dx \le C_{1}\int _{\Omega } |u_{0}|^{2_{s}^{*}-1}\psi dx \le C_{1}\Vert \psi \Vert _{L^{2}(\Omega )}\Vert u_{0}\Vert _{L^{2}(\Omega )}^{2_{s}^{*}-1}|\Omega |^{\frac{2-2_{s}^{*}}{2}}. \end{aligned} \end{aligned}$$

This finishes the proof of the necessity.

Then, we prove the sufficiency: Suppose that \(\gamma >1\) and \(f \in L^{1}(\Omega )\) is positive a.e. in \(\Omega \). If there is \(u_{0}\in G_{0}\) such that

$$\begin{aligned} \int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx<+\infty \end{aligned}$$

holds, then \(u_{0}\) is the unique positive solution of problem (1.1).

We divide this proof into the following two parts:

\(\mathrm {\mathbf {Part\ 1.} }\) Suppose that the subsequence \(\left\{ u_{k}\right\} \subset \mathbb X\) in Lemma 4.1 satisfies \(\left\{ u_{k}\right\} \subset \mathbb X\setminus \mathbb X_{0}\).

On account of \(\left\{ u_{k}\right\} \subset \mathbb X\setminus \mathbb X_{0}\), we have

$$\begin{aligned} a\Vert u_{k}\Vert ^{2}+b\Vert u_{k}\Vert ^{2m}+\int _{\Omega }h(x)|u_{x}|^{2_{s}^{*}}dx>\int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx. \end{aligned}$$

For any \( \varphi \in G_{0}\) with \(\varphi \ge 0\), we choose \(\tau >0\) small enough. Since \(\gamma >1\), we have

$$\begin{aligned} \int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx\ge \int _{\Omega }f(x)|u_{k}+\tau \varphi |^{1-\gamma }dx. \end{aligned}$$

By the continuity, we have

$$\begin{aligned} \begin{aligned}&\lim _{\tau \rightarrow 0}a\Vert u_{k}+\tau \varphi \Vert ^{2}+b\Vert u_{k}+\tau \varphi \Vert ^{2m}+\int _{\Omega }h(x)|u_{x}+\tau \varphi |^{2_{s}^{*}}dx\\&\quad =a\Vert u_{k}\Vert ^{2}+b\Vert u_{k}\Vert ^{2m}+\int _{\Omega }h(x)|u_{x}|^{2_{s}^{*}}dx\\&\quad >\int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx\ge \int _{\Omega }f(x)|u_{k}+\tau \varphi |^{1-\gamma }dx, \end{aligned} \end{aligned}$$

which implies that \(u_{k}+\tau \varphi \in \mathbb X\) for \(\tau >0\) small enough. Now, we use (ii) in (4.1). Then, we get

$$\begin{aligned} \frac{||u_{k}-(u_{k}+\tau \varphi )||}{k}\ge \mathcal {I}(u_{k})-\mathcal {I}(u_{k}+\tau \varphi ), \end{aligned}$$

that is

$$\begin{aligned} \frac{\tau ||\varphi ||}{k}\ge & {} a\frac{||u_{k}||^{2}-||u_{k}+\tau \varphi ||^{2}}{2}+b\frac{||u_{k}||^{2m}-||u_{k}+\tau \varphi ||^{2m}}{2m}\nonumber \\{} & {} +\frac{\int _{\Omega }\left[ h(x)|u_{k}|^{2_{s}^{*}}-h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}dx\right] }{2_{s}^{*}}\nonumber \\{} & {} -\frac{\int _{\Omega }\left[ f(x)|u_{k}|^{1-\gamma }-f(x)|u_{k}+\tau \varphi |^{1-\gamma }\right] dx}{1-\gamma }. \end{aligned}$$
(5.1)

Next, we divide (5.1) by \(\tau >0\) and take the inferior limit \(\tau \rightarrow 0\). Along this direction, we first decompose the equation (5.1). Notice that

$$\begin{aligned} \begin{aligned} \frac{a}{2\tau }\big (||u_{k}||^{2}-||u_{k}+\tau \varphi ||^{2}\big )&=\frac{a}{2\tau }\big (\langle u_{k},u_{k}\rangle -\langle u_{k}+\tau \varphi ,u_{k}+\tau \varphi \rangle \big )\\&=\frac{a}{2\tau }\big [\langle u_{k},u_{k}\rangle -\left( \langle u_{k},u_{k}\rangle +2\tau \langle u_{k},\varphi \rangle +\tau ^{2}\langle \varphi ,\varphi \rangle \right) \big ]\\&=-a\langle u_{k},\varphi \rangle -\frac{a}{2}\tau ||\varphi ||^{2}. \end{aligned}\end{aligned}$$
(5.2)

Similarly,

$$\begin{aligned}{} & {} \frac{b}{2m\tau }\big (\Vert u_{k}\Vert ^{2m}-\Vert u_{k}+\tau \varphi \Vert ^{2m}\big ) \nonumber \\{} & {} \quad =\frac{b}{2m\tau }\big (\langle u_{k},u_{k}\rangle ^{m}-\langle u_{k}+\tau \varphi ,u_{k}+\tau \varphi \rangle ^{m}\big )\nonumber \\{} & {} \quad =\frac{b}{2m\tau }\big (\langle u_{k},u_{k}\rangle ^{m}-\big (\langle u_{k},u_{k}\rangle +2\tau \langle u_{k},\varphi \rangle +\tau ^{2}\langle \varphi , \varphi \rangle \big )^{m}\big ). \end{aligned}$$
(5.3)

Regarding the third item, we need to prove

$$\begin{aligned} \liminf _{\tau \rightarrow 0}\int _{\Omega }\frac{h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}-h(x)|u_{k}|^{2_{s}^{*}}dx}{2_{s}^{*}\tau } =\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx. \end{aligned}$$
(5.4)

In fact, according to the mean value theorem, there exists \(0<\theta <1\), and we get

$$\begin{aligned} \begin{aligned} \frac{h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}-h(x)|u_{k}|^{2_{s}^{*}}dx}{2_{s}^{*}\tau } =h(x)|u_{k}+\theta \tau \varphi |^{2_{s}^{*}-1}\varphi \le h(x)\big (|u_{k}|^{2_{s}^{*}-1}\varphi +|\varphi |^{2_{s}^{*}}\big ). \end{aligned} \end{aligned}$$

Notice that from the Hölder inequality we can deduce that

$$\begin{aligned}&\int _{\Omega } h(x)\left( |u_{k}|^{2_{s}^{*}-1}\varphi +|\varphi |^{2_{s}^{*}}\right) dx\\&\le C_{1}\left( \int _{\Omega }\varphi |u_{k}|^{2_{s}^{*}-1}dx+\int _{\Omega }\varphi ^{2_{s}^{*}}dx\right) \\&\le C_{1}\left[ \left( \int _{\Omega }\varphi ^{2}dx\right) ^{\frac{1}{2}} \left( \int _{\Omega }|u_{k}|^{2(2_{s}^{*}-1)}dx\right) ^{\frac{1}{2}}+\left( \int _{\Omega }\varphi ^{2}dx\right) ^{\frac{1}{2}}\left( \int _{\Omega }\varphi ^{2(2_{s}^{*}-1)}dx\right) ^{\frac{1}{2}}\right] \\&\le C_{1}\Vert \varphi \Vert _{L^{2}(\Omega )}\left[ \left( \int _{\Omega }|u_{k}|^{\frac{2(2_{s}^{*}-1)}{2_{s}^{*}-1}}dx\right) ^{2_{s}^{*}-1} \left( \int _{\Omega }1^{\frac{1}{2-2_{s}^{*}}}dx\right) ^{2-2_{s}^{*}}\right] ^{\frac{1}{2}}\\&\quad +C_{1}\Vert \varphi \Vert _{L^{2}(\Omega )}\left[ \left( \int _{\Omega }\varphi ^{\frac{2(2_{s}^{*}-1)}{2_{s}^{*}-1}}dx\right) ^{2_{s}^{*}-1} \left( \int _{\Omega }1^{\frac{1}{2-2_{s}^{*}}}dx\right) ^{2-2_{s}^{*}}\right] ^{\frac{1}{2}}\\&\le C_{1}\Vert \varphi \Vert _{L^{2}(\Omega )}|\Omega |^{\frac{2-2_{s}^{*}}{2}}\left( \Vert u_{k}\Vert _{L^{2}(\Omega )}^{2_{s}^{*}-1}+\Vert \varphi \Vert _{L^{2}(\Omega )}^{2_{s}^{*}-1}\right) \\&< \infty , \end{aligned}$$

which yields that \(h(x)\big (|u_{k}|^{2_{s}^{*}-1}\varphi +|\varphi |^{2_{s}^{*}}\big ) \in L^{1}(\Omega )\) is a nonnegative measurable function. Consequently, we can prove (5.4) on the grounds of the dominated convergence theorem.

Next, together with (5.1), (5.2), (5.3) and (5.4), we use Fatou’s lemma to deduce that

$$\begin{aligned} \begin{aligned}&\liminf _{\tau \rightarrow 0}\bigg [\frac{\tau ||\varphi ||}{\tau k} +\frac{a}{2\tau }\big (||u_{k}+\tau \varphi ||^{2}-||u_{k}||^{2}\big )+\frac{b}{2m\tau }\big (\Vert u_{k}+\tau \varphi \Vert ^{2m}-\Vert u_{k}\Vert ^{2m}\big )\\&\quad +\int _{\Omega }\frac{h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}-h(x)|u_{k}|^{2_{s}^{*}}dx}{2_{s}^{*}\tau }\bigg ]\\&\ge \liminf _{\tau \rightarrow 0}\frac{\int _{\Omega }[f(x)|u_{k}+\tau \varphi |^{1-\gamma }-f(x)|u_{k}|^{1-\gamma }]dx}{\tau (1-\gamma )}\\&\ge \int _{\Omega }f(x)|u_{k}|^{-\gamma }\varphi dx. \end{aligned} \end{aligned}$$

Therefore,

$$\begin{aligned} \begin{aligned} \frac{\Vert \varphi \Vert }{k}+a\langle u_{k},\varphi \rangle +b\langle u_{k},u_{k}\rangle ^{m-1}\langle u_{k},\varphi \rangle +\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx \ge \int _{\Omega }f(x)|u_{k}|^{-\gamma }\varphi dx. \end{aligned} \end{aligned}$$

Based on these facts, letting \(k \rightarrow \infty \), by Lemma 4.1 and Fatou’s lemma, we have

$$\begin{aligned}{} & {} a\langle u_{0},\varphi \rangle +b\Vert u_{0}\Vert ^{2m-2}\langle u_{0},\varphi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\varphi dx\nonumber \\{} & {} \quad \ge \int _{\Omega }f(x)|u_{0}|^{-\gamma }\varphi dx, \forall \varphi \in G_{0},\varphi \ge 0. \end{aligned}$$
(5.5)

\(\mathrm {\mathbf {Part\ 2.}}\) Suppose the subsequence \(\left\{ u_{k}\right\} \subset \mathbb X\) in Lemma 4.1 belongs to \(\mathbb X_{0}.\) By Lemma 3.3, we get a series of functions \(\zeta _{k}\) such that \(\zeta _{k}(0)=1\) and \(\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )\in \mathbb X_{0}\) for \(\tau >0\) sufficiently small and \(\varphi \in G_{0}\) with \(\varphi \ge 0.\) Since \(u_{k} \in \mathbb X_{0}\) and \(\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )\in \mathbb X_{0}\), we can deduce that

$$\begin{aligned} a\Vert u_{k}\Vert ^{2}+b\Vert u_{k}\Vert ^{2m}+\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx=0 \end{aligned}$$
(5.6)

and

$$\begin{aligned}{} & {} a\zeta _{k}^{2}(\tau \varphi )\Vert u_{k}+\tau \varphi \Vert ^{2}+b\zeta _{k}^{2m}(\tau \varphi )\Vert u_{k}+\tau \varphi \Vert ^{2m}+\zeta _{k}^{2_{s}^{*}}(\tau \varphi )\int _{\Omega }h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}dx\nonumber \\{} & {} \quad -\zeta _{k}^{1-\gamma }(\tau \varphi )\int _{\Omega }f(x)|u_{k}+\tau \varphi |^{1-\gamma }dx=0. \end{aligned}$$
(5.7)

We set \(\zeta _{k}^{'}(0)\) as the derivative of \(\zeta _{k}\) at zero with \((\zeta _{k}^{'},\varphi ):=\lim _{\tau \rightarrow 0}\frac{\zeta _{k}(\tau \varphi )-1}{\tau } \in [-\infty ,+\infty ]\) for any \(\varphi \in G_{0}\). If the limit does not exist, we choose another positive sequence \(\tau _{n} \rightarrow 0\) as \(n\rightarrow \infty \) (instead of \(\tau \rightarrow 0\)) and \(t_{n}=\lim _{n\rightarrow \infty }\frac{\zeta _{k}(\tau _{n}\varphi )-1}{\tau _{n}}\) exists, and then, we replace \(\zeta _{k}^{'}(0)\) by \(t_{n}(0)\).

Now, we claim that if \(\left\{ u_{k}\right\} \subset \mathbb X_{0}\) satisfies (5.6) and (5.7), then \((\zeta _{k}^{'},\varphi )\) is uniformly bounded for any \(\varphi \in G_{0}\) with \(\varphi \ge 0\).

We consider that \(\left\{ u_{k}\right\} \subset \mathbb X_{0}\), \(\zeta (\tau \varphi )(u_{k}+\tau \varphi )\in \mathbb X_{0}\). By (5.6) and (5.7), we get

$$\begin{aligned}{} & {} a[(\zeta _{k}^{2}(\tau \varphi )-1)\Vert u_{k}+\tau \varphi \Vert ^{2}+\Vert u_{k}+\tau \varphi \Vert ^{2}-\Vert u_{k}\Vert ^{2}]\\{} & {} \quad +b[(\zeta _{k}^{2m}(\tau \varphi )-1)\Vert u_{k}+\tau \varphi \Vert ^{2m}+\Vert u_{k}+\tau \varphi \Vert ^{2m}-\Vert u_{k}\Vert ^{2m}]\\{} & {} \quad +[\zeta _{k}^{2_{s}^{*}}(\tau \varphi )-1]\int _{\Omega }h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}dx+\int _{\Omega }h(x)[(u_{k}+\tau \varphi )^{2_{s}^{*}}-u_{k}^{2_{s}^{*}}]dx\\{} & {} \quad -[(\zeta _{k}^{1-\gamma }(\tau \varphi )-1)]\int _{\Omega }f(x)|u_{k}+\tau \varphi |^{1-\gamma }\\{} & {} \quad -\int _{\Omega }[f(x)|u_{k}+\tau \varphi |^{1-\gamma }-f(x)|u_{k}|^{1-\gamma }]dx=0. \end{aligned}$$

After that, dividing the above equation by \(\tau >0\), because \(\gamma >1\), we obtain

$$\begin{aligned}{} & {} \frac{\zeta _{k}(\tau \varphi )-1}{\tau }\big [a(\zeta _{k}(\tau \varphi )+1)\Vert u_{k}+\tau \varphi \Vert ^{2}+b\frac{\zeta _{k}^{2m}(\tau \varphi )-1}{\zeta _{k}(\tau \varphi )-1}\Vert u_{k}+\tau \varphi \Vert ^{2m}\\{} & {} +\frac{\zeta _{k}^{2_{s}^{*}}(\tau \varphi )-1}{\zeta _{k}(\tau \varphi )-1}\int _{\Omega }h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}dx -\frac{\zeta _{k}^{1-\gamma }(\tau \varphi )-1}{\zeta _{k}(\tau \varphi )-1}\int _{\Omega }f(x)|u_{k}+\tau \varphi |^{1-\gamma }dx\big ]\\{} & {} +a\frac{\Vert u_{k}+\tau \varphi \Vert ^{2}-\Vert u_{k}\Vert ^{2}}{\tau }+b\frac{\Vert u_{k}+\tau \varphi \Vert ^{2m}-\Vert u_{k}\Vert ^{2m}}{\tau }\\{} & {} +\int _{\Omega }\frac{h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}-h(x)|u_{k}|^{2_{s}^{*}}}{\tau }dx\le 0. \end{aligned}$$

Letting \(\tau \rightarrow 0\), by (5.12), (5.3) and (5.4) again, we infer that

$$\begin{aligned} \begin{aligned}&(\zeta _{k}^{'},\varphi )\Big [2a\Vert u_{k}\Vert ^{2}+2mb\Vert u_{k}\Vert ^{2m}+2_{s}^{*}\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx-(1-\gamma )\int _{\Omega }f(x)|u_{k}|^{1-\gamma }\Big ]\\&+2a\langle u_{k},\varphi \rangle +2mb\Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle +2_{s}^{*}\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx\le 0. \end{aligned} \end{aligned}$$
(5.8)

Then, using (5.6) and (5.8), we have

$$\begin{aligned} \begin{aligned}&(\zeta _{k}^{'},\varphi )\Big [a(1+\gamma )\Vert u_{k}\Vert ^{2}+(2m-1+\gamma )b\Vert u_{k}\Vert ^{2m}+(2_{s}^{*}-1+\gamma )\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx\Big ]\\&+2a\langle u_{k},\varphi \rangle +2mb\Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle +2_{s}^{*}\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx\le 0, \end{aligned} \end{aligned}$$
(5.9)

that is

$$\begin{aligned} (\zeta _{k}^{'},\varphi )\le \frac{-(2a\langle u_{k},\varphi \rangle +2mb\Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle +2_{s}^{*}\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx)}{a(1+\gamma )\Vert u_{k}\Vert ^{2}+(2m-1+\gamma )b\Vert u_{k}\Vert ^{2m}+(2_{s}^{*}-1+\gamma )\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx}. \end{aligned}$$

Since \(\left\{ u_{k}\right\} \) is bounded in \(G_{0}\), above inequality implies that \((\xi _{k}^{'},\varphi )\) is bounded from above uniformly, that is \((\zeta _{k}^{'},\varphi )\ne +\infty \).

Besides, by (4.1)-(ii), we get

$$\begin{aligned} \begin{aligned} \frac{\Vert u_{k}-\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )\Vert }{k}\ge \mathcal {I}(u_{k})-\mathcal {I}[\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )] \end{aligned} \end{aligned}$$
(5.10)

and

$$\begin{aligned} \begin{aligned} \frac{\Vert u_{k}-\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )\Vert }{k}&=\frac{\Vert (1-\zeta _{k}(\tau \varphi ))u_{k}-\zeta _{k}(\tau \varphi )\tau \varphi \Vert }{k}\\&\le \frac{\Vert (1-\zeta _{k}(\tau \varphi ))u_{k}\Vert }{k}+\frac{\Vert -\zeta _{k}(\tau \varphi )\tau \varphi \Vert }{k}\\&\le |\zeta _{k}(\tau \varphi )-1|\frac{\Vert u_{k}\Vert }{k}+\tau \zeta _{k}(\tau \varphi )\frac{\Vert \varphi \Vert }{k}, \end{aligned} \end{aligned}$$
(5.11)

which implies

$$\begin{aligned}{} & {} \Big |\zeta _{k}(\tau \varphi )-1\Big |\frac{\Vert u_{k}\Vert }{k}+\tau \zeta _{k}(\tau \varphi )\frac{\Vert \varphi \Vert }{k}\ge \mathcal {I}(u_{k})-\mathcal {I}[\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )]\nonumber \\{} & {} =\frac{a(1+\gamma )}{2(1-\gamma )}\Big [(\zeta _{k}^{2}(\tau \varphi )-1)\Vert u_{k}+\tau \varphi \Vert ^{2}+(\Vert u_{k}+\tau \varphi \Vert ^{2}-\Vert u_{k}\Vert ^{2})\Big ]\nonumber \\{} & {} \quad +\frac{b(2m-1+\gamma )}{2m(1-\gamma )}\Big [(\zeta _{k}^{2m}(\tau \varphi )-1)\Vert u_{k}+\tau \varphi \Vert ^{2m}+(\Vert u_{k}+\tau \varphi \Vert ^{2m}-\Vert u_{k}\Vert ^{2m})\Big ]\nonumber \\{} & {} \quad +\frac{2_{s}^{*}-1+\gamma }{2_{s}^{*}(1-\gamma )}\Big [(\zeta _{k}^{2_{s}^{*}}(\tau \varphi )-1)\int _{\Omega }h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}} +\int _{\Omega }h(x)|u_{k}\nonumber \\{} & {} \quad +\tau \varphi |^{2_{s}^{*}}-h(x)|u_{k}|^{2_{s}^{*}}dx\Big ]. \end{aligned}$$
(5.12)

Dividing (5.12) by \(\tau >0\), and next letting \(\tau \rightarrow 0\), we deduce that

$$\begin{aligned}{} & {} \Big |(\zeta _{k}^{'},\varphi )\Big |\frac{\Vert u_{k}\Vert }{k}+\lim _{\tau \rightarrow 0}\zeta _{k}(\tau \varphi )\frac{\Vert \varphi \Vert }{k}\\{} & {} \ge a\frac{1+\gamma }{1-\gamma }\left[ (\zeta _{k}^{'},\varphi )\Vert u_{k}\Vert ^{2}+\langle u_{k},\varphi \rangle \right] \\{} & {} \quad +b\frac{2m-1+\gamma }{1-\gamma }\left[ (\zeta _{k}^{'},\varphi )\Vert u_{k}\Vert ^{2m}+\Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle \right] \\{} & {} \quad +\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\left[ (\zeta _{k}^{'},\varphi )\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx+\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx\right] . \end{aligned}$$

That is

$$\begin{aligned} \Big |(\zeta _{k}^{'},\varphi )\Big |\frac{\Vert u_{k}\Vert }{k}+\frac{\Vert \varphi \Vert }{k}\ge & {} (\zeta _{k}^{'},\varphi )\left[ a\frac{1+\gamma }{1-\gamma }\Vert u_{k}\Vert ^{2}+b\frac{2m-1+\gamma }{1-\gamma } \Vert u_{k}\Vert ^{2m}\right. \\{} & {} \left. +\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx\right] \\{} & {} +a\frac{1+\gamma }{1-\gamma }\langle u_{k},\varphi \rangle +b\frac{2m-1+\gamma }{1-\gamma } \Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle \\{} & {} +\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx, \end{aligned}$$

which implies that \((\zeta _{k}^{'},\varphi )\ne -\infty \) thanks to \(\gamma >1\). Hence, \((\zeta _{k}^{'},\varphi )\) is bounded from below uniformly for all large k. We can use the method of disproportion to prove this. If \((\zeta _{k}^{'},\varphi )=-\infty \),

$$\begin{aligned}- & {} (\zeta _{k}^{'},\varphi )\left[ \frac{\Vert u_{k}\Vert }{k}+a\frac{1+\gamma }{1-\gamma }\Vert u_{k}\Vert ^{2}+b\frac{2m-1+\gamma }{1-\gamma } \Vert u_{k}\Vert ^{2m}\right. \nonumber \\{} & {} \left. +\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx\right] \nonumber \\{} & {} \ge a\frac{1+\gamma }{1-\gamma }\langle u_{k},\varphi \rangle +b\frac{2m-1+\gamma }{1-\gamma } \Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle \nonumber \\{} & {} \quad +\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx-\frac{\Vert \varphi \Vert }{k}. \end{aligned}$$
(5.13)

Then, let \(k \rightarrow \infty \), we have

$$\begin{aligned} (\zeta _{k}^{'},\varphi )\ge \frac{a\frac{1+\gamma }{1-\gamma }\langle u_{0},\varphi \rangle +b\frac{2m-1+\gamma }{1-\gamma } \Vert u_{0}\Vert ^{2m-2}\langle u_{0},\varphi \rangle +\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\varphi dx}{-[a\frac{1+\gamma }{1-\gamma }\Vert u_{0}\Vert ^{2}+b\frac{2m-1+\gamma }{1-\gamma } \Vert u_{0}\Vert ^{2m}+\frac{2_{s}^{*}-1+\gamma }{1-\gamma }\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}}dx]}. \end{aligned}$$

This is contradictory. To sum up, there is a constant \(C_{3}>0\) such that \(|(\zeta _{k}^{'},\varphi )|\le C_{3}\). This ends the claim.

Now, we use (5.10) and (5.11) again and divide by \(\tau >0\).

$$\begin{aligned}{} & {} \Big |\frac{\zeta _{k}(\tau \varphi )-1}{\tau }\Big |\frac{\Vert u_{k}\Vert }{k}+\zeta _{k}(\tau \varphi )\frac{\Vert \ \varphi \Vert }{k} \ge \frac{\Vert u_{k}-\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )\Vert }{{k}\tau }\nonumber \\{} & {} \quad \ge \frac{\mathcal {I}(u_{k})-\mathcal {I}[\zeta _{k}(\tau \varphi )(u_{k}+\tau \varphi )]}{\tau }\nonumber \\{} & {} \quad =-\frac{\zeta _{k}(\tau \varphi )-1}{\tau }\Big [\frac{a(\zeta _{k}(\tau \varphi )+1)}{2}\Vert u_{k}+\tau \varphi \Vert ^{2}+\frac{b(\zeta _{k}^{2m}(\tau \varphi )-1)}{2m(\zeta _{k}(\tau \varphi )-1)} \Vert u_{k}+\tau \varphi \Vert ^{2m}\nonumber \\{} & {} \qquad +\frac{\zeta _{k}^{2_{s}^{*}}(\tau \varphi )-1}{2_{s}^{*}(\zeta _{k}(\tau \varphi )-1)}\int _{\Omega }h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}dx\nonumber \\{} & {} \qquad -\frac{\zeta _{k}^{1-\gamma }(\tau \varphi )-1}{(1-\gamma )(\zeta _{k}(\tau \varphi )-1)}\int _{\Omega }f(x)|u_{k}+\tau \varphi |^{1-\gamma }dx\Big ]\nonumber \\{} & {} \qquad -\Big [\frac{a(\Vert u_{k}+\tau \varphi \Vert ^{2}-\Vert u_{k}\Vert ^{2})}{2\tau }+\frac{b(\Vert u_{k}+\tau \varphi \Vert ^{2m}-\Vert u_{k}\Vert ^{2m})}{2m\tau }\nonumber \\{} & {} \qquad +\int _{\Omega }\frac{h(x)|u_{k}+\tau \varphi |^{2_{s}^{*}}-h(x)|u_{k}|^{2_{s}^{*}}}{2_{s}^{*}\tau }\nonumber \\{} & {} \quad -\int _{\Omega }\frac{f(x)|u_{k}+\tau \varphi |^{1-\gamma }-f(x)|u_{k}|^{1-\gamma }}{(1-\gamma )\tau }dx\Big ]. \end{aligned}$$
(5.14)

Letting \(\tau \rightarrow 0\), due to \(\gamma >1\), Fatou’s lemma can be used. From the above inequality, we can get

$$\begin{aligned}{} & {} \big |(\zeta _{k}^{'},\varphi )\big |\frac{\Vert u_{k}\Vert }{k}+\lim _{\tau \rightarrow 0}\zeta _{k}(\tau \varphi )\frac{\Vert \varphi \Vert }{k}=|( \zeta _{k}^{'},\varphi )|\frac{\Vert u_{k}\Vert }{k}+\frac{\Vert \varphi \Vert }{k}\\{} & {} \quad \ge -(\zeta _{k},\varphi )\Big [a\Vert u_{k}\Vert ^{2}+b\Vert u_{k}\Vert ^{2m}+\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u_{k}|^{1-\gamma }dx\Big ]\\{} & {} \qquad -\Big [a\langle u_{k},\varphi \rangle +b\Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle +\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx\\{} & {} \qquad -\int _{\Omega }\liminf _{\tau \rightarrow 0}\frac{f(x)|u_{k}+\tau \varphi |^{1-\gamma }-f(x)|u_{k}|^{1-\gamma }}{(1-\gamma )\tau }dx\Big ]\\{} & {} \quad =-\Big [a\langle u_{k},\varphi \rangle +b\Vert u_{k}\Vert ^{2m-2}\langle u_{k},\varphi \rangle +\int _{\Omega }h(x)|u_{k}|^{2_{s}^{*}-1}\varphi dx-\int _{\Omega }f(x)|u_{k}|^{-\gamma }\varphi dx\Big ], \end{aligned}$$

owing to \(u_{k}\in \mathbb X_{0}\). Since \(|(\zeta _{k}^{'},\varphi )|\le C_{3}\) uniformly for large k, Fatou’s lemma yields that \(f(x)u_{k}^{-\gamma }\varphi \) is integrable. Letting \(k\rightarrow \infty \), we can get the same result as Part 1:

$$\begin{aligned}{} & {} a\langle u_{0},\varphi \rangle +b\Vert u_{0}\Vert ^{2m-2}\langle u_{0},\varphi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\varphi dx\\{} & {} \quad \ge \int _{\Omega }f(x)|u_{0}|^{-\gamma }\varphi dx, \forall \varphi \in G_{0},\ \varphi \ge 0. \end{aligned}$$

Combining Part 1 and Part 2, we can get the same result, that is (5.5).

Next, we need to prove that (5.5) applicable to any arbitrary \(\phi \in G_{0}\). We consider \(\psi _{\epsilon }=u_{0}+\epsilon \phi \) with \(\epsilon >0\) and \(\phi \in G_{0}.\) Denoting \(\Omega _{\epsilon }=\left\{ x\in \mathbb R^{N}: \psi _{\epsilon }(x)\le 0\right\} \). Then, from (5.5) with test function \(\phi =\psi _{\epsilon }^{+}\), we get

$$\begin{aligned} 0{} & {} \le (a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{+}\rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\psi _{\epsilon }^{+} dx -\int _{\Omega }f(x)|u_{0}|^{-\gamma }\psi _{\epsilon }^{+} dx\\{} & {} \le (a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }+\psi _{\epsilon }^{-}\rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}(\psi _{\epsilon }+\psi _{\epsilon }^{-}) dx\\{} & {} \quad -\int _{\Omega }f(x)|u_{0}|^{-\gamma }(\psi _{\epsilon }+\psi _{\epsilon }^{-}) dx\\{} & {} =(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},u_{0}+\epsilon \phi \rangle +(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{-}\rangle \\{} & {} \quad +\Big (\int _{\Omega }-\int _{\Omega _{\epsilon }}\Big )[h(x)|u_{0}|^{2_{s}^{*}-1}(u_{0}+\epsilon \phi )-f(x)|u_{0}|^{-\gamma }(u_{0}+\epsilon \phi )]dx\\{} & {} =\big [a\Vert u_{0}\Vert ^{2}+b\Vert u_{0}\Vert ^{2m}+\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx\big ]\\{} & {} \quad +\epsilon \big [(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx\big ]\\{} & {} \quad +(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{-}\rangle -\int _{\Omega _{\epsilon }}[h(x)|u_{0}|^{2_{s}^{*}-1}(u_{0}+\epsilon \phi )\\{} & {} \quad -f(x)|u_{0}|^{-\gamma }(u_{0}+\epsilon \phi )]dx\\{} & {} =\big [a\Vert u_{0}\Vert ^{2}+b\Vert u_{0}\Vert ^{2m}+\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}}dx-\int _{\Omega }f(x)|u_{0}|^{1-\gamma }dx\big ]\\{} & {} \quad +\epsilon \big [(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx\big ]\\{} & {} \quad +(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{-}\rangle -\int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}(u_{0}\\{} & {} \quad +\epsilon \phi )dx+\int _{\Omega _{\epsilon }}f(x)|u_{0}|^{-\gamma }(u_{0}+\epsilon \phi )dx. \end{aligned}$$

Note that \(u_{0}\in \mathbb X_{0}\) and \(u_{0}+\epsilon \phi \le 0\) in \(\Omega _{\epsilon }\), thus

$$\begin{aligned} \int _{\Omega _{\epsilon }}f(x)|u_{0}|^{-\gamma }(u_{0}+\epsilon \phi )dx<0 \end{aligned}$$

and

$$\begin{aligned} -\int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}(u_{0}+\epsilon \phi )dx>0. \end{aligned}$$

With these facts in the mind, we get

$$\begin{aligned}{} & {} 0\le \epsilon \big [(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx\big ]\nonumber \\{} & {} \quad +(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{-}\rangle -\int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}(u_{0}+\epsilon \phi )dx\nonumber \\{} & {} \quad +\int _{\Omega _{\epsilon }}f(x)|u_{0}|^{-\gamma }(u_{0}+\epsilon \phi )dx\nonumber \\{} & {} \le \epsilon \big [(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx\big ]\nonumber \\{} & {} \quad +(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{-}\rangle -\int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}(u_{0}+\epsilon \phi )dx\nonumber \\{} & {} \le \epsilon \big [(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx\big ]\nonumber \\{} & {} \quad +(a+b\Vert u_{0}\Vert ^{2m-2})\langle u_{0},\psi _{\epsilon }^{-}\rangle -\epsilon \int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx. \end{aligned}$$
(5.15)

Then, denote

$$\begin{aligned} \Re _{\epsilon }(x,y)=\frac{(u_{0}(x)-u_{0}(y))(\psi _{\epsilon }^{-}(x)-\psi _{\epsilon }^{-}(y))}{|x-y|^{N+2s}} \end{aligned}$$

and

$$\begin{aligned} \Re (x,y)=\frac{(u_{0}(x)-u_{0}(y))(\phi (x)-\phi (y))}{|x-y|^{N+2s}}. \end{aligned}$$

By the definition of scalar product and the symmetry of the fractional kernel, we have

$$\begin{aligned} \langle u_{0},\psi ^{-}_{\varepsilon }\rangle&=\iint _{E}\frac{(u_{0}(x)-u_{0}(y))(\psi ^{-}_{\epsilon }(x)-\psi ^{-}_{\epsilon }(y))}{|x-y|^{N+2s}}dxdy\\&=\iint _{(\mathbb R^{N}\times \mathbb R^{N})\setminus (\mathcal {C}\Omega \times \mathcal {C}\Omega )}\Re _{\epsilon }(x,y)dxdy\\&=\left( \iint _{\Omega \times \Omega }+\iint _{\Omega \times (\mathbb {R}^{N} \setminus \Omega )} +\iint _{(\mathbb {R}^{N}\setminus \Omega )\times \Omega }\right) \Re _{\epsilon }(x,y)dxdy\\&=\iint _{\Omega \times \Omega }\Re _{\epsilon }(x,y)dxdy +2\iint _{\Omega \times (\mathbb {R}^{N}\setminus \Omega )}\Re _{\epsilon }(x,y)dxdy. \end{aligned}$$

When \(\psi _{\epsilon }\) is not in \(\Omega _{\epsilon }\), that is \(\psi _{\epsilon }\ge 0\), then \(\psi _{\epsilon }^{-}=0\). Hence, we have

$$\begin{aligned} \begin{aligned}&\iint _{\Omega \times \Omega }\Re _{\epsilon }(x,y)dxdy +2\iint _{\Omega \times (\mathbb {R}^{N}\setminus \Omega )}\Re _{\epsilon }(x,y)dxdy\\&=\left( \iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }} +2\iint _{\Omega _{\epsilon }\times (\Omega \setminus \Omega _{\varepsilon })} +2\iint _{\Omega _{\epsilon }\times (\mathbb {R}^{N}\setminus \Omega )}\right) \Re _{\epsilon }(x,y)dxdy\\&=\iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }}\Re _{\epsilon }(x,y)dxdy +2\iint _{\Omega _{\epsilon }\times (\mathbb {R}^{N}\setminus \Omega _{\epsilon })}\Re _{\epsilon }(x,y)dxdy. \end{aligned} \end{aligned}$$

Next,

$$\begin{aligned}{} & {} \iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }}\frac{(u_{0}(x)-u_{0}(y))(\psi ^{-}_{\epsilon }(x)-\psi ^{-}_{\epsilon }(y))}{|x-y|^{N+2s}}dxdy=\iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }}\\{} & {} \quad \times \frac{(u_{0}(x)-u_{0}(y))(\psi ^{+}_{\epsilon }(x)-\psi ^{+}_{\epsilon }(y))}{|x-y|^{N+2s}}dxdy\\{} & {} \quad -\iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }}\frac{(u_{0}(x)-u_{0}(y))(\psi _{\epsilon }(x)-\psi _{\epsilon }(y))}{|x-y|^{N+2s}}dxdy \le -\epsilon \iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }}\Re (x,y)dxdy. \end{aligned}$$

Similarly,

$$\begin{aligned} \begin{aligned} 2\iint _{\Omega _{\epsilon }\times (\mathbb {R}^{N}\setminus \Omega _{\epsilon })}\Re _{\varepsilon }(x,y)dxdy \le -2\epsilon \iint _{\Omega _{\epsilon }\times (\mathbb {R}^{N}\backslash \Omega _{\epsilon })}\Re (x,y)dxdy. \end{aligned} \end{aligned}$$

In combination with the above, we can obtain

$$\begin{aligned} \langle u_{0},\psi ^{-}_{\epsilon }\rangle\le & {} -\epsilon \left( \iint _{\Omega _{\epsilon }\times \Omega _{\epsilon }} +2\iint _{\Omega _{\varepsilon }\times (\mathbb {R}^{N}\setminus \Omega _{\epsilon })}\right) \Re (x,y)dxdy\nonumber \\\le & {} 2\epsilon \iint _{\Omega _{\epsilon }\times \mathbb {R}^{N}}|\Re (x,y)|dxdy. \end{aligned}$$
(5.16)

Now using Hölder inequality, for \(u_{0}, \phi \in G_{0}\), we have

$$\begin{aligned} \begin{aligned} \iint _{\Omega _{\varepsilon }\times \mathbb {R}^{N}}\frac{u_{0}(x)-u_{0}(y)}{|x-y|^{\frac{N+2s}{2}}}\cdot \frac{\phi (x)-\phi (y)}{|x-y|^{\frac{N+2s}{2}}}dxdy \le ||u_{0}|| ||\varphi ||<\infty , \end{aligned} \end{aligned}$$

which implies that \(\Re (x,y) \in L^{1}(\mathbb R^{N}\times \mathbb R^{N}).\) Besides, for any \(\sigma >0\), there exists \(R_{\sigma }\) sufficiently large. According to the definition of \(\Omega _{\epsilon }\), \(u_{0}+\epsilon \phi \le 0\) in \(\Omega _{\epsilon }\). Due to \(u_{0}>0\) and \(\epsilon >0\), we can deduce that for any \(\psi _{\epsilon }\in \Omega _{\epsilon }\), \(\phi (x)<0\) is satisfied. According to \(\text {supp}\phi =\left\{ x\in \mathbb R^{N}: \phi (x)\ne 0\right\} \), we infer that \(\Omega _{\epsilon }\subset \text {supp}\phi \). Since

$$\begin{aligned} \iint _{\Omega _{\epsilon }\times \mathbb {R}^{N}}|\Re (x,y)|dxdy =\iint _{\Omega _{\epsilon }\times (\mathbb {R}^{N}\backslash B_{R_{\sigma }})}|\Re (x,y)|dxdy +\iint _{\Omega _{\epsilon }\times B_{R_{\sigma }}}|\Re (x,y)|dxdy, \end{aligned}$$

for the first item, we get

$$\begin{aligned} \begin{aligned} \iint _{\Omega _{\varepsilon }\times (\mathbb {R}^{N}\setminus B_{R_{\sigma }})}|\Re (x,y)|dxdy&<\iint _{({\text {supp}\phi })\times (\mathbb {R}^{N}\setminus B_{R_{\sigma }})}|\Re (x,y)|dxdy&<\frac{\sigma }{2}. \end{aligned} \end{aligned}$$

Also, we know that \(|\Omega _{\epsilon }\times B_{R_{\sigma }}|\rightarrow 0\) as \(\epsilon \rightarrow 0^{+}\), which implies the absolute continuity of the integral, and hence, there exists \(\delta _{\sigma }\) and \(\epsilon _{\sigma }\) such that for any \(\epsilon \in (0,\epsilon _{\sigma }]\),

$$\begin{aligned} |\Omega _{\epsilon }\times B_{R_{\sigma }}|<\delta _{\sigma },\text {and}\,\iint _{\Omega _{\epsilon }\times B_{R_{\sigma }}}|\Re (x,y)|dxdy <\frac{\sigma }{2}. \end{aligned}$$

Consequently, for any \(\epsilon \in (0,\epsilon _{\sigma }]\),

$$\begin{aligned} \iint _{\Omega _{\varepsilon }\times \mathbb {R}^{N}}|\Re (x,y)|dxdy<\sigma , \lim _{\varepsilon \rightarrow 0^{+}}\iint _{\Omega _{\varepsilon }\times \mathbb {R}^{N}}|\Re (x,y)|dxdy=0. \end{aligned}$$

Thus, according to (5.16), we get

$$\begin{aligned} \lim _{\epsilon \rightarrow 0^{+}}\frac{1}{\epsilon }\langle u_{0},\psi ^{-}_{\varepsilon }\rangle =0. \end{aligned}$$
(5.17)

With respect to \(\int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx\), since meas \(\{u_{0}+\epsilon \phi \le 0\}\rightarrow 0\) as \(\epsilon \rightarrow 0\), we obtain

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\int _{\Omega _{\epsilon }}h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx=0. \end{aligned}$$
(5.18)

Finally, dividing by \(\epsilon \) and letting \(\epsilon \rightarrow 0\) in (5.15), we get from (5.17) and (5.18) that

$$\begin{aligned}{} & {} a\langle u_{0},\phi \rangle +b||u_{0}||^{2m-2}\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx\nonumber \\{} & {} \quad -\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx\ge 0. \end{aligned}$$
(5.19)

According to the arbitrariness of \(\phi \), we know that (5.19) fits any \(\phi \in G_{0}.\) The inequality also applies equally well for \(-\phi \). Hence, one can see that

$$\begin{aligned} a\langle u_{0},\phi \rangle +b||u_{0}||^{2m-2}\langle u_{0},\phi \rangle +\int _{\Omega }h(x)|u_{0}|^{2_{s}^{*}-1}\phi dx-\int _{\Omega }f(x)|u_{0}|^{-\gamma }\phi dx=0. \end{aligned}$$

Finally, we claim that \(u_{0}\) is the unique solution of problem (1.1). Otherwise, suppose that \(v_{0}\) is another solution of problem (1.1). Then, it is easy to see that

$$\begin{aligned}{} & {} a\langle u_{0}-v_{0},u_{0}-v_{0}\rangle +b(\Vert u_{0}\Vert ^{2m-2}\langle u_{0},u_{0}-v_{0}\rangle -\Vert v_{0}\Vert ^{2m-2}\langle v_{0},u_{0}-v_{0}\rangle )\nonumber \\{} & {} \quad +\int _{\Omega }h(x)(u_{0}^{2_{s}^{*}-1}-v_{0}^{2_{s}^{*}-1})(u_{0}-v_{0})dx-\int _{\Omega }f(x)(u_{0}^{-\gamma }-v_{0}^{-\gamma })(u_{0}-v_{0})dx\nonumber \\{} & {} =a\Vert u_{0}-v_{0}\Vert ^{2}+b[\Vert u_{0}\Vert ^{2m}-\Vert u_{0}\Vert ^{2m-2}\langle u_{0},v_{0}\rangle -\Vert v_{0}\Vert ^{2m-2}\langle u_{0},v_{0}\rangle +\Vert v_{0}\Vert ^{2m}]\nonumber \\{} & {} \quad +\int _{\Omega }h(x)(u_{0}^{2_{s}^{*}-1}-v_{0}^{2_{s}^{*}-1})(u_{0}-v_{0})dx\nonumber \\{} & {} \quad -\int _{\Omega }f(x)(u_{0}^{-\gamma }-v_{0}^{-\gamma })(u_{0}-v_{0})dx=0. \end{aligned}$$
(5.20)

Since \(\gamma >1\) and \(2_{s}^{*}-1=(N+2s)/(N-2s)>1\), the following inequalities can be easily obtained:

$$\begin{aligned} \int _{\Omega }f(x)(u_{0}^{-\gamma }-v_{0}^{-\gamma })(u_{0}-v_{0})dx=\int _{\Omega }f(x)\frac{v_{0}^{\gamma }-u_{0}^{\gamma }}{u_{0}^{\gamma }v_{0}^{\gamma }}(u_{0}-v_{0}) \le 0, \end{aligned}$$
(5.21)

and

$$\begin{aligned} \int _{\Omega }h(x)(u_{0}^{2_{s}^{*}-1}-v_{0}^{2_{s}^{*}-1})(u_{0}-v_{0})\ge 0. \end{aligned}$$
(5.22)

Denote

$$\begin{aligned} \jmath (u_{0},v_{0})=\Vert u_{0}\Vert ^{2m}-\Vert u_{0}\Vert ^{2m-2}\langle u_{0},v_{0}\rangle -\Vert v_{0}\Vert ^{2m-2}\langle u_{0},v_{0}\rangle +\Vert v_{0}\Vert ^{2m}. \end{aligned}$$

By the Hölder inequality, we can get

$$\begin{aligned} \begin{aligned} \jmath (u_{0},v_{0})&\ge (\Vert u_{0}\Vert ^{2m-1}-\Vert v_{0}\Vert ^{2m-1})(\Vert u_{0}\Vert -\Vert v_{0}\Vert ) \ge 0. \end{aligned} \end{aligned}$$
(5.23)

Since if \(a>0\), using (5.21), (5.22) and (5.23) in (5.20), we can get \(\Vert u_{0}-v_{0}\Vert \le 0\). Then, \(\Vert u_{0}-v_{0}\Vert =0\). If \(a=0\), it also follows from (5.20), (5.21), (5.22) and (5.23) that \(u_{0}=v_{0}\). Thus, for every \(a\ge 0\) one has \(u_{0}=v_{0}\). Therefore, \(u_{0}\) is the unique positive solution of problem (1.1). This completes the proof of Theorem 1.1.