1 Introduction

In this paper, we study the following elliptic system:

$$\begin{aligned} \displaystyle \left\{ \begin{array}{l} - \Delta _p u - \mu \displaystyle \frac{u^{p-1} }{|x|^p} = \frac{\eta _1 }{p^*} H_u(u,v) + \frac{\eta _2}{p^*(t)} \, \frac{ Q_u(u,v) }{|x|^t}, \\ - \Delta _p v - \mu \displaystyle \frac{v^{p-1} }{|x|^p} = \frac{\eta _1 }{p^*} H_v(u,v) + \frac{\eta _2}{p^*(t)} \, \frac{ Q_v(u,v) }{|x|^t}, \\ u,v>0, \ \ \ (u, v) \in \mathcal {D} \times \mathcal {D} , \end{array} \right. \end{aligned}$$
(1.1)

where \(1< p<N, \ 0\le \mu <\bar{\mu }:= \bigr ( {(N-p)}/{p} \bigr )^p,\ 0<t<p, \ \eta _1>0, \ \eta _2 >0,\, -\Delta _p \cdot := -\text {div} (|\nabla \cdot |^{p-2} \cdot )\) is the p–Laplace operator, the space \(\mathcal {D}:= D^{1,p} ({\mathbb {R}}^N) \) denotes the completion of \(C_0^\infty ({\mathbb {R}}^N )\) with respect to \((\int _{{\mathbb {R}}^N} |\nabla \cdot |^p\,\text {d}x )^{1/p}\), \(\bar{\mu } \) is the best Hardy constant, \(p^*:= {Np}/{(N-p)}\) is the critical Sobolev exponent, and \(p^*(t):= {p(N-t)}/{(N-p)}\) is the critical Hardy–Sobolev exponent with \(p^*(0)=p^*\). \(H_u, H_v, Q_u\), and \(Q_v\) are the partial derivatives of the 2–variable \(C^1\)–functions H(u, v) and Q(u, v), respectively. The functions H and Q satisfy the following conditions:

\((\mathcal {H}) \, H, Q\in C^1({\mathbb {R}}^+ \times {\mathbb {R}}^+, {\mathbb {R}}^+)\),

and the 1–homogenous functions G and \(\bar{G}\) are concave, where G and \(\bar{G}\) are defined as follows:

$$\begin{aligned} G(\alpha ^{p^*}, \beta ^{p^*})=H(\alpha ,\beta ) , \ \ \bar{G}(\alpha ^{p^*(t)}, \beta ^{p^*(t)})=Q(\alpha , \beta ) ,\quad \forall \, \alpha ,\beta \ge 0. \end{aligned}$$

The following properties are important and well known:

\((\mathcal {H}')\) Suppose F(s, t) is a q-homogeneous differential function with \(q\ge 1\). Then

  1. (i)

    \(sF_s(s,t)+tF_t(s,t)= q F(s,t), \ \ \ \forall s,t\in {\mathbb {R}}\);

  2. (ii)

    \(C_F\) is attained at some \((s_0,t_0)\in {\mathbb {R}}^2\), where

    $$\begin{aligned} C_F := \max \{F(s,t)| s,t\in {\mathbb {R}}, \ |s|^q+|t|^q=1 \}; \end{aligned}$$
  3. (iii)

    \(|F(s,t)|\le C_F (|s|^q+|t|^q), \ \ \ \forall \, s,t\in {\mathbb {R}}\);

  4. (iv)

    \(F_s(s,t)\) and \(F_t(s,t)\) are \((q-1)\)-homogeneous.

In this paper, we work in the product space \(\mathcal {D}\times \mathcal {D}\). The corresponding energy functional of (1.1) is defined on \(\mathcal {D}\times \mathcal {D}\) by

$$\begin{aligned} \left. \begin{array}{ll} I(u, v) := &{} \displaystyle \frac{1}{p}\int _{{\mathbb {R}}^N} \Bigr ( |\nabla u |^p + |\nabla v |^p - \mu \frac{ \, |u|^p + |v|^p }{|x|^p} \Bigr )\text {d}x \\ &{} \displaystyle - \ \frac{ \eta _1 }{p^*}\int _{{\mathbb {R}}^N} H(|u|,|v|) \text {d}x - \frac{ \eta _2 }{p^*(t)}\int _{{\mathbb {R}}^N} \frac{ Q(|u|,|v|)}{|x|^t}\text {d}x . \end{array} \right. \end{aligned}$$

Then \(I\in C^1 (\mathcal {D}\times \mathcal {D}, {\mathbb {R}})\). A pair of functions \((u, v)\in \mathcal {D}\times \mathcal {D}\) is said to be a solution of (1.1) if \(u,v>0\), and

$$\begin{aligned} (u,v)\ne (0,0), \ \ \ \ \ \langle I'(u,v), (\varphi , \phi ) \rangle =0 , \ \ \forall \, (\varphi , \phi ) \in \mathcal {D}\times \mathcal {D}, \end{aligned}$$

where \(I'(u,v)\) denotes the Fréchet derivative of I at (u, v).

Problem (1.1) is related to the Hardy and Hardy–Sobolev inequalities [8, 20]):

$$\begin{aligned}&\int _{{\mathbb {R}}^N}\frac{|u|^p}{|x |^{p}}\,\text {d}x \,\le \, \frac{1}{\bar{\mu }}\int _{{\mathbb {R}}^N}|\nabla u|^p\,\text {d}x \,, \forall \,\,u\in C_0^\infty ({{\mathbb {R}}^N})\,, \end{aligned}$$
(1.2)
$$\begin{aligned}&\quad \Bigr (\int _{{\mathbb {R}}^N}\frac{|u|^{p^*(t)}}{|x |^{t}}\,\text {d}x \Bigr )^{\frac{p}{ p^*(t)}}\,\le \,C(p,t)\int _{{\mathbb {R}}^N}|\nabla u|^p\,\text {d}x \,, \forall \,\,u\in C_0^\infty ({{\mathbb {R}}^N})\,, \end{aligned}$$
(1.3)

where C(p, t) is a constant depending on p and t , \(1<p<N\) and \(0\le t<p\).

By (1.2) the operator \(L:=(-\Delta _p \cdot -\mu { | \cdot |^{p-2} \cdot }/{|x|^p})\) is positive for all \(\mu <\bar{\mu }\), and therefore, the following equivalent norm of \(\mathcal {D}\) can be defined:

$$\begin{aligned} \Vert u\Vert := \Bigr (\int _{{\mathbb {R}}^N}\Bigr ( |\nabla u|^p - \mu \,\frac{|u|^p}{|x|^{p}} \Bigr )\text {d}x \Bigr )^{\frac{1}{p} }, \ \ \ \forall \, u\in \mathcal {D}. \end{aligned}$$

Suppose \((\mathcal {H})\) holds. By \((\mathcal {H}')\), (1.2) and (1.1), the following best Hardy–Sobolev constants are well defined:

$$\begin{aligned}&\displaystyle S(\mu , t) := \inf _{u\in \mathcal {D} \setminus \{0 \}}\displaystyle \frac{ \displaystyle \int _{{\mathbb {R}}^N}\Bigr ( |\nabla u|^p- \mu \,\frac{|u|^p}{|x|^{p}} \Bigr )\text {d}x }{\Bigr ( \displaystyle \int _{{\mathbb {R}}^N} {\frac{|u|^{p^{*}(t) }}{|x|^t}} \text {d}x \Bigr )^{\frac{p}{p^*(t) }}} \, , \end{aligned}$$
(1.4)
$$\begin{aligned}&\displaystyle S_H( \mu , 0) := \inf _{u, v \in \mathcal {D} \setminus \{0 \}}\displaystyle \frac{ \displaystyle \int _{{\mathbb {R}}^N}\Bigr ( |\nabla u|^p + |\nabla v|^p - \mu \,\frac{|u|^p+|v|^p}{|x|^{p}} \Bigr )\text {d}x }{\Bigr ( \displaystyle \int _{{\mathbb {R}}^N} H(|u|,|v|) \text {d}x \Bigr )^{\frac{p}{p^* }}}, \end{aligned}$$
(1.5)
$$\begin{aligned}&\displaystyle S_Q( \mu ,t ) := \inf _{u, v \in \mathcal {D} \setminus \{0 \}}\displaystyle \frac{ \displaystyle \int _{{\mathbb {R}}^N}\Bigr ( |\nabla u|^p + |\nabla v|^p - \mu \,\frac{|u|^p+|v|^p}{|x|^{p}} \Bigr )\text {d}x }{\Bigr ( \displaystyle \int _{{\mathbb {R}}^N} {\frac{Q(|u|,|v|) }{|x|^t}} \text {d}x \Bigr )^{\frac{p}{p^*(t) }}}, \end{aligned}$$
(1.6)

where \( 0\le t<p, -\infty < \mu < \bar{\mu }\). It should be mentioned that the strongly coupled terms \(\int _{{\mathbb {R}}^N} H(|u|,|v|)\text {d}x\) and \(\int _{{\mathbb {R}}^N} \frac{Q(|u|,|v|)}{|x|^t}\text {d}x\) are critical in the senses of Sobolev or Hardy–Sobolev embedding. Morais Filho et al. studied the constant \(S_H(0,0)\) and proved the existence of solutions for a quasilinear elliptic systems in [17]. Alves et al. studied in [3] the following best constant and found its extremals:

$$\begin{aligned} A( \sigma , \tau ) := \inf _{u, v \in D^{1,2}({\mathbb {R}}^N) \setminus \{0 \}}\displaystyle \frac{ \displaystyle \int _{{\mathbb {R}}^N}\bigr ( |\nabla u|^2+ |\nabla v|^2 \bigr )\text {d}x }{\Bigr ( \displaystyle \int _{{\mathbb {R}}^N} { {|u|^{ \sigma } |v|^\tau } } \text {d}x \Bigr )^{\frac{2}{2^* }}}, \end{aligned}$$
(1.7)

where \(1<\sigma , \tau < 2^* -1, \ \ \ \sigma +\tau =2^*:= {2N}/{(N-2)}\). Note that \(A( \sigma , \tau )\) in (1.7) is a special case of \(S_H(0,0)\). The methods and conclusions in [3] and [17] are very stimulating.

In recent years, much attention has been paid to the semilinear and quasilinear elliptic problems involving the Hardy and Hardy–Sobolev inequalities, and many results were obtained providing us very good insight into the problems (e.g., [1, 5, 6, 9–11, 14, 15, 18, 19, 22, 23, 30, 32, 33], and the references therein). In particular, Filippucci et al. studied in [18] the following problem:

$$\begin{aligned} \displaystyle \left\{ \begin{array}{l} - \Delta _p u - \mu \displaystyle \frac{u^{p-1}}{|x|^p} = u ^{p^*-1} + \frac{ u^{p^*(s)-1} }{|x|^s}, \\ u \in \mathcal {D} , \ \ \ u > 0 \ \ \text {in} \ \ {\mathbb {R}}^N , \\ -\infty < \mu <\bar{\mu }, \ \ \ 0<s<p. \end{array} \right. \end{aligned}$$
(1.8)

The main difficulty of studying (1.8) is that the critical Hardy–Sobolev and Sobolev exponents appear simultaneously in the equation and induce more difficulties. By very technic and complicated analysis, the authors of [18] proved the existence of positive solutions to (1.8) by the Mountain–Pass theorem [4] and the concentration compactness principle [26, 27]. The extremals of the best constant \(S(\mu ,t)\) in (1.4) and some related singular quasilinear elliptic problems were investigated in [1, 18, 19] and [23], and we infer that, for all \(0\le t<p , \ 0\le \mu <\bar{\mu } , \) the best constant \(S_{\mu , t } \) is achieved by the implicit extremal function:

$$\begin{aligned} V _{ \mu , t } ^\varepsilon (x)= \varepsilon ^{\frac{p-N}{p}}U_{ \mu ,t } ( \varepsilon ^{-1}x ) \,, \ \ \ \forall \ \varepsilon >0\,, \end{aligned}$$
(1.9)

which satisfies

$$\begin{aligned} \displaystyle \int _{{\mathbb {R}}^N} \Bigr (|\nabla V _{ \mu , t } ^\varepsilon (x)|^p-\mu \frac{|V _{ \mu , t} ^\varepsilon (x)|^p}{|x |^p}\Bigr ) =\displaystyle \int _{{\mathbb {R}}^N } \frac{|V _{ \mu , t} ^\varepsilon (x)|^{p^*(t)}}{|x |^t} = (S_{\mu , t}) ^\frac{N-t}{p-t}, \end{aligned}$$

where \(\, U_{ \mu , t }(x ) \) is some radial function.

On the other hand, the singular elliptic systems involving the Hardy and Hardy–Sobolev inequalities have been seldom studied, we can only find several results in [2, 7, 16, 21, 24, 25, 28] and [29], where some nonlinear singular critical systems were investigated, the corresponding best Hardy–Sobolev constants were studied and existence results of solutions were obtained. The main difficulties of studying singular elliptic systems are that the singularity may occur and the strongly coupled terms may cause more difficulties.

To continue, we define

$$\begin{aligned}&\displaystyle M_H := \max \bigr \{H(|\alpha |,|\beta |)^\frac{p}{p^*}\bigr | \alpha ,\beta \in {\mathbb {R}}, \ |\alpha | ^{p }+ |\beta | ^{p }=1 \bigr \}; \end{aligned}$$
(1.10)
$$\begin{aligned}&\displaystyle M_Q := \max \bigr \{Q(|\alpha |,|\beta |)^\frac{p}{p^*(t)}\bigr | \alpha ,\beta \in {\mathbb {R}}, \ |\alpha | ^{p }+ |\beta | ^{p }=1 \bigr \}. \end{aligned}$$
(1.11)

Then there exist \((\alpha _i,\beta _i)\in {\mathbb {R}}^+\times {\mathbb {R}}^+, \ i=1,2\), such that \(M_H\) and \(M_Q\) are achieved respectively, that is,

$$\begin{aligned}&\displaystyle M_H = H(\alpha _1,\beta _1)^\frac{p}{p^*} , \ \ \ \alpha _1 ^{p }+ \beta _1 ^{p}=1, \end{aligned}$$
(1.12)
$$\begin{aligned}&\displaystyle M_Q = Q(\alpha _2,\beta _2)^\frac{p}{p^*(t)} , \ \ \ \alpha _2 ^{p }+ \beta _2^{p}=1. \end{aligned}$$
(1.13)

In this paper, stimulated by the references mentioned above, we investigate (1.1). The main results of this paper are summarized in the following theorems. To the best of our knowledge, the conclusions are new even in the case \(\mu =0\).

Theorem 1.1

Suppose that \( 0\le t<p, \ -\infty < \mu < \bar{\mu } \) and \((\mathcal {H})\) holds. Then

  1. (i)

    \(S_H(\mu , 0)=M_H^{-1}S(\mu ,0), \ \ S_Q(\mu , t)=M_Q^{-1}S(\mu ,t)\).

  2. (ii)

    For all \(0\le \mu < \bar{\mu }\), \(S_H (\mu ,0)\) has the minimizers \(\bigr ( \alpha _1 V_{\mu ,0} ^\varepsilon (x), \ \beta _1 V_{\mu ,0} ^\varepsilon (x)\bigr ) \), \(S_Q (\mu ,t)\) has the minimizers \(\bigr ( \alpha _2 V_{\mu ,t} ^\varepsilon (x), \ \beta _2 V_{\mu ,t} ^\varepsilon (x)\bigr )\), where \(V_{\mu ,t}^\varepsilon (x)\) are defined as in (1.9).

Theorem 1.2

Suppose that \(1< p<N, \ 0\le \mu <\bar{\mu }, \ 0<t<p, \ \eta _1>0\), \(\eta _2>0\) and \((\mathcal {H} ) \) holds. Then the problem (1.1) has a solution.

Remark 1.1

The coefficients \(1/p^{*}\) and \( {1}/{p^*(t)}\) in (1.1) are only used for the convenience of computation and have no particular meanings. By Theorem 1.1, the existence of solutions to (1.1) is obvious in anyone of the following cases: (i) \(\eta _1=0, \ \eta _2>0, \ t\ge 0;\) (ii) \(\eta _1>0, \ \eta _2=0, \ t\ge 0\); (iii) \(t=0, \ \eta _1>0, \ \eta _2> 0\).

Remark 1.2

The following problem is an example of (1.1) :

$$\begin{aligned} \displaystyle \left\{ \begin{array}{l} - \Delta _p u - \mu \displaystyle \frac{u^{p-1} }{|x|^p} = \frac{ \sigma _1}{p^*} u ^{\sigma _1-1} v ^{\tau _1 } + \frac{ \sigma _2}{p^*(t)} \, \frac{ u ^{\sigma _2-1} v ^{\tau _2 } }{|x|^t}, \\ - \Delta _p v - \mu \displaystyle \frac{v^{p-1} }{|x|^p} = \frac{ \tau _1 }{p^*} u ^{\sigma _1 } v ^{\tau _1-1 } + \frac{ \tau _2}{p^*(t)} \, \frac{ u ^{\sigma _2 } v ^{\tau _2-1 } }{|x|^t}, \\ u,v>0, \ \ \ (u, v) \in \mathcal {D} \times \mathcal {D} , \end{array} \right. \end{aligned}$$
(1.14)

where the parameters satisfy the following condition:

Note that (1.14) involves the critical Hardy–Sobolev and Sobolev exponents and admits a solution by Theorem 1.2.

This paper is organized as follows: Theorem 1.1 is verified in Sect. 2, and some preliminary results are established in Sect. 3, and Theorem 1.2 is proved in Sect. 4. In the following argument, \(\Vert u\Vert =(\int _{{{\mathbb {R}}^N}} (|\nabla u|^p -\mu {|u|^p} {|x|^{-p}} )\text {d}x )^{1/p}\) denotes the equivalent norm of the space \(\mathcal {D}, and\, \Vert (u, v)\Vert _{\mathcal {D}\times \mathcal {D}}=(\Vert u\Vert ^p+\Vert v\Vert ^p)^{1/p}\) is the norm of the space \(\mathcal {D} \times \mathcal {D}\). For all \(\varepsilon >0\) small enough, \(O(\varepsilon ^t)\) denotes the quantity satisfying \(|O(\varepsilon ^t)|/\varepsilon ^t \le C,\,o(\varepsilon ^t)\) means \(|o(\varepsilon ^t)|/\varepsilon ^t \rightarrow 0 \) as \(\varepsilon \rightarrow 0\) and o(1) is a generic infinitesimal value. In particular, the quantity \(O_1(\varepsilon ^t)\) means that there exist the constants \(C_1, C_2>0\) such that \(C_1\varepsilon ^t \le O_1(\varepsilon ^t) \le C_2\varepsilon ^t\) as \(\varepsilon \) small. We always denote positive constants as C and omit \(\text {d}x \) in integrals for convenience.

2 The Best Constants \(S_H(\mu ,0)\) and \(S_Q(\mu ,t)\)

In this section, we study \(S_H(\mu ,0)\) and \(S_Q(\mu ,t)\) and verify Theorem 1.1.

Proof of Theorem 1.1

(i) We only show the proof for \(S_Q(\mu ,t)\). The argument is similar to that of [17], where the best constant \(S_H(0,0)\) was studied.

Let \(w\in \mathcal {D} \setminus \{0\} \) and \((\alpha _2,\beta _2)\) be defined as in (1.13). Choosing \((u,v) =(\alpha _2w , \beta _2 w)\) in (1.6) we have

$$\begin{aligned} \frac{ (|\alpha _2|^p+|\beta _2|^p )\displaystyle \int _{{\mathbb {R}}^N} \Bigr (|\nabla w |^p- \mu \frac{|w|^p}{|x|^p}\Bigr ) }{ {| Q(\alpha _2,\beta _2)| ^{\frac{ p}{p^*(t)}}} \Bigr ( \displaystyle \int _{{\mathbb {R}}^N} \frac{|w |^{p^*(t) } }{|x|^t} \Bigr )^{\frac{p}{p^*(t)}}} \ge S_Q(\mu ,t). \end{aligned}$$
(2.1)

Taking the infimum as \(w\in \mathcal {D} \setminus \{0\}\) in (2.1), by (1.4) and (1.10)–(1.13) we have

$$\begin{aligned} M_Q^{-1} S(\mu ,t) \ge S_Q(\mu ,t). \end{aligned}$$
(2.2)

For any \(u,v\in \mathcal {D}\setminus \{0\}\), by Proposition 1 of [17] we have that

$$\begin{aligned} \int _{{\mathbb {R}}^N} \frac{Q(|u|,|v|) }{|x|^t}= & {} \int _{{\mathbb {R}}^N} Q\bigr ( |x|^{-\frac{t}{p^*(t)}}|u|, |x|^{-\frac{t}{p^*(t)}}| v|\bigr ) \nonumber \\\le & {} Q \bigr ( \Vert \, |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}, \Vert \,|x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)} \bigr ).\qquad \end{aligned}$$
(2.3)

Set

$$\begin{aligned} \theta :=\;\Bigr (\Vert \, |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p + \Vert \,|x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p\Bigr )^{-\frac{1}{p}}. \end{aligned}$$

Then

$$\begin{aligned} \Vert \theta |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p + \Vert \theta |x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p =1. \end{aligned}$$
(2.4)

From (1.11), (1.13), (2.3), and (2.4) it follows that

$$\begin{aligned} \left. \begin{array}{ll} &{} \frac{ \displaystyle \int _{{\mathbb {R}}^N}\Bigr ( |\nabla u|^p + |\nabla v|^p - \mu \frac{|u|^p+|v|^p}{|x|^p} \Bigr ) }{\Bigr ( \displaystyle \int _{{\mathbb {R}}^N} \frac{Q(|u|,|v|)}{|x|^t} \Bigr )^{\frac{p}{\alpha +\beta }}} \\ &{} \quad \displaystyle \ge S(\mu ,t)\frac{ \Bigr (\displaystyle \int _{{\mathbb {R}}^N} \frac{|u|^{p^*(t)}}{|x|^t} \Bigr )^\frac{p}{p^*(t)}\,+\,\Bigr (\displaystyle \int _{{\mathbb {R}}^N} \frac{|v|^{p^*(t)}}{|x|^t} \Bigr )^\frac{p}{p^*(t)} }{\Bigr (Q \bigr ( \Vert \, |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}, \Vert \,|x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)} \bigr )\Bigr )^{\frac{p}{p^*(t)}}} \\ &{}\quad \displaystyle =S(\mu ,t)\frac{ \Vert \, |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p + \Vert \,|x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p }{\Bigr (Q \bigr ( \Vert \, |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}, \Vert \,|x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)} \bigr )\Bigr )^{\frac{p}{p^*(t)}}} \\ &{}\quad \displaystyle =S(\mu ,t)\frac{ \Vert \theta |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p + \Vert \theta |x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}^p }{\Bigr (Q \bigr ( \Vert \theta |x|^{-\frac{t}{p^*(t)}}u \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)}, \Vert \theta |x|^{-\frac{t}{p^*(t)}}v \Vert _{L^{p^*(t)} ({\mathbb {R}}^N)} \bigr )\Bigr )^{\frac{p}{p^*(t)}}} \\ &{}\quad \displaystyle \ge \frac{1}{ | Q(\alpha _2,\beta _2)| ^{\frac{ p}{p^*(t)}} } S(\mu ,t) = M_Q^{-1} S(\mu ,t). \end{array} \right. \end{aligned}$$

Taking the infimum as \(u,v\in \mathcal {D}\setminus \{0\} \) we have

$$\begin{aligned} M_Q^{-1} S(\mu ,t) \le S_Q(\mu ,t), \end{aligned}$$

which together with (2.2) implies that

$$\begin{aligned} S_Q(\mu ,t)= M_Q^{-1} S(\mu ,t) . \end{aligned}$$

(ii) From (i), (1.5), and (1.6) the desired result follows. \(\square \)

3 Appropriate Palais–Smale Sequence

To find positive solutions of (1.1), we define the functional J on \(\mathcal {D}\times \mathcal {D}\) by

$$\begin{aligned} J(u, v) := \frac{1}{p} \Vert (u,v)\Vert ^p - \frac{ \eta _1 }{p^*}\int _{{\mathbb {R}}^N} H(u_+,v_+) - \frac{ \eta _2 }{p^*(t)}\int _{{\mathbb {R}}^N} \frac{Q(u_+,v_+)}{|x|^t} , \end{aligned}$$

where \(w_+=\max \{w,0\} \) for all \(w\in \mathcal {D}\). Then \(J\in C^1(\mathcal {D}\times \mathcal {D}, {\mathbb {R}}) \) according to \((\mathcal {H})\) and a solution of (1.1) is a nontrivial critical point of J. We follow the argument similar to that of [16], where the problem (1.8) was investigated.

Lemma 3.1

(Mountain–Pass lemma, [4]) Let E be a Banach space and \(\Phi \in C^1(E)\). Assume that

  1. (i)

    \(\Phi (0)=0\).

  2. (ii)

    There exist \(\lambda , R>0\) such that \(\Phi (u)\ge \lambda \) for all \(u\in E\) with \(\Vert u\Vert _E=R\).

  3. (iii)

    There exists \(v_0\in E\) such that \( \lim \sup _{ t\rightarrow \infty } \Phi (tv_0)<0\).

Take \(t_0>0\) such that \(\Vert t_0v_0\Vert _E>R\) and \(\Phi (t_0v_0)<0\). Set

$$\begin{aligned} \Gamma := \{ \gamma \in C([0,1], E)| \gamma (0) =0 \ \text {and} \ \gamma (1)=t_0v_0 \}, \quad c:=\inf _{\gamma \in \Gamma } \sup _{t\in [0,1]} \Phi (\gamma (t)). \end{aligned}$$

Then there exists a Palais–Smale sequence at level c for \(\Phi \), that is, there exists a sequence \(\{u_k \}\subset E\) such that

$$\begin{aligned} \lim _{k\rightarrow \infty } \Phi (u_k ) =c , \quad \lim _{k\rightarrow \infty } \Phi '(u_k ) =0 \, \, \text { strongly in } \, \, E^{-1} . \end{aligned}$$

Lemma 3.2

Suppose that \((\mathcal {H} )\) holds. Set

$$\begin{aligned} c^*:= \min \bigg \{ \frac{1}{N} \eta _1 ^\frac{p-N}{p} S_H(\mu ,0)^\frac{N}{p}, \ \frac{p-t}{p(N-t)} \eta _2 ^\frac{p-N}{p-t} S_Q( \mu , t)^\frac{N-t}{p-t}\bigg \}. \end{aligned}$$

Then for some \(c\in (0,c^*)\), there exists a Palais–Smale sequence at level c for J, that is there exists a sequence \(\{(u_k, v_k) \}\subset \mathcal {D}\times \mathcal {D}\) such that

$$\begin{aligned} \lim _{k\rightarrow \infty } J (u_k,v_k) =c , \quad \lim _{k\rightarrow \infty } J'(u_k,v_k) =0 \,\, \text { strongly in } \, \, (\mathcal {D}\times \mathcal {D})^{-1} . \end{aligned}$$

Proof

We divide the argument into several steps. \(\square \)

Claim 1

The functional J verifies the hypotheses of Lemma 3.1 at any \((u,v)\in \mathcal {D}\times \mathcal {D}\) with \( (u_+, v_+) \ne (0,0)\).

In fact, \(J\in C^1(\mathcal {D}\times \mathcal {D},{\mathbb {R}}), \ J(0,0)=0\). From (1.6) it follows that

$$\begin{aligned} \left. \begin{array}{ll} J(u,v) &{} \displaystyle \ge \frac{1}{p} \Vert (u,v)\Vert ^p \!-\! \frac{\eta _1}{p^*S_H(\mu ,0)^\frac{p^*}{p}}\Vert (u,v)\Vert ^{p^*}\!-\! \frac{\eta _2}{p^*(t)S_Q(\mu ,t)^\frac{p^*(t)}{p}}\Vert (u,v)\Vert ^{p^*(t)} \\ &{} \displaystyle =\,\Bigr (C_1-C_2\Vert (u,v)\Vert ^{p^*-p}-C_3\Vert (u,v)\Vert ^{p^*(t)-p}\Bigr )\Vert (u,v) \Vert ^p, \end{array} \right. \end{aligned}$$

where \(C_i, \ i=1,2,3, \) are positive constants. Then there exist \(\lambda , R>0\), such that \(J(u,v)\ge \lambda \) for all \((u,v)\in \mathcal {D}\times \mathcal {D} \) with \(\Vert (u,v)\Vert =R\). Furthermore, for any \((u,v)\in \mathcal {D}\times \mathcal {D}\) with \( (u_+, v_+) \ne (0,0)\), we have

$$\begin{aligned} \displaystyle \lim _{t\rightarrow +\infty }J(tu,tv)=-\infty , \end{aligned}$$

which implies that there exists \(t_{(u,v)}>0\) such that \(\Vert (t_{(u,v)}u, t_{(u,v)} v)\Vert >R\) and \(J(tu,tv)<0\) for all \(t> t_{(u,v)}\). Define

$$\begin{aligned} \begin{array}{c} \displaystyle \Gamma _{(u,v)} := \{ \gamma \in C([0,1], \mathcal {D}\times \mathcal {D})| \gamma (0) =(0,0) \ \text {and} \ \gamma (1)=(t_{(u,v)}u, t_{(u,v)} v) \}, \\ \displaystyle c_{(u,v)}:=\inf _{\gamma \in \Gamma _{(u,v)}} \sup _{t\in [0,1]} J(\gamma (t)). \end{array} \end{aligned}$$

Then the hypotheses of Lemma 3.1 are satisfied and there exists a sequence \(\{(u_k, v_k) \}\subset \mathcal {D}\times \mathcal {D}\) such that

$$\begin{aligned} \lim _{k\rightarrow \infty } J (u_k,v_k) =c_{(u,v)} , \quad \lim _{k\rightarrow \infty } J'(u_k,v_k) =0 \ \ \text {strongly in} \,\, (\mathcal {D}\times \mathcal {D})^{-1} . \end{aligned}$$

In particular, we have that

$$\begin{aligned} c_{(u,v)}\ge \lambda >0, \quad \forall \, (u,v)\in \mathcal {D}\times \mathcal {D}\setminus \{(0,0)\}. \end{aligned}$$

Claim 2

There exists \((u,v)\in \mathcal {D}\times \mathcal {D}\setminus \{(0,0)\}\) such that \(u, v \ge 0\) and

$$\begin{aligned} c_{(u,v)} <\frac{1}{N} \eta _1 ^\frac{p-N}{p} S_H(\mu ,0)^\frac{N}{p}. \end{aligned}$$

In fact, since \(\mu \in [0,\bar{\mu })\), by Theorem 1.1 we can choose \((u, v)=\bigr (\alpha _1V^\varepsilon _{\mu ,0}(x), \beta _1 V^\varepsilon _{\mu ,0}(x)\bigr )\), the extremals of \(S_H(\mu ,0)\). Then

$$\begin{aligned} \left. \begin{array}{ll} c_{(u,v)} &{}\displaystyle \le \sup _{t\ge 0} J(tu,tv) \le \sup _{t\ge 0}K(t) \\ &{}\displaystyle =\frac{1}{N } \bigg ( \frac{\Vert (u,v)\Vert ^p }{\bigr ( \eta _1 \int _{{\mathbb {R}}^N} H(u,v) \bigr )^{p/p^*} }\bigg )^{p^*/ (p^*-p)} \\ &{}\displaystyle = \frac{1}{N } \eta _1 ^\frac{p-N}{p} S_H( \mu , 0)^\frac{N}{p}, \end{array} \right. \end{aligned}$$

where

$$\begin{aligned} K(t):= \frac{t^p}{p}\Vert (u,v)\Vert ^p - \eta _1\frac{t^{p^*}}{p^*} \int _{{\mathbb {R}}^N} H(u,v). \end{aligned}$$

Let \(t_1, t_2>0\) be the points where \( \sup _{t\ge 0} J(tu,tv) \) and \( \sup _{t\ge 0}K(t) \) are attained, respectively. Suppose that \(J(t_1u,t_1v) = K(t_2)\). Then

$$\begin{aligned} K(t_1) - \eta _2 \frac{t_1^{p^*(t)}}{p^*(t)} \int _{{\mathbb {R}}^N} \frac{Q(u,v)}{|x|^t}=K(t_2), \end{aligned}$$

which implies that \(K(t_2)<K(t_1)\), a contradiction with the definition of \(t_2\). Consequently,

$$\begin{aligned} c_{(u,v)} \le \sup _{t\ge 0} J(tu,tv) <\sup _{t\ge 0}K(t)=\frac{1}{N} \eta _1 ^\frac{p-N}{p} S_H(\mu ,0)^\frac{N}{p}. \end{aligned}$$

Claim 3

There exists \((u,v)\in \mathcal {D}\times \mathcal {D}\setminus \{(0,0)\}\) such that \(u, v \ge 0\) and

$$\begin{aligned} 0<c_{(u,v)} <c^*. \end{aligned}$$

In fact, by Theorem 1.1 we can choose \((u, v)=\bigr (\alpha _2 V^\varepsilon _{\mu ,t}(x), \beta _2 V^\varepsilon _{\mu ,t}(x)\bigr ) >0\), the extremals of \(S_Q(\mu , t)\). Then arguing as above we can obtain that

$$\begin{aligned} \left. \begin{array}{ll} c_{(u,v)} &{} \displaystyle \le \sup _{t\ge 0} J(tu,tv) \\ &{}\displaystyle <\sup _{t\ge 0}\Bigr ( \frac{t^p}{p}\Vert (u,v)\Vert ^p - \eta _2\frac{t^{p^*(t)}}{p^*(t)} \int _{{\mathbb {R}}^N} \frac{Q(u,v)}{|x|^t} \Bigr ) \\ &{}\displaystyle =\frac{p-t}{p(N-t)} \eta _2 ^\frac{p-N}{p-t} S_Q( \mu , t)^\frac{N-t}{p-t}, \end{array} \right. \end{aligned}$$

which together with claim 2 implies that claim 3 holds.

From Lemma 3.1 and claims 1–3 it follows the conclusions of Lemma 3.2 for a suitable \((u,v)\in \mathcal {D}\times \mathcal {D}\).

Lemma 3.3

Let \(\{(u_k, v_k) \}\subset \mathcal {D}\times \mathcal {D}\) be a Palais–Smale sequence at the level \(c <c^*\) as in Lemma 3.2. If \(u_k \rightharpoonup 0\) and \(v_k \rightharpoonup 0\) weakly in \(\mathcal {D}\) as \(k\rightarrow \infty \), then there exists \(\varepsilon _0>0\) such that for all \(\delta >0\), either

$$\begin{aligned} \lim _{k\rightarrow \infty } \int _{B_\delta (0)} H\bigr ((u_k)_+, (v_k)_+ \bigr ) =0 \quad or \quad \lim _{k\rightarrow \infty } \int _{B_\delta (0)} H\bigr ((u_k)_+, (v_k)_+ \bigr ) \ge \varepsilon _0. \end{aligned}$$

Proof

The argument needs several steps. \(\square \)

Claim 4

For all \(\Omega \subset \subset {\mathbb {R}}^N \setminus \{0\}\), up to a subsequence, we have

$$\begin{aligned} \lim _{k\rightarrow \infty } \int _\Omega \frac{|u_k|^p}{|x|^p}= & {} \lim _{k\rightarrow \infty } \int _\Omega \frac{|v_k|^p}{|x|^p}= \lim _{k\rightarrow \infty } \int _\Omega \frac{ Q ( |u_k| , |v_k| )}{|x|^t}=0, \end{aligned}$$
(3.1)
$$\begin{aligned} \lim _{k\rightarrow \infty } \int _\Omega |\nabla u_k|^p= & {} \lim _{k\rightarrow \infty } \int _\Omega |\nabla v_k|^p = \lim _{k\rightarrow \infty } \int _\Omega H ( |u_k| , |v_k|) =0. \end{aligned}$$
(3.2)

In fact, since \(\Omega \subset \subset {\mathbb {R}}^N \setminus \{0\}\), the embedding \(\mathcal {D} \hookrightarrow L^q(\Omega )\) is compact for any \(1\le q<p^*\), \(|x|^{-1}\) is bounded on \(\Omega \) and \( p^*(t)<p^*\). Then (3.1) follows from \((\mathcal {H}')\) and we only need to verify (3.2).

Arguing as in Proposition 2 of [18], take \(\varphi \in C_0^\infty ({\mathbb {R}}^N\setminus \{0\}) \) such that \(0\le \varphi \le 1\) and \(\varphi |_\Omega \equiv 1\). Note that the weak convergence of \(\{ u_k\}\) and \(\{v_k\}\) in \(\mathcal {D}\) implies the boundedness. Then

$$\begin{aligned}&\int _{{\mathbb {R}}^N} | \nabla u_k|^{p-1}|\nabla (\varphi ^p)| |u_k| \le \Vert \nabla u_k\Vert _p^{p-1} \Vert u_k\Vert _{L^p(\mathrm{supp}|\nabla \varphi |)} =o(1), \\&\quad \int _{{\mathbb {R}}^N} | \nabla v_k|^{p-1}|\nabla (\varphi ^p)| |v_k| \le \Vert \nabla v_k\Vert _p^{p-1} \Vert v_k\Vert _{L^p(\mathrm{supp}|\nabla \varphi |)} =o(1), \\&\quad \int _{{\mathbb {R}}^N} \Bigr ( |\varphi \nabla u_k|^p + |\varphi \nabla v_k|^p \Bigr )=\int _{{\mathbb {R}}^N} \Bigr ( |\nabla (\varphi u_k)|^p + |\nabla (\varphi v_k)|^p \Bigr ) +o(1). \end{aligned}$$

Furthermore,

$$\begin{aligned} \left. \begin{array}{ll} o(1) &{}\displaystyle =\langle J'(u_k,v_k), (\varphi ^p u_k, \varphi ^p v_k) \rangle \\ &{}\displaystyle = \int _{{\mathbb {R}}^N} \Bigr ( |\varphi \nabla u_k|^p + |\varphi \nabla v_k|^p \Bigr )- \eta _1 \int _{{\mathbb {R}}^N} \varphi ^p H ((u_k)_+, (v_k)_+ )\\ &{}\displaystyle \quad + \ O\Bigr (\int _{{\mathbb {R}}^N} ( | \nabla u_k|^{p-1}|\nabla (\varphi ^p)| |u_k|+ | \nabla v_k|^{p-1}|\nabla (\varphi ^p)| |v_k|)\Bigr ) +o(1) \\ &{}\displaystyle =\int _{{\mathbb {R}}^N} \Bigr ( |\varphi \nabla u_k|^p + |\varphi \nabla v_k|^p \Bigr )- \eta _1 \int _{{\mathbb {R}}^N} \varphi ^p ( H ((u_k)_+, (v_k)_+ ) +o(1) \\ &{}\displaystyle =\int _{{\mathbb {R}}^N} \Bigr ( |\nabla (\varphi u_k)|^p + |\nabla (\varphi v_k)|^p \Bigr )- \eta _1 \int _{{\mathbb {R}}^N} \varphi ^p ( H ((u_k)_+, (v_k)_+ ) +o(1) \\ &{}\displaystyle \ge \Vert \varphi u_k\Vert ^p + \Vert \varphi v_k\Vert ^p - \eta _1 \int _{{\mathbb {R}}^N} \varphi ^p H ((u_k)_+, (v_k)_+ ) +o(1), \end{array} \right. \end{aligned}$$

which implies that

$$\begin{aligned} \left. \begin{array}{ll} &{}\Vert \varphi u_k\Vert ^p + \Vert \varphi v_k\Vert ^p \\ &{}\quad \displaystyle \le \eta _1 \int _{{\mathbb {R}}^N} \varphi ^p H ((u_k)_+, (v_k)_+ ) +o(1) \\ &{}\quad \displaystyle \le \eta _1 \Bigr (\int _{{\mathbb {R}}^N} H ((u_k)_+, (v_k)_+ )\Bigr )^{(p^*-p)/p^*} \Bigr (\int _{{\mathbb {R}}^N} H ( |\varphi u_k| , |\varphi v_k| )\Bigr )^{ p /p^*} +o(1) \\ &{}\quad \displaystyle \le \eta _1 \Bigr (\int _{{\mathbb {R}}^N} H ((u_k)_+, (v_k)_+ )\Bigr )^{(p^*-p)/p^*} S_H( \mu ,0)^{-1} \Vert (\varphi u_k,\varphi v_k ) \Vert ^p +o(1), \end{array} \right. \end{aligned}$$

and therefore,

$$\begin{aligned} \bigg (1- \eta _1 \Bigr (\int _{{\mathbb {R}}^N} H ((u_k)_+, (v_k)_+ ) \Bigr )^{(p^*-p)/p^*} S_H( \mu ,0)^{-1} \bigg ) \Vert (\varphi u_k,\varphi v_k ) \Vert ^p\le o(1).\nonumber \\ \end{aligned}$$
(3.3)

On the other hand,

$$\begin{aligned} J(u_k, v_k) - \frac{1}{p} \langle J'(u_k,v_k), (u_k,v_k) \rangle =c+o(1) \Vert (u_k,v_k) \Vert =c+o(1), \end{aligned}$$

which implies that

$$\begin{aligned} c+ o(1)= & {} \eta _1 \Bigr ( \frac{1}{p} - \frac{1}{p^*} \Bigr ) \int _{{\mathbb {R}}^N} H ((u_k)_+, (v_k)_+ ) \\&+\,\eta _2 \Bigr ( \frac{1}{p} - \frac{1}{p^*(t)} \Bigr ) \int _{{\mathbb {R}}^N} \frac{Q ((u_k)_+, (v_k)_+ )}{|x|^t}. \end{aligned}$$

Consequently,

$$\begin{aligned} \eta _1 \int _{{\mathbb {R}}^N} H ((u_k)_+, (v_k)_+ ) \le cN+ o(1), \end{aligned}$$
(3.4)

which together with (3.3) implies that

$$\begin{aligned} \Bigr (1- \eta _1^\frac{N-p}{N}(cN)^{p/N} S_H( \mu ,0)^{-1} \Bigr ) \Vert (\varphi u_k,\varphi v_k ) \Vert ^p\le o(1). \end{aligned}$$

Since \(c<c^*\), we have that

$$\begin{aligned} \lim _{k\rightarrow \infty } \Vert (\varphi u_k,\varphi v_k ) \Vert ^p=0, \end{aligned}$$

and therefore,

$$\begin{aligned} \lim _{k\rightarrow \infty } \int _{{\mathbb {R}}^N} H(|\varphi u_k| , |\varphi v_k| )=0. \end{aligned}$$

Then the definition of \(\varphi \) implies that (3.2) holds and claim 4 is proved.

Claim 5

For all \(\delta >0\), define the quantities:

$$\begin{aligned}&\displaystyle \tau = \limsup _{k\rightarrow \infty } \int _{B_\delta (0)} H((u_k)_+ , (v_k)_+ ), \quad \omega = \limsup _{k\rightarrow \infty } \int _{B_\delta (0)} \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t}, \\&\displaystyle \gamma = \limsup _{k\rightarrow \infty } \int _{B_\delta (0)} \Bigr (|\nabla u_k|^p + |\nabla v_k|^p -\mu \frac{|u_k|^p + |v_k|^p }{|x|^p}\Bigr ). \end{aligned}$$

Then

$$\begin{aligned} S_H( \mu , 0) \, \tau ^\frac{p}{p^*} \le \gamma , \quad S_Q( \mu , t) \, \omega ^\frac{p}{p^*(t)} \le \gamma . \end{aligned}$$
(3.5)

Furthermore,

$$\begin{aligned} \gamma \le \eta _1 \tau + \eta _2 \omega . \end{aligned}$$
(3.6)

In fact, according to claim 4, \(\tau , \omega \), and \(\gamma \) are well defined and independent of \(\delta \). Take \(\varphi \in C_0^\infty ({\mathbb {R}}^N ) \) such that \(0\le \varphi \le 1\) and \(\varphi |_{B_\delta (0)} \equiv 1\). Then we have

$$\begin{aligned} S_H( \mu , 0)\Bigr (\int _{{\mathbb {R}}^N} H((\phi u_k)_+ , (\phi v_k)_+ ) \Bigr )^\frac{p}{p^*} \le \Vert (\varphi u_k, \varphi v_k) \Vert ^p. \end{aligned}$$

As \(k\rightarrow \infty \), claim 4 implies that

$$\begin{aligned} \left. \begin{array}{ll} &{} \displaystyle S_H( \mu , 0)\Bigr (\int _{B_\delta (0)} H((u_k)_+ , (v_k)_+ ) \Bigr )^\frac{p}{p^*} \\ &{} \quad \displaystyle \le \int _{B_\delta (0)}\Bigr (|\nabla u_k|^p+ |\nabla v_k|^p -\mu \frac{|u_k|^p+|v_k|^p}{|x|^p}\Bigr )+o(1) . \end{array} \right. \end{aligned}$$

Consequently,

$$\begin{aligned} S_H( \mu , 0) \, \tau ^\frac{p}{p^*} \le \gamma . \end{aligned}$$

The second inequality in (3.5) can be verified similarly.

Since \(\varphi u_k, \varphi v_k \in \mathcal {D}\) and \(\lim _{k\rightarrow \infty } \langle J'(u_k,v_k), (\varphi u_k, \varphi v_k) \rangle =0, \) by claim 4 and the definitions of \(\tau , \omega \), and \(\gamma , \) we deduce that \(\gamma \le \eta _1 \tau + \eta _2 \omega . \) Claim 5 is verified.

From (3.6) it follows that

$$\begin{aligned} S_H( \mu , 0) \, \tau ^\frac{p}{p^*} \le \gamma \le \eta _1 \tau + \eta _2 \omega , \end{aligned}$$

which implies that

$$\begin{aligned} \tau ^\frac{p}{p^*}\left( S_H( \mu , 0) - \eta _1\tau ^ \frac{p^*-p}{p^*} \right) \le \eta _2\omega . \end{aligned}$$
(3.7)

From (3.4) it follows that

$$\begin{aligned} \eta _1\tau \le cN <c^* N < \eta _1 ^\frac{p-N}{p} S_H( \mu , 0) ^\frac{N}{p}= \eta _1 ^\frac{p-N}{p}S_H( \mu , 0) ^\frac{p^*}{p^*-p}. \end{aligned}$$
(3.8)

By (3.7) and (3.8), there exists a constant \(C_1=C_1( \mu , c,\eta _1, \eta _2)>0\) such that

$$\begin{aligned} \tau ^\frac{p}{p^*} \le C_1 \omega . \end{aligned}$$
(3.9)

Similarly, there exists a positive constant \(C_2=C_2( \mu , c,t, \eta _1,\eta _2)\) such that

$$\begin{aligned} \omega ^\frac{p}{p^*(t)} \le C_2 \tau . \end{aligned}$$
(3.10)

Then it follows from (3.9) and (3.10) that there exists a positive constant \(\varepsilon _0=\varepsilon _0(N, p, \mu , c,t)\) such that

$$\begin{aligned} \text {either} \, \, \tau =\omega =0\;\text {or}\;\min \{\tau , \omega \}\ge \varepsilon _0. \end{aligned}$$

The proof of Lemma 3.3 is complete.

4 Existence of Positive Solutions

Lemma 4.1

Let \(\{(u_k,v_k) \}\) be the sequence defined as in Lemma 3.3. Then

$$\begin{aligned} \Lambda := \limsup _{k\rightarrow \infty } \int _{{\mathbb {R}}^N} H((u_k)_+ , (v_k)_+ ) >0. \end{aligned}$$
(4.1)

Proof

Arguing by contradiction, we assume that

$$\begin{aligned} \lim _ {k\rightarrow \infty } \int _{{\mathbb {R}}^N} H((u_k)_+ , (v_k)_+ ) =0. \end{aligned}$$
(4.2)

Since \(\lim _{k\rightarrow \infty } \langle J'(u_k,v_k), ( u_k, v_k) \rangle =0\), by (4.1) we have

$$\begin{aligned} \Vert (u_k,v_k) \Vert ^p = \eta _2 \int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t}+o(1), \quad k\rightarrow \infty . \end{aligned}$$

Then

$$\begin{aligned}&\left. \begin{array}{ll} &{}\displaystyle S_Q( \mu ,t) \Bigr ( \int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t} \Bigr )^\frac{p}{p^*(t)} \\ &{}\quad \displaystyle \le \Vert (u_k,v_k) \Vert ^p = \eta _2 \int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t}+o(1), \end{array} \right. \nonumber \\&\qquad \left. \begin{array}{ll} &{}\displaystyle \Bigr ( \int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t} \Bigr )^\frac{p}{p^*(t)} \\ &{} \displaystyle \times \,\bigg ( S_Q( \mu ,t) - \eta _2 \Bigr ( \int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t} \Bigr )^\frac{p^*(t)- p}{p^*(t)} \bigg ) \le o(1). \end{array} \right. \end{aligned}$$
(4.3)

From (3.4) and (4.2) it follows that

$$\begin{aligned} \eta _2 \int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t}= \frac{cp(N-t)}{p-t}+o(1)< \frac{c^*p(N-t)}{p-t}+o(1), \end{aligned}$$

which together with (4.3) implies that

$$\begin{aligned} \lim _{k\rightarrow \infty }\int _{{\mathbb {R}}^N } \frac{Q((u_k)_+ , (v_k)_+ ) }{|x|^t}=0, \end{aligned}$$

a contradiction with (3.4) and the fact that \(c\in (0, c^*)\). \(\square \)

Lemma 4.2

Let \(\{(u_k,v_k) \}\) be defined as in Lemma 3.3. Then there exists \(\varepsilon _1\in (0, {\varepsilon _0}/{2}]\), with \(\varepsilon _0 \) given in Lemma 3.3, such that for all \(\varepsilon \in (0, \varepsilon _1)\), there exists a positive sequence \(\{r_k\}\subset {\mathbb {R}}\) such that \(\{( \tilde{u}_k, \tilde{v}_k) \}:= \bigr \{ \bigr ( r_k^{(N-p)/p} u_k(r_k x) , r_k^{(N-p)/p} v_k(r_k x)\bigr )\bigr \} \subset \mathcal {D}\times \mathcal {D}\), is again a Palais–Smale sequence of the type given in Lemma 3.3 and satisfies

$$\begin{aligned} \int _{B_1(0)} H((\tilde{u}_k)_+ , (\tilde{v}_k)_+ ) =\varepsilon , \quad \forall \, k\in \mathbb {N}. \end{aligned}$$
(4.4)

Proof

Let \(\varepsilon _0, \Lambda \) be defined as in Lemma 3.3 and (4.1), respectively. Set \(\varepsilon _1:= \min \{ {\varepsilon _0}/{2}, \Lambda \}\) and fix \(\varepsilon \in (0,\varepsilon _1)\). Up to a subsequence (still denoted by \(\{(u_k,v_k) \}\)), for any \(k\in \mathbb {N}\), there exists \(r_k>0\) such that

$$\begin{aligned} \int _{B_{r_k}(0)} H((u_k)_+ , (v_k)_+ ) =\varepsilon , \quad \forall \, k\in \mathbb {N}. \end{aligned}$$

Then the scaling invariance implies that \(\{ (\tilde{u}_k, \tilde{v}_k )\}\) satisfies (4.4) and is also a Palais–Smale sequence of the type given in Lemma 3.3. \(\square \)

Proof of Theorem 1.2

Since \(\{ (\tilde{u}_k, \tilde{v}_k )\}\) satisfies (4.4) and is also a Palais–Smale sequence, we have that

$$\begin{aligned} \left. \begin{array}{ll} &{} \displaystyle C(1 + \Vert ( \tilde{u}_k, \tilde{v}_k )\Vert ) \\ &{}\quad \displaystyle \ge J(\tilde{u}_k, \tilde{v}_k) - \frac{1}{p^*(t)} \langle J'(\tilde{u}_k,\tilde{v}_k), (\tilde{u}_k,\tilde{v}_k) \rangle \\ &{}\quad \displaystyle = \left( \frac{1}{p} - \frac{1}{p^*(t)} \right) \Vert (\tilde{u}_k, \tilde{v}_k ) \Vert ^p +\eta _1 \left( \frac{1}{p^(t)} - \frac{1}{p^*} \right) \int _{{\mathbb {R}}^N} H((\tilde{u}_k)_+ , (\tilde{v}_k)_+ ) \\ &{}\quad \displaystyle \ge \left( \frac{1}{p} - \frac{1}{p^*(t)} \right) \Vert (\tilde{u}_k, \tilde{v}_k ) \Vert ^p, \end{array} \right. \end{aligned}$$

which implies that \(\{ (\tilde{u}_k, \tilde{v}_k )\}\) is bounded in \(\mathcal {D}\times \mathcal {D}\). Up to a subsequence, there exists \(\tilde{u}, \tilde{v} \in \mathcal {D} \) such that

$$\begin{aligned} \tilde{u}_k \rightharpoonup \tilde{u} \,\, \text { weakly, } \quad \tilde{v}_k \rightharpoonup \tilde{v} \, \, \text { weakly, } \, \, k\rightarrow \infty . \end{aligned}$$

If \(\tilde{u}\equiv \tilde{v}\equiv 0\), from Lemma 3.3 it follows that either

$$\begin{aligned} \lim _{k\rightarrow \infty } \int _{B_1 (0)} H((\tilde{u}_k)_+ , (\tilde{v}_k)_+ ) =0 \quad or \quad \lim _{k\rightarrow \infty } \int _{B_1 (0)} H((\tilde{u}_k)_+ , (\tilde{v}_k)_+ )\ge \varepsilon _0, \end{aligned}$$

which contradicts (4.4) as \(0<\varepsilon < \varepsilon _0/2\). Then \((\tilde{u}, \tilde{v})\not \equiv (0,0)\). Arguing as in [12] (see also [13, 31, 33]), we deduce that \((\tilde{u}, \tilde{v})\) is a solution of the following problem:

$$\begin{aligned} \displaystyle \left\{ \begin{array}{l} - \Delta _p u - \mu \displaystyle \frac{u^{p-1}}{|x|^p} = \frac{\eta _1 }{p^*} H_u(u_+,v_+) + \frac{\eta _2 }{p^*(t)} \, \frac{ Q_u(u_+,v_+)}{|x|^t}, \\ - \Delta _p v - \mu \displaystyle \frac{v^{p-1}}{|x|^p} = \frac{\eta _1}{p^*} H_v(u_+,v_+) + \frac{\eta _2}{p^*(t)} \, \frac{Q_v(u_+,v_+)}{|x|^t}, \\ (u, v) \in \mathcal {D} \times \mathcal {D} . \end{array} \right. \end{aligned}$$
(4.5)

Set \(w_-=\max \{ -w, 0\}\) for all \(w\in \mathcal {D}\setminus \{0\}\). Multiplying the first equation in (4.5) by \(\tilde{u}_{-}\) and the second by \(\tilde{v}_-\), and integrating, we have that \(\Vert \tilde{u}_-\Vert =\Vert \tilde{v}_-\Vert =0\), which implies that \(\tilde{u}_- = \tilde{v}_- =0\), and therefore, \((\tilde{u}, \tilde{v}) \) is a nonnegative nontrivial solution of (4.5). If \(\tilde{u} \equiv 0\), by \((\mathcal {H})\) and (4.5) we get \(\tilde{v} \equiv 0\). Similarly, \(\tilde{v} \equiv 0 \) also implies \(\tilde{u} \equiv 0\). Then \(\tilde{u} \not \equiv 0\) and \( \tilde{v} \not \equiv 0\). From the maximum principle it follows that \(\tilde{u}, \tilde{v}>0\) in \({{\mathbb {R}}^N}\) and \((\tilde{u}, \tilde{v})\) is a solution of the problem (1.1).

The proof of Theorem 1.2 is complete. \(\square \)