1 Introduction

In this paper, we are interested in critical exponential Kirchhoff systems on the Heisenberg group \({\mathbb {H}}^{n}\):

$$\begin{aligned} {\left\{ \begin{array}{ll} -K_1(\int _{\Omega }|\nabla _{{\mathbb {H}}^{n}} u|^{Q}d\xi )\Delta _{Q} u =\lambda f_1(\xi , u, v), &{} \hbox {for} \ \xi \in \Omega , \\ -K_2(\int _{\Omega }|\nabla _{{\mathbb {H}}^{n}} v|^{Q}d\xi )\Delta _{Q} v =\lambda f_2(\xi , u, v), &{} \hbox {for} \ \xi \in \Omega , \\ u = v =0, &{} \hbox {for} \ \xi \in \partial \Omega , \end{array}\right. } \end{aligned}$$
(1.1)

where \(\Delta _{Q}\) is the Q-Laplacian operator on the Heisenberg group \({\mathbb {H}}^{n}\), defined by

$$\begin{aligned} \Delta _{Q}(\cdot )={\text {div}}_{{\mathbb {H}}^{n}}(\left| \nabla _{{\mathbb {H}}^{n}} (\cdot )\right| _{{\mathbb {H}}^{n}}^{Q-2} \nabla _{{\mathbb {H}}^{n}}(\cdot )), \end{aligned}$$

\(\Omega\) is a bounded open smooth subset of the Heisenberg group \({\mathbb {H}}^{n},\) and \(\lambda >0\) is a positive parameter.

There are already several interesting papers devoted to the study of the Heisenberg group \({\mathbb {H}}^{n}\). For example, Pucci and Temperini [21] studied the existence of entire nontrivial solutions of (pq) critical systems on the Heisenberg group \({\mathbb {H}}^{n}\), by using the variational methods and the concentration-compactness principle. Pucci and Temperini [20], they accomplished the conclusions of Pucci and Temperini [21] and worked out some kind of elliptic systems involving critical nonlinearities and Hardy terms on the Heisenberg group \({\mathbb {H}}^{n}\). There are additional interesting results in Liang and Pucci [13], Pucci [18], Pucci [19] and Pucci and Temperini [22].

In the Euclidean case, Kirchhoff-type problems have attracted wave after wave of scholars. Kirchhoff [11] established a model given by the hyperbolic equation

$$\begin{aligned} \rho \frac{\partial ^2u}{\partial t^2}-\left( \frac{p_0}{h}+\frac{E}{2L} \int _0^L\left| \frac{\partial u}{\partial x}\right| ^2dx\right) \frac{\partial ^2u}{\partial x^2}=0, \end{aligned}$$

where parameters \(\rho\), \(p_0\), h, E, L are constants with some physical meaning, which extends the classical D’Alembert wave equation for free vibrations of elastic strings. In particular, Kirchhoff equation models also appear in physical and biological systems. We refer the reader to Alves et al. [2] for more details. After Kirchhoff’s work, Figueiredo and Severo [10] studied the following problem

$$\begin{aligned} \left\{ \begin{array}{cl} &{}-m\left( \int _{\Omega }|\nabla u|^{2} d x\right) \Delta u=f(x, u) \text{ in } \Omega , \\ &{}u=0 \text{ on } \partial \Omega , \end{array}\right. \end{aligned}$$

and addressed the existence of ground state solutions of the problems on \({\mathbb {R}}^{2}\) by using the minimax techniques with the Trudinger–Moser inequality. Mingqi et al. [15] studied the existence and multiplicity of solutions for a class of perturbed fractional Kirchhoff type problems with singular exponential nonlinearity. Alves and Boudjeriou [1] obtained the existence of a nontrivial solution for a class of nonlocal problems by using the dynamical methods. For several interesting results recovering the Kirchhoff-type problems, we refer to Ambrosio et al. [3], Caponi and Pucci [6], Ming1i et al. [14], and Pucci et al. [23], and the references therein.

Recently, some authors have focused their attention to the problem with critical exponential growth in the Euclidean case, see Aouaoui [4], Albuquerque et al. [8], Moser [16], and Trudinger [24]. In the Heisenberg group \({\mathbb {H}}^{n}\) case, Cohn and Lu [7] have established a new version of the Trudinger–Moser inequality: Let \(\Omega \subset {\mathbb {H}}^{n}\) and assume that \(|\Omega |<\infty\) and \(0<\alpha \le \alpha _{Q}.\) Then

$$\begin{aligned} \sup _{u \in W_{0}^{1, Q}(\Omega ),\left\| \nabla _{{\mathbb {H}}^{n}} u\right\| _{L^{Q}(\Omega )} \le 1} \frac{1}{|\Omega |} \int _{\Omega } e ^{\alpha |u(\xi )|^{\frac{Q}{Q-1}} }d \xi \le C_{0}<\infty , \end{aligned}$$
(1.2)

where \(C_{0}>0\) is a constant which depends only on \(Q=2 n+2\). Moreover,

$$\begin{aligned} \alpha _{Q}=Q \sigma _{Q}^{1 /(Q-1)} \quad \text{ and } \quad \sigma _{Q}=\int _{\rho (z, t)=1}|z|^{Q} d \mu . \end{aligned}$$

Deng and Tian [9] have established the existence of nontrivial solutions for the non-degenerate Kirchhoff elliptic system with nonlinear term have critical exponential growth. Although the study of critical Kirchhoff-type problems is more meaningful, there are some authors working on the degenerate Kirchhoff problem. From a physical point of view, the fact that \(M(0)=0\) means that the base tension of the string is zero, which is a very realistic model. To the best of our knowledge, the existence results for system (1.1) in the degenerate case are not yet known for the Heisenberg group \({\mathbb {H}}^{n}\).

For these reasons, we mainly consider the critical exponential Kirchhoff systems (1.1) on the Heisenberg group. We say that \(f_i\) satisfies critical exponential growth at \(+\infty\) provided that there exists \(\alpha _0 > 0\) such that

$$\begin{aligned} \lim \limits _{|(u, v)| \rightarrow +\infty } \frac{|f_{i}(\xi , u, v)|}{e^{{\alpha |(u, v)|^{\frac{Q}{Q-1}}}}}= {\left\{ \begin{array}{ll}0 &{} \text{ uniformly } \text{ on } \xi \in \Omega ,\text { for all }\alpha >\alpha _{0}; \\ +\infty &{} \text{ uniformly } \text{ on } \xi \in \Omega ,\text { for all }\alpha <\alpha _{0}.\end{array}\right. } \end{aligned}$$
(1.3)

In view of the critical exponential growth of the nonlinear terms \(f_{i}\), we work out the problem of the “lack of compactness” by using a new version of the Trudinger–Moser inequality for the Heisenberg group \({\mathbb {H}}^{n}\).

Throughout the paper, Kirchhoff-type functions \(K_i\) and \(f_i\) will satisfy the following conditions:

  • \(({\mathcal {K}})\) \(({\mathcal {K}}_1)\) There exist \(q\in [1,n)\) and \(\sigma \in [1, q),\) satisfying

    $$\begin{aligned} \sigma {\mathcal {K}}_{i}(t):=\sigma \int _{0}^{t} K_{i}(s) ds\ge K_{i}(t)t, \quad \hbox {for all} \quad t\geqslant 0. \end{aligned}$$

    \(({\mathcal {K}}_2)\) There exist \(k_1,k_{2} > 0\) such that \(K_{i}(t) \ge k_i t^{\sigma -1}\), where \(K(0) = 0,\) for all \(t \geqslant 0\).

  • \(({\mathcal {F}})\) \((F_1)\) \(\lim _{w \rightarrow 0} \frac{f_{i}(\xi ,w)}{|w|^{Q-1}}=0,\) for all \(w=(u, v)\) and \(|w|=\sqrt{u^{2}+v^{2}}\).

    \((F_2)\) \(F \in C^{1}({\mathbb {R}} \times {\mathbb {R}}, {\mathbb {R}})\) and there exists \(\mu >\sigma Q\) such that

    $$\begin{aligned} 0<\mu F(\xi ,w) \leqslant \nabla F(\xi ,w) w, \quad \hbox { for all} \ w \in {\mathbb {R}}^{2}, \ \hbox { where} \ \nabla F=\left( f_{1}, f_{2}\right) . \end{aligned}$$

    \((F_3)\) \(\liminf _{w \rightarrow 0^{+}} \frac{F(\xi , w)}{|w|^\mu }=: \eta >0.\)

Remark 1.1

Note that some typical examples of functions \(K_{i}:{\mathbb {R}}_{+} \rightarrow {\mathbb {R}}_{+}\) which are nondecreasing and satisfy conditions \(({\mathcal {K}}_1)\) and \(({\mathcal {K}}_2)\), are given by

$$\begin{aligned} K_{i}(t)=a_{i}+b_{i} t^{\sigma -1}, \ \hbox { for all} \ t \in {\mathbb {R}}, \ \hbox { where} \ a \in {\mathbb {R}}, b \in {\mathbb {R}} \ \hbox { and} \ a+b>0. \end{aligned}$$

Clearly, conditions \(({\mathcal {K}}_1)\) and \(({\mathcal {K}}_2)\) also cover the degenerate case, that is when \(a=0\).

Remark 1.2

By condition \(\left( F_{2}\right)\), we can easily get that \(F(\xi , w) / |w|^\mu\) is nondecreasing for all \(w>0\). Thus, for all \(w \geqslant 0,\) we obtain \(F(\xi , w) \geqslant \eta |w|^\mu\) by invoking condition \(\left( F_{3}\right)\).

The main result of this paper is as follows.

Theorem 1.1

Assume that conditions \(({\mathcal {K}})\) and \(({\mathcal {F}})\) are satisfied and that the nonlinear terms \(f_{i}\) have critical exponential growth. Then system (1.1) has at least one nontrivial solution for all \(\lambda >0\), when \(\eta >0\) from condition \(\left( F_{3}\right)\) is large enough.

In conclusion, we describe the structure of the paper. In Sect. 2 we collect all necessary preliminaries. In Sect. 3 we study the mountain pass geometry. In Sect. 4 we verify the compactness condition. Finally, in Sect. 5 we present the proof of our main result.

2 Preliminaries

We begin by recalling some key facts about the Heisenberg group \({\mathbb {H}}^{n}\), i.e. a Lie group of topological dimension \(2 n+1\) and the background manifold \({\mathbb {R}}^{2 n+2}\). We define

$$\begin{aligned}&\xi \circ \xi ^{\prime }=\tau _{\xi }\left( \xi ^{\prime }\right) =\left( x+x^{\prime }, y+y^{\prime }, t+t^{\prime }+2\left( x^{\prime } y-y^{\prime } x\right) \right) , \ \hbox { for all} \ \\&\xi =(x,y,z),\xi ^{\prime }=(x',y',z') \in {\mathbb {H}}^{n}. \end{aligned}$$

Furthermore,

$$\begin{aligned} \delta _{s}(\xi )=\left( s x, s y, s^{2} t\right) , \ \hbox { where} \ s>0, \end{aligned}$$

gives a natural group of dilations on \({\mathbb {H}}^{n}\), so

$$\begin{aligned} \delta _{s}\left( \xi _{0} \circ \xi \right) =\delta _{s}\left( \xi _{0}\right) \circ \delta _{s}(\xi ). \end{aligned}$$

The Jacobian determinant of dilatations \(\delta _{s}: {\mathbb {H}}^{n} \rightarrow {\mathbb {H}}^{n}\) is constant for all \(\xi =(x, y, t) \in {\mathbb {H}}^{n}\) and \(\delta _{s}={\mathbb {R}}^{2 n+2}\). Next,

$$\begin{aligned} B_{R}\left( \xi _{0}\right) =\left\{ \xi \in {\mathbb {H}}^{n}: d_{K}\left( \xi , \xi _{0}\right) <R\right\} \end{aligned}$$

denotes the Korányi open ball with radius R, centered at \(\xi _{0}\) (see Leonardi and Masnou [12]).

The horizontal gradient \(\nabla _{H}=(X, Y)\) and

$$\begin{aligned} X=\frac{\partial }{\partial x}+2 y \frac{\partial }{\partial t}, Y=\frac{\partial }{\partial y}-2 x \frac{\partial }{\partial t} \end{aligned}$$

give rise to the Lie algebra of left-invariant vector fields on \({\mathbb {H}}^{n}\). Finally, \(\Delta _{H} u={\text {div}}_{H}\left( \nabla _{H} u\right)\) represents the Kohn-Laplacian \(\Delta _{H}\). In addition, the degenerate elliptic operator \(\Delta _{H}\) satisfies Bony’s maximum principle (see Bony [5]).

Next, we define the classical Sobolev space \(W_{0}^{1, Q}(\Omega )\) as the closure of \({\mathcal {C}}_{0}^{\infty }(\Omega )\) with respect to the norm

$$\begin{aligned} \Vert u\Vert :=\left\| \nabla _{H^{n}} u\right\| _{W_{0}^{1, Q}}. \end{aligned}$$

Let

$$\begin{aligned} W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) :=W_{0}^{1, Q}(\Omega ) \times W_{0}^{1, Q}(\Omega ), \end{aligned}$$

endowed by the norm

$$\begin{aligned} \Vert (u, v)\Vert _{W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) }=(\Vert u\Vert _{W_{0}^{1, Q}(\Omega )}^{Q}+\Vert v\Vert _{W_{0}^{1, Q}(\Omega )}^{Q})^{1/Q}. \end{aligned}$$

System (1.1) is variational and the corresponding energy functional \(I_{\lambda }: W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) \rightarrow {\mathbb {R}}\) is given by

$$\begin{aligned} I_{\lambda }(u,v):=\frac{1}{Q}{\mathcal {K}}_{1}\left( \int _{\Omega }\left| \nabla _{{\mathbb {H}}^{n}} u\right| ^{Q} d \xi \right) +\frac{1}{Q} {\mathcal {K}}_{2} \left( \int _{\Omega }\left| \nabla _{{\mathbb {H}}^{n}} v\right| ^{Q} d \xi \right) -\lambda \int _{\Omega } F(\xi , u,v) d \xi . \end{aligned}$$

By \(\left( F_{1}\right) ,\left( F_{2}\right)\) and (1.3), there exists for \(\varepsilon>0,r>Q\), a constant \(C=C(\varepsilon ,r)>0\) such that

$$\begin{aligned} |f_{1}(\xi , w)|+|f_{2} (\xi , w)| \le \varepsilon |w|^{Q-1}+C|w|^{r-1} e ^{\alpha |w|^{Q^{\prime }}}, \quad \text{ for } \text{ all } \alpha >\alpha _{0}. \end{aligned}$$
(2.1)

Consequently, we have

$$\begin{aligned} F(\xi , w) \le \frac{\varepsilon }{\theta Q}|w|^{Q}+C|w|^{r} e ^{\alpha |w|^{Q^{\prime }}}, \quad \text{ for } \text{ all } \alpha >\alpha _{0}, \end{aligned}$$
(2.2)

where \(Q^{\prime }=\frac{Q}{Q-1}\). Since

$$\begin{aligned} (a+b)^{m} \le 2^{m-1}\left( a^{m}+b^{m}\right) , \ \text{ for } \text{ all } a, b \ge 0, \ m>0, \end{aligned}$$
(2.3)

we can obtain the following inequality

$$\begin{aligned} |w|^{Q^{\prime }}=\left( |u|^{2}+|v|^{2}\right) ^{\frac{Q^{\prime }}{2}} \le 2^{Q^{\prime }-1}\left( |u|^{Q^{\prime }}+|v|^{Q^{\prime }}\right) . \end{aligned}$$

Thus, invoking the Hölder inequality and (1.2), we get

$$\begin{aligned} \begin{aligned} \int _{\Omega } e ^{\alpha |w|^{Q^{\prime }}} d \xi&\le \int _{\Omega } e^{\left( \alpha 2^{Q^{\prime }-1}|u|^{Q^{\prime }}\right) } e ^{\left( \alpha 2^{Q^{\prime }-1}|v|^{Q^{\prime }}\right) } d \xi \\&\le \left( \int _{\Omega }e ^{\alpha 2^{Q^{\prime }}|u|^{Q^{\prime }}}d \xi \right) ^{1 / 2}\left( \int _{\Omega }e ^{\alpha 2^{Q^{\prime }}|v|^{Q^{\prime }} }d \xi \right) ^{1 / 2}<\infty . \end{aligned} \end{aligned}$$

Moreover, we can conclude that \(I_{\lambda }\in C^{1}( W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) , {\mathbb {R}})\) is well-defined and that the derivative of \(I_{\lambda }\) is

$$\begin{aligned} \left\langle I_{\lambda }^{\prime }(u, v),(\varphi , \psi )\right\rangle= & {} K_{1}(\Vert u\Vert ^{Q}) \int _{\Omega }\left| \nabla _{{\mathbb {H}}^{n}} u\right| _{{\mathbb {H}}^{n}}^{Q-2} \nabla _{{\mathbb {H}}^{n}} u \nabla _{{\mathbb {H}}^{n}} \varphi d\xi \\&+\, K_{2}\left( \Vert v\Vert ^{Q}\right) \int _{\Omega }\left| \nabla _{{\mathbb {H}}^{n}} v\right| _{{\mathbb {H}}^{n}}^{Q-2} \nabla _{{\mathbb {H}}^{n}} v \nabla _{{\mathbb {H}}^{n}} \psi d \xi \\&-\,\lambda \int _{\Omega } f_{1}(\xi , u, v)\varphi d\xi -\lambda \int _{\Omega }f_{2}(\xi , u, v)\psi d \xi, \ \ \hbox {for all} \ (u,v),(\varphi , \psi ) \in W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) . \end{aligned}$$

Therefore the solutions of system (1.1) coincide with the critical points of \(I_{\lambda }\).

Lemma 2.1

Suppose that condition \(\left( F_{1}\right)\) is satisfied and that

$$\begin{aligned} t \geqslant 0, \ \ \alpha >\alpha _{0}, \ \ \Vert w\Vert \le \rho , \ \mathrm{with} \quad \alpha \rho ^{Q^{\prime }}<\frac{\alpha _{Q}}{2Q^{\prime }}. \end{aligned}$$

Then there exists \(C=C(t, \alpha ,\rho )>0\) such that

$$\begin{aligned} \int _{\Omega }e^{\alpha |w|^{Q^{\prime }}}|w|^{t} d \xi \leqslant C\Vert w\Vert ^{t}. \end{aligned}$$

Proof

Due to the Hölder inequality, we get

$$\begin{aligned} \int _{\Omega }e ^{\alpha |w|^{Q^{\prime }}}|w|^{t} d \xi \le \left( \int _{\Omega } e ^{q \alpha |w|^{Q^{\prime }}} d \xi \right) ^{\frac{1}{q}}\Vert w\Vert _{p t}^{t}, \ \hbox {where} \ p,q>1,\frac{1}{p}+\frac{1}{q}=1, \ pt\ge Q. \end{aligned}$$

Since

$$\begin{aligned} \Vert w\Vert \leqslant \rho \ \hbox { with} \ \alpha \rho ^{Q^{\prime }}<\frac{\alpha _{Q}}{2^{Q^{\prime }}}, \end{aligned}$$

there exists \(q>1\) such that

$$\begin{aligned} 2^{^{Q^{\prime }}} q \alpha \rho ^{Q^{\prime }} \le \alpha _{Q}. \end{aligned}$$

By virtue of (2.3), one has

$$\begin{aligned} \begin{aligned} \int _{\Omega } e^{q\alpha |w|^{Q^{\prime }}} d \xi&\le \int _{\Omega } e ^{\left( 2^{Q^{\prime }-1} q \alpha \left( |u|^{Q^{\prime }}+|v|^{Q^{\prime }}\right) \right) } d \xi \le \left( \int _{\Omega } e ^{\left( 2^{Q^{\prime }} q \alpha |u|^{Q^{\prime }}\right) } d \xi \right) ^{\frac{1}{2}}\left( \int _{\Omega } e ^{\left( 2^{Q^{\prime }} q \alpha |v|^{Q^{\prime }}\right) }d \xi \right) ^{\frac{1}{2}} \\&\le \left( \int _{\Omega } e ^{\left( 2^{Q^{\prime }} q \alpha \rho ^{Q^{\prime }}\left( \frac{|u|}{\Vert w\Vert }\right) ^{Q^{\prime }}\right) } d \xi \right) ^{\frac{1}{2}}\left( \int _{\Omega } e ^{2^{Q^{\prime }} q \alpha \rho ^{Q^{\prime }}\left( \frac{|v|}{\Vert w\Vert }\right) ^{Q^{\prime }}}d \xi \right) ^{\frac{1}{2}} \le C, \end{aligned} \end{aligned}$$

which implies that the conclusion of Lemma 2.1 is valid.\(\square\)

For other background information we refer the reader to the comprehensive monograph by Papageorgiou et al. [17].

3 Mountain pass geometry

In this section we shall prove that \(I_{\lambda }\) satisfies the mountain pass geometry.

Lemma 3.1

Suppose that conditions \(({\mathcal {K}})\), \(({\mathcal {F}})\) are satisfied and that \(f_{i}\) have exponential critical growth. Then the following properties hold:

  • \(\left( I_{1}\right)\) There exist \(\iota >0\) and \(\kappa >0\) such that \(I_{\lambda }(u, v) \geqslant \iota ,\) for all \(\Vert (u, v)\Vert =\kappa\).

  • \(\left( I_{2}\right)\) There exists \((e,e) \in W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right)\) with \(\Vert (e, e)\Vert >\kappa\) such that \(I_{\lambda }(e, e)<0\).

Proof

First, we prove assertion \((I_1).\) If \(r>\sigma Q\), \(\alpha >\alpha _{0},\) and \(0<\varepsilon <\min \left\{ k_{1}, k_{2}\right\} ,\) then by virtue of \((F_{1})\) and (2.2), we have

$$\begin{aligned} I_{\lambda }(u, v)\geqslant & {} \frac{1}{\sigma Q}\left( \min \left\{ k_{1}, k_{2}\right\} -\lambda \varepsilon \right) \Vert (u, v)\Vert ^{\sigma Q}\\&-\,C_{1}\lambda \int _{\Omega }e^{\alpha |w|^{Q^{\prime }}}|(u, v)|^{r} d\xi , \ \hbox { for all} \ (u, v) \in W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) . \end{aligned}$$

By Lemma 2.1 and the following inequality

$$\begin{aligned} \Vert (u, v)\Vert =\kappa < \left( \frac{\alpha _{Q}}{2\alpha Q^{\prime }}\right) ^{\frac{1}{Q^{\prime }}}, \end{aligned}$$

we obtain

$$\begin{aligned} I_{\lambda }(u, v) \geqslant \frac{1}{\sigma Q}\left( \min \left\{ k_{1}, k_{2}\right\} -\lambda \varepsilon \right) \kappa ^{\sigma Q}-C _{2}\kappa ^{r}. \end{aligned}$$

Next, we choose

$$\begin{aligned} \kappa < \left( \frac{\alpha _{Q}}{2\alpha Q^{\prime }}\right) ^{\frac{1}{Q^{\prime }}} \end{aligned}$$

so small that

$$\begin{aligned} \frac{1}{\sigma Q}\left( \min \left\{ k_{1}, k_{2}\right\} -\lambda \varepsilon \right) -C_{2} \kappa ^{r-\sigma Q}>0. \end{aligned}$$

This implies that

$$\begin{aligned} I_{\lambda }(u, v) \geqslant \iota :=\kappa ^{\sigma Q}\left[ \frac{1}{\sigma Q}\left( \min \left\{ k_{1}, k_{2}\right\} -\lambda \varepsilon \right) -C_{2} \kappa ^{r-\sigma Q}\right] , \ \hbox { for all} \ \Vert (u, v)\Vert =\kappa . \end{aligned}$$

Next, we prove assertion \(\left( I_{2}\right)\). Let \(\psi \in C_{0 }^{\infty }(B_{R}(\xi _{0}))\) be such that \(\psi \geqslant 0\) on \(B_{R}(\xi _{0})\) and let \(K={\text {supp}}(\psi )\). Invoking condition \(({\mathcal {K}})\), we obtain

$$\begin{aligned} {\mathcal {K}}(t) \le c_{i} t^{\sigma }+d_{i},\ \text{ for } \text{ all } \ t \geqslant 0, \ \hbox {where} \ c_{i}, d_{i}>0, i= 1, 2. \end{aligned}$$

By virtue of Remark 1.2, we get

$$\begin{aligned} I_{\lambda }(t \psi , t \psi ) \leqslant \frac{t^{\sigma Q}}{Q}\Vert \psi \Vert ^{\sigma Q}\left( c_{1}+d_{1}\right) +\frac{1}{Q}\left( c_{2}+d_{2}\right) -\lambda \eta t^{\mu } \int _{\Omega } \psi ^{\mu }d\xi . \end{aligned}$$

Since \(\mu >\sigma Q,\) we can conclude that

$$\begin{aligned} I_{\lambda }(t \psi , t \psi ) \rightarrow -\infty , \ \hbox { as} \ t \rightarrow +\infty . \end{aligned}$$

Therefore, we get the claim by using \(e:=t \psi\) for a sufficiently large \(t>0\). \(\square\)

4 Compactness condition

In this section, we shall prove the following compactness condition.

Lemma 4.1

Suppose that the following inequality holds

$$\begin{aligned} c<\frac{\nu _{0}(\mu -\sigma Q) \alpha _{Q}^{\sigma (Q-1)}}{ \alpha _{0}^{\sigma (Q-1)} Q\mu \sigma ^{\sigma Q+1}}, \ \mathrm{where} \ \nu _{0}:=\min \left\{ k_{1}, k_{2}\right\} . \end{aligned}$$
(4.1)

Then there exists a \((\mathrm {PS})_{c}\) sequence for \(I_{\lambda }\) \(\left\{ \left( u_{n}, v_{n}\right) \right\}\) such that the functional \(I_{\lambda }\) satisfies the Palais–Smale condition at level c.

Proof

First, let

$$\begin{aligned} \left\{ \left( u_{n}, v_{n}\right) \right\} \subset W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) \end{aligned}$$

be a \((PS)_{c}\) sequence for \(I_{\lambda }\). If \(\inf _{n\in {\mathbb {N}}}\Vert w_{n}\Vert = 0\), then

$$\begin{aligned} w_{n} \rightarrow 0 \ \hbox { in} \ W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) , \ \hbox { as} \ n \rightarrow \infty . \end{aligned}$$

Therefore, we shall use \(\inf _{n\in \mathbb N}\Vert w_{n}\Vert >0\) in the sequel.

By conditions \(({\mathcal {K}})\) and \(\left( F_{2}\right)\), we have

$$\begin{aligned} c+o_{n}(1)\left\| \left( u_{n}, v_{n}\right) \right\|\geqslant & {} I_{\lambda }\left( u_{n}, v_{n}\right) -\frac{1}{\mu } I_{\lambda }^{\prime }\left( u_{n}, v_{n}\right) \left( u_{n} v_{n}\right) \nonumber \\\geqslant & {} \left( \frac{1}{\sigma Q}-\frac{1}{\mu }\right) K_{1}\left( \left\| u_{n}\right\| ^{Q}\right) \left\| u_{n}\right\| ^{Q}+\left( \frac{1}{\sigma Q}-\frac{1}{\mu }\right) K_{2}\left( \left\| v_{n}\right\| ^{Q}\right) \left\| v_{n}\right\| ^{Q}\nonumber \\&+\,\frac{\lambda }{\mu } \int \left[ \nabla F\left( \xi ,u_{n}, v_{n}\right) \left( u_{n}, v_{n}\right) -\mu F\left( \xi ,u_{n}, v_{n}\right) \right] d \xi , \end{aligned}$$
(4.2)

which combined with conditions \(\left( H_{1}\right)\) and \(\left( F_{2}\right) ,\) implies

$$\begin{aligned} c+o_{n}(1)\left\| \left( u_{n}, v_{n}\right) \right\| \geqslant \left( \frac{\mu -\sigma Q}{\sigma Q\mu }\right) \nu _{0}\left\| \left( u_{n}, v_{n}\right) \right\| ^{\sigma Q}, \ \hbox {where} \ \nu _{0}:=\min \left\{ k_{1}, k_{2}\right\} . \end{aligned}$$

Hence, (4.2) implies that the sequence \(\left\{ \left( u_{n}, v_{n}\right) \right\}\) is bounded on \(W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right)\) and that

$$\begin{aligned} \limsup _{n \rightarrow +\infty }\left\| \left( u_{n}, v_{n}\right) \right\| ^{\sigma Q} \leqslant \frac{\sigma Q\mu }{(\mu -\sigma Q) \nu _{0}} c. \end{aligned}$$

Next, let \(\left( u_{0}, v_{0}\right) \in W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right)\) be such that \(\left( u_{n}, v_{n}\right) \rightharpoonup \left( u_{0}, v_{0}\right)\) weakly in \(W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right)\). We shall prove the convergence

$$\begin{aligned} \int _{\Omega } f_{1}\left( \xi ,u_{n}, v_{n}\right) \left( u_{n}-u_{0}\right) \mathrm {d} \xi \rightarrow 0, \ \int _{\Omega } f_{2}\left( \xi ,u_{n}, v_{n}\right) \left( u_{n}-u_{0}\right) \mathrm {d} \xi \rightarrow 0, \ \hbox {as} \ n \rightarrow +\infty . \end{aligned}$$
(4.3)

Let \(\varepsilon>0, \alpha>\alpha _{0}, s>1\), and \(s^{\prime }=s /(s-1).\) By (2.1) and Hölder’s inequality, we have

$$\begin{aligned}&\left| \int _{\Omega } f_{1}\left( \xi ,u_{n}, v_{n}\right) \left( u_{n}-u_{0}\right) d\xi \right| \leqslant \varepsilon \left\| \left( u_{n}, v_{n}\right) \right\| _{Q}^{Q-1}\left\| u_{n}-u_{0}\right\| _{Q^{\prime }}\\&\quad +\,C\left( \int _{\Omega }e^{\alpha s|w_{n}|^{Q^{\prime }}}d \xi \right) ^{1 / s}\left\| u_{n}-u_{0}\right\| _{s^{\prime }}. \end{aligned}$$

Thus, invoking (2.3) and the compactness of the embedding \(W_{0}^{1, Q}\left( \Omega \right) \hookrightarrow L^{s^{\prime }}\left( \Omega \right)\), we obtain

$$\begin{aligned} \left| \int _{\Omega } f_{1}\left( \xi ,u_{n}, v_{n}\right) \left( u_{n}-u_{0}\right) d\xi \right|\leqslant & {} \varepsilon C+C\left( \int _{\Omega }e^{s\alpha 2^{Q^{\prime }-1} (|u_{n}|^{Q^{\prime }}+|v_{n}|^{Q^{\prime }})} \mathrm {d} \xi \right) ^{\frac{1}{s}} o_{n}(1)\nonumber \\\leqslant & {} \varepsilon C+C\left( \int _{\Omega } e ^{2^{Q^{\prime }} s \alpha |u_{n}|^{Q^{\prime }}} d \xi \right) ^{\frac{1}{2s}}\left( \int _{\Omega } e^{2^{Q^{\prime }}s \alpha |v_{n}|^{Q^{\prime }}}d \xi \right) ^{\frac{1}{2s}}. \end{aligned}$$
(4.4)

Due to

$$\begin{aligned} \int _{\Omega } e ^{2^{Q^{\prime }} s \alpha |u_{n}|^{Q^{\prime }}} d \xi \leqslant \int _{\Omega } e ^{\left( \sigma ^{Q^{\prime }} s \alpha |w_{n}|^{Q^{\prime }}\left( \frac{|u_{n}|}{\Vert w_{n}\Vert }\right) ^{Q^{\prime }}\right) } d \xi , \end{aligned}$$

we can choose \(\delta >0\) such that

$$\begin{aligned} \sigma ^{Q^{\prime }}\alpha _{0}\left\| \left( u_{n}, v_{n}\right) \right\| ^{Q^{\prime }} \leqslant \alpha _{Q}-\delta , \ \hbox { for sufficiently large} \ n \in {\mathbb {N}}. \end{aligned}$$

Therefore, we have

$$\begin{aligned} \sigma ^{Q^{\prime }}s\alpha \left\| \left( u_{n}, v_{n}\right) \right\| ^{Q^{\prime }} <\alpha _{Q}, \ \hbox { for sufficiently large} \ n \in {\mathbb {N}}, \end{aligned}$$

where \(\alpha >\alpha _{0}\) is close to \(\alpha _{0}\) and \(s>1\) is close to 1.

It then follows from (1.2) that it suffices to show that the following holds

$$\begin{aligned} \int _{\Omega } e ^{\sigma ^{Q^{\prime }} s \alpha |u_{n}|^{Q^{\prime }}} d \xi \leqslant C. \end{aligned}$$

Similarly, we can get

$$\begin{aligned} \int _{\Omega } e ^{\sigma ^{Q^{\prime }}\sigma \alpha |v_{n}|^{Q^{\prime }}}d \xi \leqslant C. \end{aligned}$$

As above, we get (4.3). Finally, we define

$$\begin{aligned} \Phi (u, v):=\frac{1}{Q} {\mathcal {K}}_{1}\left( \Vert u\Vert ^{Q}\right) +\frac{1}{Q} {\mathcal {K}}_{2}\left( \Vert v\Vert ^{Q}\right) , \end{aligned}$$

where \({\mathcal {K}}_{i}\) is the convexity, when \(\left( {\mathcal {K}}\right)\) holds.

Due to weak lower semicontinuity, we have

$$\begin{aligned} \frac{1}{Q} {\mathcal {K}}_{1}\left( \left\| u_{0}\right\| ^{Q}\right) +\frac{1}{Q}{\mathcal {K}}_{2}\left( \left\| v_{0}\right\| ^{Q}\right) \leqslant \frac{1}{Q} \liminf _{n \rightarrow +\infty } {\mathcal {K}}_{1}\left( \left\| u_{n}\right\| ^{Q}\right) +\frac{1}{Q} \liminf _{n \rightarrow +\infty } {\mathcal {K}}_{2}\left( \left\| v_{n}\right\| ^{Q}\right) . \end{aligned}$$
(4.5)

Moreover, by virtue of (4.3) and convexity of \(\Phi (u, v)\), we obtain

$$\begin{aligned} \begin{aligned} \Phi \left( u_{0}, v_{0}\right) -\Phi \left( u_{n}, v_{n}\right)&\geqslant \Phi ^{\prime }\left( u_{n}, v_{n}\right) \left( u_{0}-u_{n}, v_{0}-v_{n}\right) \\ {}&= I_{\lambda }^{\prime }\left( u_{n}, v_{n}\right) \left( u_{0}-u_{n}, v_{0}-v_{n}\right) +\lambda \int _{\Omega } f_{1}\left( \xi ,u_{n}, v_{n}\right) \left( u_{0}-u_{n}\right) \mathrm {d} \xi \\&\quad +\lambda \int _{\Omega } f_{2}\left( \xi ,u_{n}, v_{n}\right) \left( v_{0}-v_{n}\right) \mathrm {d}\xi . \end{aligned} \end{aligned}$$

Therefore, we have

$$\begin{aligned} \Phi \left( u_{0}, v_{0}\right) +o_{n}(1) \geqslant \Phi \left( u_{n}, v_{n}\right) \end{aligned}$$

and we get

$$\begin{aligned} \begin{aligned} \frac{1}{Q}{\mathcal {K}}_{1} \left( \left\| u_{0}\right\| ^{Q}\right) +\frac{1}{Q}{\mathcal {K}}_{2}\left( \left\| v_{0}\right\| ^{Q}\right)&\geqslant \limsup _{n \rightarrow +\infty } \Phi \left( u_{n}, v_{n}\right) \\&\geqslant \frac{1}{Q} \liminf _{n \rightarrow +\infty } {\mathcal {K}}_{1}\left( \left\| u_{n}\right\| ^{Q}\right) +\frac{1}{Q} \liminf _{n \rightarrow +\infty } {\mathcal {K}}_{2}\left( \left\| v_{n}\right\| ^{Q}\right) . \end{aligned} \end{aligned}$$

This fact together with (4.5), yields the contradiction. Therefore,

$$\begin{aligned} \frac{1}{Q} {\mathcal {K}}_{1}\left( \left\| u_{n}\right\| ^{Q}\right) \rightarrow \frac{1}{Q} {\mathcal {K}}_{1}\left( \left\| u_{0}\right\| ^{Q}\right) \quad \text{ and } \quad \frac{1}{Q} {\mathcal {K}}_{2}\left( \left\| v_{n}\right\| ^{Q}\right) \rightarrow \frac{1}{Q} {\mathcal {K}}_{2}\left( \left\| v_{0}\right\| ^{Q}\right) , \ \hbox {as} \ n \rightarrow +\infty . \end{aligned}$$

We can conclude that

$$\begin{aligned} \left\| u_{n}\right\| ^{Q} \rightarrow \left\| u_{0}\right\| ^{Q} \ \hbox {and} \ \left\| v_{n}\right\| ^{Q} \rightarrow \left\| v_{0}\right\| ^{Q}, \end{aligned}$$

since \(K_1(t)\) and \(K_2(t)\) are increasing for \(t>0,\) as \(n \rightarrow +\infty\). Therefore, \(\left( u_{n}, v_{n}\right) \rightarrow \left( u_{0}, v_{0}\right)\) strongly in E,  and the proof is complete. \(\square\)

5 Proof of Theorem 1.1

We claim that

$$\begin{aligned} c^{*}:=\inf _{\gamma \in \Gamma } \max _{t \in [0,1]} I_{\lambda }(\gamma (t)), \end{aligned}$$
(5.1)

where

$$\begin{aligned} \Gamma :=\left\{ \gamma \in C([0,1], W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right) ): \gamma (0)=(0,0) \text{ and } I_{\lambda }(\gamma (1))<0\right\} . \end{aligned}$$

If we assume that (5.1) holds, then Lemmas 3.1 and 4.1 and the Mountain pass lemma yield the existence of nontrivial critical points of \(I_{\lambda }\).

Lemma 5.1

Assume that

$$\begin{aligned} \eta >\max \left\{ \eta _{1},\frac{\left( Q \eta _{1}\right) ^{\frac{\mu }{Q}}}{\mu }\left( \frac{\sigma ^{\sigma Q+1}\alpha _{0}^{\sigma (Q-1)} \lambda \pi (\mu -Q)}{\nu _{0}\alpha _{Q}^{\sigma (Q-1)}(\mu -\sigma Q)}\right) ^{\frac{\mu -Q}{Q}}\right\} , \end{aligned}$$
(5.2)

where

$$\begin{aligned} \eta _{1}:=[{\mathcal {K}}_{1}(2\varepsilon ^{Q}\pi )+{\mathcal {K}}_{2}(2\varepsilon ^{Q}\pi )] /(Q \lambda \pi ) \quad \mathrm{and} \quad \ v_{0}:=\min \left\{ k_{1}, k_{2}\right\} . \end{aligned}$$

Then the following inequality holds

$$\begin{aligned} c^{*}< \frac{\nu _{0}(\mu -\sigma Q) \alpha _{Q}^{\sigma (Q-1)}}{ \alpha _{0}^{\sigma (Q-1)} Q\mu \sigma ^{\sigma Q+1}}. \end{aligned}$$
(5.3)

Proof

In order to prove (5.3), let \(\varepsilon >0\) be so small that there exists a cut-off function \(\psi _{\varepsilon }\in C_{0}^{\infty }( B_{R}(\xi _{0}))\) such that

$$\begin{aligned} 0 \leqslant \psi _{\varepsilon } \leqslant 1, \ {\text {supp}}(\psi _{\varepsilon }) \subset B_{\varepsilon }(0), \ \psi _{\varepsilon } \equiv 1\ \hbox {on} \ B_{\frac{\varepsilon }{2}}(0), \ |\nabla \psi _{\varepsilon }| \leqslant \frac{4}{\varepsilon }. \end{aligned}$$

Then we have

$$\begin{aligned} \Vert \psi _{\varepsilon }\Vert ^{Q}=\int _{B_{\varepsilon }(0)}|\nabla \psi _{\varepsilon }|^{Q} d\xi +\int _{B_{\varepsilon }(0)}\psi _{\varepsilon }^{Q} d \xi \leqslant 2\left| B_{\varepsilon }(0)\right| =2\left| B_{1}(0)\right| \varepsilon ^{Q}=2\varepsilon ^{Q}\pi . \end{aligned}$$

On the other hand, since \(\eta >\eta _{1}\), we obtain by \(\left( F_{3}\right) ,\)

$$\begin{aligned} I_{\lambda }(\psi _{\varepsilon }, \psi _{\varepsilon })<\frac{{\mathcal {K}}_{1}(2\varepsilon ^{Q}\pi ) +{\mathcal {K}}_{2}(2\varepsilon ^{Q}\pi )}{Q}-\lambda \eta _{1} \pi =0. \end{aligned}$$
(5.4)

By the definition of \(\gamma (t):=(t \psi _{\varepsilon }, t \psi _{\varepsilon })\), we get the path \(\gamma :[0,1] \rightarrow W_{0}^{1, Q}\left( \Omega , {\mathbb {R}}^{2}\right)\). Then \(\gamma \in \Gamma\) by (5.4), and we obtain

$$\begin{aligned} \begin{aligned} c^{*}&\leqslant \max _{t \in [0,1]}\left[ \frac{1}{Q} {\mathcal {K}}_{1}\left( t^{Q}\Vert \psi _{\varepsilon }\Vert ^{Q}\right) +\frac{1}{Q} {\mathcal {K}}_{2}\left( t^{Q}\Vert \psi _{\varepsilon }\Vert ^{Q}\right) -\lambda \int _{\Omega } F(t \psi _{\varepsilon }, t \psi _{\varepsilon }) \mathrm {d}\xi \right] \\&\leqslant \max _{t \in [0,1]}\left[ \left( \frac{{\mathcal {K}}_{1}(2\varepsilon ^{Q}\pi )+{\mathcal {K}}_{2}(2\varepsilon ^{Q}\pi )}{Q}\right) t^{Q}-\lambda \eta t^{\mu } \int _{B_{\frac{\varepsilon }{2}}(0)} \psi _{\varepsilon }^{\mu } \mathrm {d}\xi \right] \leqslant \lambda \pi \max _{t \in [0,+\infty )}\left[ \eta _{1} t^{Q}-\eta t^{\mu }\right] , \end{aligned} \end{aligned}$$

where \({\mathcal {K}}_{1}\) and \({\mathcal {K}}_{2}\) are convex. Consequently, we get

$$\begin{aligned} \max _{t \in [0,+\infty )}\left[ \eta _{1} t^{Q}-\eta t^{\mu }\right] =\frac{1}{\eta ^{\frac{Q}{\mu -Q}}}(\mu -Q) Q^{\frac{Q}{\mu -Q}}\left( \frac{\eta _{1}}{\mu }\right) ^{\frac{\mu }{\mu -Q}}. \end{aligned}$$

Moreover, we have

$$\begin{aligned} c^{*} \leqslant \frac{\lambda \pi }{\eta ^{\frac{Q}{\mu -Q}}}(\mu -Q) Q^{\frac{Q}{\mu -Q}}\left( \frac{\eta _{1}}{\mu }\right) ^{\frac{\mu }{\mu -Q}}. \end{aligned}$$

Therefore (5.3) holds when \(\eta\) satisfies (5.2). \(\square\)