A Class of Kirchhoff-Type Problems Involving the Concave–Convex Nonlinearities and Steep Potential Well

Abstract

This paper is concerned with the following Kirchhoff-type problem:

$$\begin{aligned} {\left\{ \begin{array}{ll} -\left( a+b\displaystyle {\int }_{\mathbb {R}^3}|\nabla u|^2\,\mathrm{d}x\right) \triangle u+\lambda V(x)u=g(x,u)+f(x,u)\quad \mathrm {in}\ {\mathbb {R}^3},\\ u\in H^{1}(\mathbb {R}^3), \end{array}\right. } \end{aligned}$$

where a, b and $\lambda$ are real positive parameters. The nonlinearity $g(x,u)+f(x,u)$ may involve a combination of concave and convex terms. By assuming that V represents a potential well with the bottom $V^{-1}(0)$, under some suitable assumptions on $f, g\in {\mathcal {C}}({\mathbb {R}}^3\times {\mathbb {R}},{\mathbb {R}})$, we obtain a positive energy solution $u_{b,\lambda }^{+}$ via combining the truncation technique and get the asymptotic behavior of $u_{b,\lambda }^{+}$ as $b\rightarrow 0$ and $\lambda \rightarrow +\infty$. Moreover, we also give the existence of a negative energy solution $u_{b,\lambda }^{-}$ via Ekeland variational principle.

1 Introduction

In this paper, we are concerned about the existence and asymptotic behavior of nontrivial solutions for the following Kirchhoff-type equations with concave–convex nonlinearities:

where a, b and $\lambda$ are real positive parameters, $V\in {\mathcal {C}}({\mathbb {R}}^3,{\mathbb {R}})$ and $f,g\in {\mathcal {C}}({\mathbb {R}}^3\times {\mathbb {R}},{\mathbb {R}})$. We assume that the potential V(x) satisfies the following conditions:    ($V_1$) $V(x) \in {\mathcal {C}}({\mathbb {R}}^3, {\mathbb {R}})$ and $V(x) \ge 0$ on ${\mathbb {R}}^3$;

($V_2$) there exists $c>0$ such that ${\mathcal {V}}_c:=\{x\in {\mathbb {R}}^3:V(x)<c\}$ is nonempty and has finite measure;

($V_3$) $\Omega =\mathrm {int}\,\,V^{-1}(0)$ is a nonempty open set with a smooth boundary and ${\overline{\Omega }}=V^{-1}(0)$.

It is well know that problem (${\mathcal {K}}_{b,\lambda }$) originates from the stationary analogue of the Kirchhoff equation

$$\begin{aligned} u_{tt}-\left( a+b\int _{\Omega }|\nabla u|^2\,\mathrm{d}x\right) \triangle u=f(x,u). \end{aligned}$$

(1.1)

Equation (1.1) was first presented by Kirchhoff [23], which proposed a hyperbolic-type equation

$$\begin{aligned} \rho \frac{\partial ^{2}u}{\partial ^{2}t}-\left( \frac{P_0}{h}+\frac{E}{2L}\int _{0}^{L}\left| \frac{\partial u}{\partial x}\right| ^{2}\,\mathrm{d}x\right) \frac{\partial ^{2}u}{\partial ^{2}x}=0 \end{aligned}$$

for free vibrations of elastic strings. This type of model extends the classical D’Alembert’s wave equation and takes into account the chord length variation induced by transverse oscillations. For this, more details on the Kirchhoff equation and further mathematical and physical interpretation, we recommend the readers to read [3, 6, 13] and the references therein. It is worth mentioning that after Lions [25] introduced an abstract functional analysis framework, problem (1.1) began to receive a lot of attention.

In (${\mathcal {K}}_{b,\lambda }$), if we set $V(x)= 0$ and replace ${\mathbb {R}}^3$ and $g(x,u)+f(x,u)$ by a bounded domain $\Omega \subset {\mathbb {R}}^N$ and f(x, u), respectively, then we get the following Kirchhoff-type equation:

$$\begin{aligned} {\left\{ \begin{array}{ll} -\left( a+b\displaystyle \int _{\Omega }|\nabla u|^2\,\mathrm{d}x\right) \triangle u=f(x,u)\quad &{}\mathrm {in}\ {\Omega },\\ u=0 &{}\mathrm {on}\ \partial \Omega . \end{array}\right. } \end{aligned}$$

(1.2)

Such problems are often referred to be nonlocal because of the presence of the term $\left( \int _{\Omega }|\nabla u|^2\,\mathrm{d}x\right) \triangle u$ which implies that (1.2) is no longer a pointwise identity. This phenomenon provokes some mathematical difficulties, which lead to the study of such a class of problems particularly interesting.

If $\lambda =1$ and $g(x,u)=0$ in (${\mathcal {K}}_{b,\lambda }$), then(${\mathcal {K}}_{b,\lambda }$) simplifies to the following Kirchhoff problem:

$$\begin{aligned} -\left( a+b\int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x\right) \,\triangle u+V(x)u=f(x,u),\quad x\in {{\mathbb {R}}^3}. \end{aligned}$$

(1.3)

Many researchers have focused on the effect of the nonlinear term f and potential V of Eq. (1.3) on the solution. One usually assumes that $V(x)\equiv 1$, or V(x) is periodic, or $V(x)={\bar{V}}(|x|)$, or V(x) is coercive, while f may be a general nonlinearity, or a critical nonlinearity, or a subcritical nonlinearity. By using variational method, there have been many results on the existence, nonexistence and multiplicity of nontrivial solutions, for such problem depending on the assumptions of the potential V and f. See, for example, [14, 17, 19,20,21, 24, 27, 29, 37] and references therein.

When $\lambda \ne 1$, we emphasize that hypotheses ($V_1$)–($V_3$) was firstly introduced by Bartsch and Wang [5] in the study of nonlinear Schrödinger equations. It is worth mentioning that the condition $(V_3)$ plays an important role in proving the asymptotic behavior of nontrivial solutions. We note that, the conditions ($V_1$)–($V_3$) imply that $\lambda V$ is referred as the steep potential well if $\lambda$ is sufficiently large and one expects to find solutions which localized near its bottom $\Omega$; especially, Zhang and Du [36] considered the problem (${\mathcal {K}}_{b,\lambda }$) with steep potential well while $\lambda >0$, $g(x,u)=0$ and $f(x,u):=|u|^{p-2}u$ as the case of $2<p<4$. In fact, $f(x,u):=|u|^{p-2}u$ with $2<p<4$ does not satisfy the Ambrosetti–Rabinowitz condition; the boundedness of Palais–Smale sequence becomes a major difficulty in proving the existence of a positive solution. By combining the truncation technique and the parameter-dependent compactness lemma, they proved the existence of positive solutions for b small and $\lambda$ large. Various elliptic equations with steep potential well are studied in [4, 9, 11, 21, 22, 30, 32, 34] and the references therein.

In (${\mathcal {K}}_{b,\lambda }$), we note that the nonlinearity $g(x,u)+f(x,u)$ may involve a combination of concave and convex terms. Equations like (${\mathcal {K}}_{b,\lambda }$) have been extensively studied due to its strong physical background. Combined effects of concave–convex nonlinearities were firstly investigated by Ambrosetti, Brézis and Cerami [2] on the following elliptic equation:

$$\begin{aligned} -\triangle u=\lambda |u|^{q-2}u+\mu |u|^{p-2}u,\quad u\in H_0^1(\Omega ), \end{aligned}$$

(1.4)

where $\Omega \subset {\mathbb {R}}^N$ is a bounded domain, $1<q<2<p<2^{*}$ and $2^{*}=\frac{2N}{N-2}$. They obtained infinitely many solutions with negative energy for $0 < \mu \ll \lambda =1$. Subsequently, Willem [33] extended the results of (1.4) in [2]; they proved the existence of infinitely many solutions with high energy by Fountain Theorem. Afterward, Liu and Wang [26] obtained nodal solutions of Schrödinger equation involving a combination of convex and concave terms.

In addition, for Kirchhoff equation with concave–convex nonlinearities, there are many papers concerned with the existence of standing wave solutions. Chen et al. [8] studied the following Kirchhoff-type problem of the form

$$\begin{aligned} -\left( a+b\int _{\Omega }|\nabla u|^{2}\,\mathrm{d}x\right) \triangle u=k_{1}a(x)|u|^{q-2}u+k_{2}b(x)|u|^{p-2}u,\quad x\in \Omega , \end{aligned}$$

(1.5)

where $\Omega \subset {\mathbb {R}}^3$ is bounded domain, $1<q<2$, $4<p<6$, $k_{1}=1$ and $k_{2}>0$. They obtained two solutions for (1.5) with $k_2>0$ small enough. Afterward, Cheng et al. [12] also proved that (1.5) has two positive solutions and two negative solutions with $a(x)=1$ and $b(x)\in L^{\infty }(\Omega )$. More recently, Shao and Mao [28] studied the following nonlinear Kirchhoff-type problem with concave–convex nonlinearities:

$$\begin{aligned} {\left\{ \begin{array}{ll} -\left( a+b\displaystyle \int _{\Omega }|\nabla u|^2\,\mathrm{d}x\right) \triangle u=\mu g(x,u)+f(x,u)\quad &{}\mathrm {in}\ \Omega ,\\ u = 0, &{}\mathrm {on}\ \partial \Omega , \end{array}\right. } \end{aligned}$$

(1.6)

where $\Omega \subset {\mathbb {R}}^3$ is a bounded domain with smooth boundary $\partial \Omega$, $\mu \in {\mathbb {R}}$ and $f,g\in {\mathcal {C}}({\overline{\Omega }},{\mathbb {R}})$. Under the conditions that f, g satisfy the following hypotheses:

$({\tilde{g}}_1)$:

there exist constants $1<q_1<q_2<\cdots<q_m<2$ and functions $h_{i}(x)\in L^{\frac{2}{2-q_i}}({\overline{\Omega }},{\mathbb {R}}^+)(i=1,\ldots , m)$ such that

$$\begin{aligned} |g(x,u)|\le \sum _{k=1}^{\infty }h_{i}(x)|u|^{q_i-1},\ \forall (x,u)\in \Omega \times {\mathbb {R}}; \end{aligned}$$

$({\tilde{g}}_2)$:

there exists $\theta \in (1,2)$ such that $0<\frac{1}{\theta }ug(x,u)\le G(x,u):=\int _{0}^{u}g(x,t)\,\mathrm{d}t$, $\forall x\in \Omega ,\ u\in {\mathbb {R}}\setminus \{0\}$;

$({\tilde{f}}_1)$:

there exist $C>0$ and $2< p < 6$ such that $|f(x,u)|\le C(1+|u|^{p-1})$, $\forall (x,u)\in \Omega \times {\mathbb {R}}$;

$({\tilde{f}}_2)$:

$\lim _{|u|\rightarrow 0} \frac{f(x,u)}{u} = 0$ uniformly in $\Omega$;

$({\tilde{f}}_3)$:

there exists $\nu >4$ such that

$$\begin{aligned} 0<\nu F(x,u)\le uf(x,u),\ \forall x \in \Omega ,\ u\in {\mathbb {R}}\setminus \{0\}, \end{aligned}$$

where $F(x, u) = \int _{0}^{u} f(x, t)\,\mathrm{d}t$;

$({\tilde{f}}_4)$:

there exists $R>0$ such that $\inf _{x\in \Omega ,|u|\ge R}F(x,u)>0,$

they proved the existence of infinitely many high-energy solutions by using Fountain Theorem and got the existence of at least one sign-changing solution by the method of invariant sets of descending flow.

Very recently, Che and Wu [7] studied a class of Kirchhoff equation with steep potential well and concave–convex nonlinearities as follows:

$$\begin{aligned} -\left( a+b\int _{{\mathbb {R}}^N}|\nabla u|^{2}\,\mathrm{d}x\right) \triangle u+\lambda V(x)u=a(x)|u|^{q-2}u+b(x)|u|^{p-2}u \quad \mathrm {in }\ {\mathbb {R}}^N, \end{aligned}$$

where $N\ge 3$, $1<q<2<p<\mathrm {min}\{4,2^{*}\}$, $a(x)\in L^{\frac{p}{p-q}}({\mathbb {R}}^N,{\mathbb {R}}^{+})$ and $b(x)\in L^{\infty }({\mathbb {R}}^N, {\mathbb {R}})$. By combining the Ekeland variational principle and the filtration of Nehari manifold, they proved the multiplicity of positive solutions for the above problem when b is sufficiently small and $\lambda$ is large enough. Considering the same potential as $(V_1)-(V_3)$, Chen et al. [9] also studied the following Kirchhoff-type equations:

$$\begin{aligned}&-\left( a+b\int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x\right) \triangle u+\lambda V(x)u\nonumber \\&\quad =a(x)|u|^{q-2}u+b(x)|u|^{p-2}u,\quad x\in {\mathbb {R}}^3, \end{aligned}$$

(1.7)

where $\lambda >0$, $1<q<2<p<4$, $a(x)\in L^{\frac{p}{p-q}}({\mathbb {R}}^3)$ and $b(x)\in L^{\infty }({\mathbb {R}}^3)$. They proved that the above problem admits at least one positive energy solution and a negative energy solution via the truncation functional and Ekeland variational principle.

The existence and multiplicity of solutions to the Kirchhoff equations have been extensively studied over the past few decades. However, there are relatively few papers that consider the case that the nonlinearity contains the concave–convex terms at the same time. Moreover, the nonlinear term arising in these problems was always assumed to be superlinear or sublinear; little has been done in the literature on problem (${\mathcal {K}}_{b,\lambda }$) with a more general nonlinearity involving a combination of concave and convex terms. Motivated by the above works, the following questions appear naturally:

($Q_1$):

if the nonlinearity is a more general nonlinearity involving a combination of concave and convex terms, will the problem (${\mathcal {K}}_{b,\lambda }$) with steep potential well admit two nontrivial solutions?

($Q_2$):

compare with Shao and Mao [28], if function f is super-quadratic at infinity, will problem (${\mathcal {K}}_{b,\lambda }$) possess a solution?

The aim of this paper is to consider the existence and asymptotic behavior of nontrivial solutions for Kirchhoff-type equation (${\mathcal {K}}_{b,\lambda }$) with steep potential well and more general concave–convex nonlinearities. To state our main results, we make the following assumptions:

($f_1$):

$f\in {\mathcal {C}}({\mathbb {R}}^3\times {\mathbb {R}},{\mathbb {R}})$ and there exist constants $C_0>0$ and $p\in (2,6)$ such that

$$\begin{aligned} |f(x,u)|\le C_0(1+|u|^{p-1}); \end{aligned}$$

($f_2$):

there exists constant $\nu >2$ such that

$$\begin{aligned} 0<\nu F(x,u)\le f(x,u)u,\ \forall x\in {\mathbb {R}}^3,\ \forall u\in {\mathbb {R}}\setminus \{0\}, \end{aligned}$$

where $F(x,u):=\int _{0}^{u}f(x,t)\,\mathrm{d}t$;

($f_3$):

$f(x,u)=o(|u|)$ as $|u|\rightarrow 0$ uniformly for $x\in {\mathbb {R}}^3$;

($g_1$):

$g\in {\mathcal {C}}({\mathbb {R}}^3\times {\mathbb {R}},{\mathbb {R}})$ and there exist constant $1<q<2$ and function $h(x)\in L^{\frac{2}{2-q}}({\mathbb {R}}^3,{\mathbb {R}}^+)$ such that

$$\begin{aligned} |g(x,u)|\le h(x)|u|^{q-1}; \end{aligned}$$

($g_2$):

there exists $\theta \in (1,2)$ such that

$$\begin{aligned} 0 < \frac{1}{\theta }u g(x,u)\le G(x,u):=\int _{0}^{u}g(x,t)\,\mathrm{d}t,\ \forall x\in {\mathbb {R}}^3, \ u\in {\mathbb {R}}\setminus \{0\}. \end{aligned}$$

In this paper, comparing with problem (1.6), we are no longer dealing with Kirchhoff-type problems on bounded domain. Furthermore, unlike the work described in [28], we will investigate the existence of nontrivial solutions to problem (${\mathcal {K}}_{b,\lambda }$) when the condition ($f_2$) is satisfied. Obviously, ($f_2$) is weaker than the condition (${\tilde{f}}_3$). Recall that $(f_2)$ is the super-quadratic condition, if $\nu > 2$, f may not be 4-superlinear at infinity. Due to the effect of the nonlocal term, if we apply the Mountain Pass Theorem directly to the energy functional, $(PS)_c$ condition or $(C)_c$ condition of the corresponding functional is very difficult to be proved by a standard argument.

In addition, since the corresponding energy functional of (${\mathcal {K}}_{b,\lambda }$) is not bounded below on both Nehari manifold and Nehari–Pohozǎev manifold, the method in [7, 24, 31] is no longer appropriate to deal with this problem. It leads us to seek for new methods to deal with the kinds of problems. Inspired by Zhang and Du [36], to surmount the difficulty, we use the truncation technique and the parameter-dependent compactness lemma to prove that the boundedness of Cerami sequence and later the existence of the positive energy solution of (${\mathcal {K}}_{b,\lambda }$) can be proved. After that, we study the asymptotic behavior of the positive energy solution of (${\mathcal {K}}_{b,\lambda }$). Furthermore, since the nonlinearity $g(x,u)+f(x,u)$ may involve a combination of concave and convex terms, we finally investigate the existence of the negative energy solution via the Ekeland variational principle.

2 Variational Settings and Main Results

In this paper, we make use of the following notations:

$\bullet$ |M| is the Lebesgue measure of the set M.

$\bullet$ $X^{'}$ denotes the dual space of X.

$\bullet$ the weak convergence is denoted by $\rightharpoonup$, and the strong convergence is denoted by $\rightarrow$.

$\bullet$ S is the best Sobolev constant for the embedding of $D^{1,2}({\mathbb {R}}^3)$ in $L^{6}({\mathbb {R}}^3)$.

$\bullet$ C and $C_i(i=1,2,\ldots )$ denote various positive constants.

In this section, we establish the variational framework of the Equation (${\mathcal {K}}_{b,\lambda }$) as elaborated by Ding and Szulkin [16] and give some useful preliminary results. Firstly, we give the definition of some spaces. As usual, for $1\le s< +\infty$, we let

$$\begin{aligned} |u|_s =\left( \int _{{\mathbb {R}}^3}|u|^s\,\mathrm{d}x\right) ^\frac{1}{s},\quad \forall u\in L^{s}({\mathbb {R}}^3). \end{aligned}$$

Let

$$\begin{aligned} H^1({\mathbb {R}}^3)=\{u\in L^{2}({\mathbb {R}}^3):\nabla u\in L^{2}({\mathbb {R}}^3)\} \end{aligned}$$

with the inner product and norm

$$\begin{aligned} (u,v)_{H^1}=\int _{{\mathbb {R}}^3}(\nabla u \nabla v+uv)\,\mathrm{d}x,\quad \Vert u\Vert _{H^1}=(u,u)_{H^1}^{1/2}. \end{aligned}$$

Throughout this paper, we work in the following Hilbert space

$$\begin{aligned} E=\left\{ u\in H^1({\mathbb {R}}^3):\int _{{\mathbb {R}}^3}V(x)u^2\,\mathrm{d}x<\infty \right\} , \end{aligned}$$

which is equipped with the inner product and norm

$$\begin{aligned} (u,v)=\int _{{\mathbb {R}}^3}(a\nabla u \nabla v+ V(x)uv)\,\mathrm{d}x,\quad \Vert u\Vert =(u,u)^{1/2}, \end{aligned}$$

where $a>0$ is from (${\mathcal {K}}_{b,\lambda }$). For $\lambda >0$, we also need the following inner product and norm

$$\begin{aligned} (u,v)_{\lambda }=\int _{{\mathbb {R}}^3}(a\nabla u \nabla v+\lambda V(x)uv)\,\mathrm{d}x,\quad \Vert u\Vert _{\lambda }=(u,u)_{\lambda }^{1/2}. \end{aligned}$$

It is clear that $\Vert u\Vert \le \Vert u\Vert _{\lambda }$ for $\lambda \ge 1$. Set $E_{\lambda }=(E,\Vert \cdot \Vert _{\lambda })$, then we have the following lemma.

Lemma 2.1

Under the conditions $(V_{1})-(V_{2})$, the embedding $E_{\lambda } \hookrightarrow L^s({\mathbb {R}}^3)$ is continuous for $\lambda \ge 1$ and $2\le s\le 6$. Hence, there exists $d_s>0$ (independent of $\lambda \ge 1$) such that

$$\begin{aligned} |u|_s \le d_{s}\Vert u\Vert \le d_{s}\Vert u\Vert _{\lambda }, \quad \forall u\in E. \end{aligned}$$

(2.1)

Proof

From ($V_1$) and ($V_2$), we get

$$\begin{aligned} \int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x+|u|^2\,\mathrm{d}x&=\int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x+\int _{{\mathcal {V}}_{c}}|u|^2\,\mathrm{d}x+\int _{{\mathbb {R}}^3\setminus {\mathcal {V}}_{c}}|u|^2\,\mathrm{d}x\nonumber \\&\le \int _{{\mathbb {R}}^3}|\nabla u|^2+|{\mathcal {V}}_{c}|^\frac{2}{3}\left( \int _{{\mathcal {V}}_{c}}|u|^6\,\mathrm{d}x\right) ^\frac{1}{3}+c^{-1} \int _{{\mathbb {R}}^3\setminus {\mathcal {V}}_{c}}V(x)u^2\,\mathrm{d}x\nonumber \\&\le \mathrm {max}\{a^{-1}+a^{-1}|{\mathcal {V}}_{c}|^\frac{2}{3}S^{-1}, c^{-1}\} \int _{{\mathbb {R}}^3}(a|\nabla u|^2+V(x)|u|^2)\,\mathrm{d}x. \end{aligned}$$

(2.2)

It implies that $E\hookrightarrow H^{1}({\mathbb {R}}^3)$ is continuous. By (2.2), Hölder and Sobolev inequalities, as $\lambda \ge 1$, there exists $C_S$ (independent of $\lambda$) such that

$$\begin{aligned} \int _{{\mathbb {R}}^3}|u|^s\,\mathrm{d}x \le \left( \int _{{\mathbb {R}}^3}|u|^2\,\mathrm{d}x\right) ^{\frac{6-s}{4}}\left( \int _{{\mathbb {R}}^3}|u|^6\,\mathrm{d}x\right) ^{\frac{s-2}{4}} \le C_{S}S^\frac{3(2-s)}{4}\Vert u\Vert _{\lambda }^s \end{aligned}$$

for any $s\in [2,6]$. Thus, for each $s \in [2,6]$, there exists $d_s>0$ (independent of $\lambda \ge 1$) such that

$$\begin{aligned} |u|_s \le d_{s}\Vert u\Vert \le d_{s}\Vert u\Vert _{\lambda } \quad \mathrm {for}\,\,u\in E. \end{aligned}$$

Therefore, the embedding $E_{\lambda } \hookrightarrow L^s({\mathbb {R}}^3)$ is continuous. $\square$

Lemma 2.2

Assume that $(V_1)$–$(V_2)$ and ($g_1$) hold. Define $\psi (u):=\int _{{\mathbb {R}}^3}G(x,u)\,\mathrm{d}x$. Then,

• (i) $\langle \psi ^{'}(u),v\rangle =\int _{{\mathbb {R}}^3}g(x,u)v\,\mathrm{d}x$, $\forall u, v\in E_{\lambda }$;

• (ii) $\psi ^{'}:E_{\lambda }\rightarrow E_{\lambda }^{'}$ is weakly continuous.

Proof

The proof is similar to Xu and Chen [35]. For the reader’s convenience, we give the completed proof here. We will use the following inequalities:

$$\begin{aligned} {\left\{ \begin{array}{ll} |a+b|^{n}\le 2^{n-1}(|a|^{n}+|b|^{n}),&{}\quad 1\le n<\infty ,\\ |a+b|^{n}\le 2^{n}(|a|^{n}+|b|^{n}), &{}\quad 0<n<1. \end{array}\right. } \end{aligned}$$

(2.3)

(i) By ($g_1$), Lemma 2.1 and Hölder’s inequality, we have

$$\begin{aligned} \int _{{\mathbb {R}}^3}|G(x,u)|\,\mathrm{d}x\le \frac{1}{q}\int _{{\mathbb {R}}^3}|h(x)||u|^{q}\,\mathrm{d}x\le \frac{d_2^{q}}{q}|h|_\frac{2}{2-q}\Vert u\Vert _{\lambda }^{q}. \end{aligned}$$

(2.4)

Next, we prove $\langle \psi ^{'}(u),v\rangle =\int _{{\mathbb {R}}^3}g(x,u)v\,\mathrm{d}x$ by definition. By ($g_1$), (2.3), Lemma 2.1 and Hölder’s inequality, for all $u,v\in E_{\lambda }$ and $t\in [0,1]$, we get

$$\begin{aligned} \left| \int _{{\mathbb {R}}^3}g(x,u+tv)v\,\mathrm{d}x\right|&\le \int _{{\mathbb {R}}^3}(h(x)|u+tv|^{q-1})|v|\,\mathrm{d}x\\&\le \int _{{\mathbb {R}}^3}2^{q-1}h(x)(|u|^{q-1}+|v|^{q-1})|v|\,\mathrm{d}x\\&\le 2^{q-1}|h|_{\frac{2}{2-q}}(|u|_2^{q-1}+|v|_2^{q-1})|v|_2\\&<+\infty . \end{aligned}$$

Therefore, by Lebesgue’s Dominated Theorem, for all $u,v\in E_{\lambda }$ and $\xi \in (0,1)$, we obtain

$$\begin{aligned} \langle \psi ^{'}(u),v\rangle&=\lim \limits _{t\rightarrow 0}\frac{\psi (u+tv)-\psi (u)}{t}\\&=\lim \limits _{t\rightarrow 0}\int _{{\mathbb {R}}^3}\frac{G(x,u+tv)-G(x,u)}{t}\,\mathrm{d}x\\&=\lim \limits _{t\rightarrow 0}\int _{{\mathbb {R}}^3}g(x,u+\xi tv)v\,\mathrm{d}x\\&=\int _{{\mathbb {R}}^3} g(x,u)v\,\mathrm{d}x. \end{aligned}$$

(ii) Applying ($g_1$), Lemma 2.1 and Hölder’s inequality, for all $u,v\in E_{\lambda }$, we have

$$\begin{aligned} |\langle \psi ^{'}(u),v\rangle |&\le \int _{{\mathbb {R}}^3}|g(x,u)v|\,\mathrm{d}x\\&\le \int _{{\mathbb {R}}^3}h(x)|u|^{q-1}|v|\,\mathrm{d}x\\&\le d_{2}^{q+1}|h|_{\frac{2}{2-q}}\Vert u\Vert _{\lambda }^{q-1}\Vert v\Vert _{\lambda }, \end{aligned}$$

which implies that $\psi ^{'}\in E_{\lambda }^{'}$. We now prove that $\psi ^{'}: E_{\lambda }\rightarrow E_{\lambda }^{'}$ is weakly continuous. Let $u_n\rightharpoonup u$ in $E_{\lambda }$, then there exists $M_0>0$ such that

$$\begin{aligned} \Vert u_n\Vert _{\lambda }\le M_0 \quad \mathrm {and} \quad \Vert u\Vert _{\lambda }\le M_0. \end{aligned}$$

(2.5)

We claim that

$$\begin{aligned} \int _{{\mathbb {R}}^3}|g(x,u_n)-g(x,u)|^{2}\,\mathrm{d}x\rightarrow 0\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

(2.6)

On the one hand, since $h(x)\in L^{\frac{2}{2-q}}({\mathbb {R}}^3,{\mathbb {R}}^{+})$, for every $\varepsilon >0$, there exists $\tau _{\varepsilon }>0$ such that

$$\begin{aligned} \left( \int _{|x|\ge \tau _{\varepsilon }}|h(x)|^{\frac{2}{2-q}}\,\mathrm{d}x\right) ^{\frac{2-q}{2}}\le \sqrt{\varepsilon }. \end{aligned}$$

Combining this with ($g_1$), (2.3), (2.5) and Hölder’s inequality, we have

$$\begin{aligned} \int _{|x|\ge \tau _{\varepsilon }}|g(x,u_n)-g(x,u)|^{2}\,\mathrm{d}x\le & {} 4\int _{|x|\ge \tau _{\varepsilon }}h^{2}(x)(|u_n|^{2(q-1)}+|u|^{2(q-1)})\,\mathrm{d}x\nonumber \\\le & {} 4\left( \int _{|x|\ge \tau _{\varepsilon }}|h(x)|^{\frac{2}{2-q}}\,\mathrm{d}x\right) ^{2-q}(|u_n|_2^{2(q-1)}+|u|_2^{2(q-1)})\nonumber \\\le & {} 4\varepsilon (|u_n|_2^{2(q-1)}+|u|_2^{2(q-1)})\nonumber \\\le & {} 4\varepsilon d_2^{2(q-1)}(\Vert u_n\Vert _{\lambda }^{2(q-1)}+\Vert u\Vert _{\lambda }^{2(q-1)})\nonumber \\\le & {} 8\varepsilon d_2^{2(q-1)}M_0^{2(q-1)}. \end{aligned}$$

(2.7)

On the other hand, we only need to prove

$$\begin{aligned} \int _{|x|\le \tau _{\varepsilon }}(g(x,u_{n})-g(x,u))^{2}\,\mathrm{d}x \rightarrow 0,\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

(2.8)

In fact, since $u_n\rightharpoonup u$ in $E_{\lambda }$, up to subsequence, there exists a subsequence $\{u_{n_k}\}_{k\in {\mathbb {N}}}$ such that

$$\begin{aligned} u_{n_k}\rightarrow u\ \mathrm {in}\ L_{loc}^{2}({\mathbb {R}}^3)\quad \mathrm {and} \quad u_{n_{k}}(x)\rightarrow u(x)\quad \mathrm {a.e.}\ x\in {\mathbb {R}}^3\ \mathrm {as}\ k\rightarrow \infty . \end{aligned}$$

(2.9)

Arguing by contradiction, we assume that there exists constant $\varepsilon _{0}>0$ such that

$$\begin{aligned} \int _{|x|\le \tau _{\varepsilon }}(g(x,u_{n_k})-g(x,u))^{2}\,\mathrm{d}x \ge \varepsilon _{0},\quad \forall k\in {\mathbb {N}}. \end{aligned}$$

(2.10)

By (2.9) and Theorem A.1 in [33], passing to a subsequence if necessary, we can assume that

$$\begin{aligned} \sum _{k=1}^{\infty }\int _{|x|\le \tau _{\varepsilon }}|u_{n_k}-u|^2\,\mathrm{d}x<+\infty . \end{aligned}$$

Set $\omega (x)=\left( \sum \nolimits _{k=1}^{\infty }|u_{n_k}(x)-u(x)|^{2}\right) ^{\frac{1}{2}}$ for $|x|\le \tau _{\varepsilon }$, then $\int _{|x|\le \tau _{\varepsilon }}\omega ^{2}\,\mathrm{d}x<+\infty$. Applying ($g_1$) and (2.3), $\forall k\in {\mathbb {N}}$ and $|x|\le \tau _{\varepsilon }$, we have

$$\begin{aligned} |g(x,u_{n_k})-g(x,u)|^{2}&\le 4h^{2}(x)(|u_{n_k}|^{2q-2}+|u|^{2q-2})\\&\le 2^{2q+1}|h(x)|^{2}(|u_{n_k}-u|^{2q-2}+|u|^{2q-2})\\&\le 2^{2q+1}|h(x)|^{2}(|\omega |^{2q-2}+|u|^{2q-2}), \end{aligned}$$

and using Hölder’s inequality, one has

$$\begin{aligned}&\int _{|x|\le \tau _{\varepsilon }}2^{2q+1}|h(x)|^{2}(|\omega |^{2q-2}+|u|^{2q-2})\,\mathrm{d}x\\&\quad \le 2^{2q+1}|h|_{\frac{2}{2-q}}^{2}\left[ \left( \int _{|x|\le \tau _{\varepsilon }}|\omega |^{2}\,\mathrm{d}x\right) ^{q-1} +\left( \int _{|x|\le \tau _{\varepsilon }}|u|^{2}\,\mathrm{d}x\right) ^{q-1}\right] \\&\quad <+\infty . \end{aligned}$$

Thus, by Lebesgue’s Dominated Convergence Theorem, we obtain

$$\begin{aligned} \int _{|x|\le \tau _{\varepsilon }}|g(x,u_{n_k})-g(x,u)|^{2}\,\mathrm{d}x\rightarrow 0\quad \mathrm {as}\ k\rightarrow \infty . \end{aligned}$$

This contradicts (2.10) and so (2.8) holds. Equation (2.7) and (2.8) shows that claim (2.6) is true. Hence, it follows from (2.6) and Hölder’s inequality that

$$\begin{aligned} \langle \psi ^{'}(u_n)-\psi ^{'}(u),v\rangle&=\int _{{\mathbb {R}}^3}(g(x,u_n)-g(x,u))v\,\mathrm{d}x\\&\le \left( \int _{{\mathbb {R}}^3}|g(x,u_n)-g(x,u)|^2\,\mathrm{d}x\right) ^{\frac{1}{2}}|v|_2\\&\rightarrow 0\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

Thus, $\psi ^{'}$ is weakly continuous, and then, $\psi ^{'}$ is continuous, i.e., $\psi \in {\mathcal {C}}^{1}(E_{\lambda },{\mathbb {R}})$. The proof is completed. $\square$

As a consequence, the Euler–Lagrange energy functional $I_{b,\lambda }:E_{\lambda }\rightarrow {\mathbb {R}}$ associated with Eq. (${\mathcal {K}}_{b,\lambda }$) given by

$$\begin{aligned} I_{b,\lambda }(u)= & {} \frac{1}{2}\displaystyle {\int }_{{\mathbb {R}}^3}(a|\nabla u|^2 + \lambda V(x) u^2)\,\mathrm{d}x+ \frac{b}{4}\left( \displaystyle {\int }_{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x\right) ^2\\&-\,\int _{{\mathbb {R}}^3}G(x,u)\,\mathrm{d}x-\int _{{\mathbb {R}}^3}F(x,u)\,\mathrm{d}x \end{aligned}$$

is well defined and $I_{b,\lambda }\in {\mathcal {C}}^{1}(E_{\lambda },{\mathbb {R}})$. Furthermore, for any $u,v \in E_{\lambda }$, there holds

$$\begin{aligned} \langle I_{b,\lambda }^{'}(u),v \rangle= & {} \left( a+b\int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x\right) \int _{{\mathbb {R}}^3}\nabla u \nabla v\,\mathrm{d}x\nonumber \\&+\,\int _{{\mathbb {R}}^3}\lambda V(x)uv\,\mathrm{d}x-\int _{{\mathbb {R}}^3}g(x,u)v\,\mathrm{d}x-\int _{{\mathbb {R}}^3}f(x,u)v\,\mathrm{d}x. \end{aligned}$$

It is standard to verify that the weak solutions of Equation (${\mathcal {K}}_{b,\lambda }$) correspond to the critical points of the functional $I_{b,\lambda }$.

For our convenience, without loss of generality, we need to assume that $a=1$ in Theorem 2.1–2.5. At first, we establish the existence of the positive energy solutions of Equation (${\mathcal {K}}_{b,\lambda }$) in this paper.

Theorem 2.1

Suppose that $(V_{1})-(V_{3})$, $(f_{1})-(f_{3})$ and $(g_{1})-(g_{2})$ are satisfied. Then, there exist $T>0$, $b_{0}>0$, $b_{T}>0$ and $\lambda ^{*}>0$ such that for each $b \in (0,min\{b_{0}, b_{T}\})$ and $\lambda \in (\lambda ^{*},\infty )$, problem (${\mathcal {K}}_{b,\lambda }$) has at least a nontrivial solution $u_{b,\lambda }^{+}$ in $E_{\lambda }$ when

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^{q}}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}}. \end{aligned}$$

Moreover, $u_{b,\lambda }^{+}$ satisfies

$$\begin{aligned} 0 < \Vert u_{b,\lambda }^{+}\Vert _{\lambda } \le T \quad and \quad I_{b,\lambda }(u_{b,\lambda }^{+})>0. \end{aligned}$$

Next, we give the asymptotic behavior of the positive energy solution $u_{b,\lambda }^{+}$ obtained by Theorem 2.1 as $b\rightarrow 0$ and $\lambda \rightarrow +\infty$.

Theorem 2.2

Suppose that $(V_{1})-(V_{3})$, $(f_{1})-(f_{3})$ and $(g_{1})-(g_{2})$ are satisfied. If $u_{b,\lambda }^{+}$ is a nontrivial solution of problem (${\mathcal {K}}_{b,\lambda }$) obtained by Theorem 2.1, for each $b\in (0, min\{b_{0},b_T\})$ and any sequence $\{\lambda _{n}\}\subset (\lambda ^{*}, \infty )$, then $u_{b,\lambda _n}^{+}\rightarrow u_{b}^{+}$ in E as $\lambda _{n}\rightarrow +\infty$, where $u_{b}^{+}\in H_{0}^1(\Omega )$ is a nontrivial solution of

Theorem 2.3

Suppose that $(V_{1})-(V_{3})$, $(f_{1})-(f_{3})$ and $(g_{1})-(g_{2})$ are satisfied. If $u_{b,\lambda }^{+}$ is a nontrivial solution of problem (${\mathcal {K}}_{b,\lambda }$) obtained by Theorem 2.1, for each $\lambda \in (\lambda ^{*},\infty )$ and any sequence $\{b_n\}\subset (0,min\{b_{0},b_T\})$, then $u_{b_n,\lambda }^{+}\rightarrow u_{\lambda }^{+}$ in $E_{\lambda }$ as $b_n\rightarrow 0$, where $u_{\lambda }^{+}\in E_{\lambda }$ is a nontrivial solution of

Theorem 2.4

Suppose that $(V_{1})-(V_{3})$, $(f_{1})-(f_{3})$ and $(g_{1})-(g_{2})$ are satisfied. If $u_{b,\lambda }^{+}$ is a nontrivial solution of (${\mathcal {K}}_{b,\lambda }$) obtained by Theorem 2.1. Then, $u_{b, \lambda }^{+}\rightarrow u_0^{+}$ in $H^1({\mathbb {R}}^3)$ as $b\rightarrow 0$ and $\lambda \rightarrow +\infty$ up to subsequence, where $u_0^{+} \in H_{0}^{1}(\Omega )$ is a nontrivial solution of

Finally, we give the existence of the negative energy solutions of Equation ${\mathcal {K}}_{b,\lambda }$.

Theorem 2.5

Suppose that $(V_{1})-(V_{3})$, $(f_{1})-(f_{3})$ and $(g_{1})-(g_{2})$ are satisfied. Then, there exist $\rho '>0$ and $\lambda _{*}>0$ such that for all $\lambda >\lambda _{*}$ and

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^{q}}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}}, \end{aligned}$$

problem (${\mathcal {K}}_{b,\lambda }$) has at least a nontrivial solution $u_{b,\lambda }^{-} \in E_{\lambda }$ satisfying

$$\begin{aligned} 0<\Vert u_{b,\lambda }^{-}\Vert _{\lambda }\le \rho ' \quad and \quad I_{b,\lambda }(u_{b,\lambda }^{-})<0. \end{aligned}$$

This paper is organized as follows. In Sect. 3, we present some preliminary results on a truncated functional. In Sect. 4, we give the existence of the positive energy solution. Furthermore, we complete the proofs of Theorem 2.2–2.4 in Sect. 5. Finally, the existence of the negative energy solution is studied in Sect. 6.

3 Some Results on Truncated Functional

In this section, we will establish some properties of a truncated functional. It is to overcome the difficulty of finding bounded Palais–Smale sequences or Cerami sequences for the associated function $I_{b,\lambda }$. Based on this, we need to give the following definitions. As Zhang and Du [36], let $\eta \in {\mathcal {C}}^{1}([0,\infty ),{\mathbb {R}})$ be a cutoff function and it satisfies $0\le \eta \le 1$ and

$$\begin{aligned} \left\{ \begin{array}{rcl} \eta (t)=1,&{}&{}{0\le t\le 1,}\\ \eta (t)=0,&{}&{}{t\ge 2,}\\ \max \limits _{t>0}|\eta ^{'}(t)|\le 2,&{}&{}{t> 0,}\\ \eta ^{'}(t)\le 0,&{}&{}{t>0.} \end{array}\right. \end{aligned}$$

Using $\eta$, for any $T>0$, we move to study the truncated functional $I_{b,\lambda }^T:E_{\lambda } \rightarrow {\mathbb {R}}$ defined by

$$\begin{aligned} I_{b,\lambda }^{T}(u)= & {} \frac{1}{2}\displaystyle {\int }_{{\mathbb {R}}^3}(|\nabla u|^2 + \lambda V(x) u^2)\,\mathrm{d}x+ \frac{b}{4}\eta \left( \frac{\Vert u\Vert _{\lambda }^2}{T^2}\right) \left( \displaystyle {\int }_{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x\right) ^2\nonumber \\&-\,\int _{{\mathbb {R}}^3}G(x,u)\,\mathrm{d}x-\int _{{\mathbb {R}}^3}F(x,u)\,\mathrm{d}x, \end{aligned}$$

(3.1)

where $\eta$ is a smooth cutoff function such that

$$\begin{aligned} \eta \left( \frac{\Vert u\Vert _{\lambda }^2}{T^2}\right) =\left\{ \begin{array}{rcl} 1, &{} &{} {\Vert u\Vert _{\lambda } \le T,}\\ 0, &{} &{} {\Vert u\Vert _{\lambda } \ge \sqrt{2} T.} \end{array}\right. \end{aligned}$$

According to $\eta \in {\mathcal {C}}^{1}([0,\infty ),{\mathbb {R}})$ and $I_{b,\lambda }\in {\mathcal {C}}^{1}(E_{\lambda },{\mathbb {R}})$, it is easy to infer that $I_{b,\lambda }^T$ is of class ${\mathcal {C}}^1$ by a standard argument. Moreover, for any $u,v \in E_{\lambda }$, we have

$$\begin{aligned} \langle (I_{b,\lambda }^{T})^{'}(u),v\rangle= & {} (u,v)_{\lambda }+b\eta \left( \frac{\Vert u\Vert _{\lambda }^2}{T^2}\right) |\nabla u|_{2}^{2}\int _{{\mathbb {R}}^3}\nabla u\nabla v\,\mathrm{d}x\nonumber \\&+\,\frac{b}{2T^2}\eta ^{'}\left( \frac{\Vert u\Vert _{\lambda }^2}{T^2}\right) (u,v)_{\lambda }|\nabla u|_{2}^4\nonumber \\&-\,\int _{{\mathbb {R}}^3}g(x,u)v\,\mathrm{d}x-\int _{{\mathbb {R}}^3}f(x,u)v\,\mathrm{d}x. \end{aligned}$$

(3.2)

With this penalization, by choosing an appropriate $T>0$ and constraining $b>0$ sufficiently small, we may obtain a Cerami sequence $\{u_n\}$ of $I_{b,\lambda }^T$ satisfying $\Vert u_n\Vert _{\lambda } \le T$. According to the definition of $\eta$, $\{u_n\}$ is also a Cerami sequence of $I_{b,\lambda }$ when $\Vert u_n\Vert _{\lambda } \le T$.

To obtain the Cerami sequence, we first show that the truncated function $I_{b,\lambda }^{T}$ satisfies the mountain pass geometry.

Lemma 3.1

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, for each $T,b>0$ and $\lambda \ge 1$, there exist $\alpha ,~\rho >0$ (independent of T, b and $\lambda$) such that for all

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^q}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}}, \end{aligned}$$

and $\Vert u\Vert _{\lambda }=\rho$,

$$\begin{aligned} I_{b,\lambda }^{T}(u)\ge \alpha . \end{aligned}$$

Proof

By $(f_1)$ and $(f_3)$, for every $\varepsilon >0$, there exists $C_\varepsilon >0$ such that

$$\begin{aligned} |f(x,u)|\le \varepsilon |u|+C_\varepsilon |u|^{p-1},\quad \forall (x,u)\in {\mathbb {R}}^3\times {\mathbb {R}}. \end{aligned}$$

(3.3)

Let $\varepsilon = \frac{1}{2d_2^2}$, where $d_2>0$ is from (2.1). Then, for each $u \in E_{\lambda }$, by (2.4) and (3.3), we have

$$\begin{aligned} I_{b,\lambda }^{T}(u)&=\frac{1}{2}\Vert u\Vert _{\lambda }^2+\frac{b}{4}\eta \left( \frac{\Vert u\Vert _{\lambda }^2}{T^2}\right) |\nabla u|_2^4-\int _{{\mathbb {R}}^3}G(x,u)\,\mathrm{d}x-\int _{{\mathbb {R}}^3}F(x,u)\,\mathrm{d}x\\&\ge \frac{1}{2}\Vert u\Vert _{\lambda }^{2}-\frac{d_2^q}{q}|h|_{\frac{2}{2-q}}\Vert u\Vert _{\lambda }^{q}-\frac{\varepsilon }{2}|u|_2^2-\frac{C_{\varepsilon }}{p}|u|_p^p\\&\ge \frac{1}{4}\Vert u\Vert _{\lambda }^{2}-\frac{d_2^q}{q}|h|_{\frac{2}{2-q}}\Vert u\Vert _{\lambda }^{q}-\frac{C_{1/{2d_{2}^{2}}}d_{p}^{p}}{p}\Vert u\Vert _{\lambda }^p\nonumber \\&=\Vert u\Vert _{\lambda }^{2}\left( \frac{1}{4}-\frac{d_2^q}{q}|h|_{\frac{2}{2-q}}\Vert u\Vert _{\lambda }^{q-2}-\frac{C_{1/{2d_{2}^{2}}}d_{p}^{p}}{p}\Vert u\Vert _{\lambda }^{p-2}\right) \nonumber \end{aligned}$$

where the constants $d_p>0$ and $C_{1/{2d_{2}^{2}}}>0$ are independent of T, b and $\lambda$.

Define

$$\begin{aligned} Q(t):=\frac{C_{1/{2d_{2}^{2}}}d_p^{p}}{p}t^{p-2}+\frac{d_2^q}{q}|h|_{\frac{2}{2-q}}t^{q-2},\quad t\ge 0, \end{aligned}$$

and then we get

$$\begin{aligned} \lim \limits _{t\rightarrow +\infty }Q(t)=\lim \limits _{t\rightarrow 0^{+}}Q(t)=+\infty , \end{aligned}$$

which implies that Q(t) is bounded below. Thus, Q(t) admits a unique minimizer $t_0$:

$$\begin{aligned} t_0=\left( \frac{d_{2}^{q}|h|_\frac{2}{2-q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^{\frac{1}{p-q}}. \end{aligned}$$

By the definition of Q(t), one has

$$\begin{aligned}&\inf \limits _{t\in [0,+\infty )}Q(t)\\&\quad =Q(t_0)\\&\quad =\frac{d_2^{q}}{q}|h|_{\frac{2}{2-q}}\left( \frac{d_{2}^{q}|h|_\frac{2}{2-q} p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{d_p^{p}C_{1/{2d_{2}^{2}}}}{p} \left( \frac{d_{2}^{q}|h|_\frac{2}{2-q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\\&\quad =(|h|_{\frac{2}{2-q}})^{\frac{p-2}{p-q}}\left( \frac{d_2^{q}}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) \\&\quad >0. \end{aligned}$$

Thus, we can choose

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^{q}}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}} \end{aligned}$$

and $\Vert u\Vert _{\lambda }=\rho =t_0>0$ such that

$$\begin{aligned} I_{b,\lambda }^{T}(u)\ge \rho ^{2}(\frac{1}{4}-Q(\rho ))\ge \frac{\rho ^{2}}{8}:=\alpha >0. \end{aligned}$$

$\square$

Lemma 3.2

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, there exists $b_{0}>0$ such that for each T, $\lambda >0$ and $b \in (0,b_{0})$, we have $I_{b,\lambda }^{T}(e)<0$ for some $e \in C_{0}^{\infty }(\Omega )$ with $|\nabla e|_2>\rho$.

Proof

We first define the functional ${\mathcal {J}}_{\lambda }: E_{\lambda }\rightarrow {\mathbb {R}}$ by

$$\begin{aligned} {\mathcal {J}}_{\lambda }(u)=\frac{1}{2}\int _{{\mathbb {R}}^3}(|\nabla u|^2 + \lambda V(x) u^2)\,\mathrm{d}x -\int _{{\mathbb {R}}^3}G(x,u)\,\mathrm{d}x-\int _{{\mathbb {R}}^3}F(x,u)\,\mathrm{d}x. \end{aligned}$$

From ($f_1$)-($f_3$), there exist $C_1,C_2> 0$ such that

$$\begin{aligned} F(x,u)\ge C_{1}|u|^{\nu }-C_{2}|u|^2,\quad \forall (x,u)\in {\mathbb {R}}^3\times {\mathbb {R}}. \end{aligned}$$

(3.4)

By virtue of ($g_2$) and (3.4), if we choose a positive smooth function $\varphi \in C_0^{\infty }(\Omega )$, we have

$$\begin{aligned} {\mathcal {J}}_{\lambda }(t\varphi )&=\frac{t^2}{2}\int _{\Omega }|\nabla \varphi |^2\,\mathrm{d}x -\int _{\Omega }G(x,t\varphi )\,\mathrm{d}x-\int _{\Omega }F(x,t\varphi )\,\mathrm{d}x \\&\le \frac{t^2}{2}\int _{\Omega }|\nabla \varphi |^2\,\mathrm{d}x+C_{1}t^2\int _{\Omega } |e|^2\,\mathrm{d}x-C_{2}t^{\nu }\int _{\Omega }|\varphi |^{\nu }\,\mathrm{d}x\\&\rightarrow -\infty \end{aligned}$$

as $t \rightarrow +\infty$ for $\nu >2$. Thus, there exist ${\tilde{t}}>0$ large enough and $e:={\tilde{t}}\varphi$ such that ${\mathcal {J}}_{\lambda }(e) \le -1$ with $|\nabla e|_2>\rho$ . Since

$$\begin{aligned} I_{b,\lambda }^T(e)={\mathcal {J}}_{\lambda }(e)+\frac{b}{4}\eta \left( \frac{\Vert e\Vert _{\lambda }^2}{T^2}\right) |\nabla e|_2^4 \le -1+\frac{b}{4}|\nabla e|_2^4, \end{aligned}$$

then there exists $b_{0}:=\frac{4}{|\nabla e|_2^4}>0$ (independent of $\lambda$ and T) such that $I_{b,\lambda }^T(e)<0$ for each T, $\lambda >0$ and $b \in (0,b_{0})$. The proof is finished. $\square$

Next, in order to prove Theorem 2.1, we shall a stronger version of the Mountain Pass Theorem in [18], which allows us to find Cerami sequences instead of Palais–Smale sequences.

Theorem 3.1

(See [18]) Let X be a real Banach space with its dual space $X^{'}$, and suppose that $J \in {\mathcal {C}}^1(X,{\mathbb {R}})$ satisfies

$$\begin{aligned} max\{J(0), J(e)\} \le \mu < \eta \le \inf \limits _{\Vert u\Vert _{X}=\rho } J(u) \end{aligned}$$

for some $\mu < \eta$, $\rho >0$ and $e \in X$ with $\Vert e\Vert _{X}>\rho$. Let $c\ge \eta$ be characterized by

$$\begin{aligned} c=\inf \limits _{\gamma \in \Gamma }\max \limits _{t \in [0,1]} J(\gamma (t)), \end{aligned}$$

where $\Gamma =\{\gamma \in {\mathcal {C}}([0,1],X):\gamma (0)=0,\,\gamma (1)=e\}$ is the set of continuous paths joining 0 and e. Then there exists a sequence $\{u_{n}\subset X\}$ such that

$J(u_n)\rightarrow c \ge \eta$    and    $(1+\Vert u_n\Vert _X)\Vert J^{'}(u_n)\Vert _{X^{'}}\rightarrow 0$   as $n\rightarrow \infty$.

By Lemmas 3.1 and 3.2, we now define the mountain pass value $c_{b,\lambda }^T$ of $I_{b,\lambda }^T$ by

$$\begin{aligned} c_{b,\lambda }^T=\inf \limits _{\gamma \in \Gamma }\max \limits _{t \in [0,1]}I_{b,\lambda }^T(\gamma (t)), \end{aligned}$$

where

$$\begin{aligned} \Gamma :=\{\gamma \in {\mathcal {C}}([0,1],E_\lambda ):\gamma (0)=0,\gamma (1)=e\}. \end{aligned}$$

Note from Lemma 3.2 that $\Gamma$ is nonempty.

Applying Lemmas 3.1, 3.2 and Theorem 3.1, we thus deduce that for any $T>0$, $\lambda \ge 1$,

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^q}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}} \end{aligned}$$

and $b \in (0, b_{0})$, there exists a Cerami sequence $\{u_n\}\subset E_\lambda$ such that

$$\begin{aligned} I_{b,\lambda }^T(u_n)\rightarrow c_{b,\lambda }^T \quad \mathrm {and} \quad (1+\Vert u_n\Vert _{\lambda })\parallel (I_{b,\lambda }^T)^{'}(u_n)\parallel _{E_{\lambda }^{'}}\rightarrow 0\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

(3.5)

Obviously, according to Lemma 3.1, $c_{b,\lambda }^T \ge \alpha > 0$. Next, we also give an estimate on the upper bound of the mountain pass value $c_{b,\lambda }^T$, which is the important part of the truncation technique.

Lemma 3.3

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, for every $T>0$, $\lambda \ge 1$ and $b\in (0,b_0)$, there exists $M>0$ (independent of T, b and $\lambda$) such that $c_{b, \lambda }^T\le M$.

Proof

By $(g_2)$, (3.4) and $e\in C_{0}^{\infty }(\Omega )$, we have

$$\begin{aligned} I_{b,\lambda }^T(te)&=\frac{t^2}{2}\int _{\Omega }|\nabla e|^2\,\mathrm{d}x+\frac{b}{4}t^4\eta \left( \frac{t^2\Vert e\Vert _{\lambda }^2}{T^2}\right) \nonumber \\&\left( \int _{\Omega }|\nabla e|^2\,\mathrm{d}x\right) ^2-\int _{\Omega }G(x,te)\,\mathrm{d}x-\int _{\Omega }F(x,te)\,\mathrm{d}x\\&\le \frac{t^2}{2}\int _{\Omega }|\nabla e|^2\,\mathrm{d}x+\frac{b_0}{4}t^4\left( \int _{\Omega }|\nabla e|^2\,\mathrm{d}x\right) ^2+C_{1}t^2 \int _{\Omega }|e|^2\,\mathrm{d}x-C_{2}t^{\nu }\int _{\Omega }|e|^{\nu }\,\mathrm{d}x. \end{aligned}$$

Thus, there exists a constant $M>0$ (independent of T, $\lambda$ and b) such that

$$\begin{aligned} c_{b,\lambda }^T\le \max \limits _{t\in [0,1]}I_{b,\lambda }^T(te_0)\le M. \end{aligned}$$

This competes the proof. $\square$

The following lemma shows that the Cerami sequence $\{u_n\}$ satisfies $\Vert u_n\Vert _{\lambda }\le T$, which is the key ingredient of this paper.

Lemma 3.4

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. If $\{u_n\}\subset E_{\lambda }$ is a sequence satisfying (3.5), then up to a subsequence, there exist $T>0$ and $b_{T}>0$ such that for each $\lambda \ge 1$ and $b \in (0,min\{b_{0},b_{T}\})$, there holds

$$\begin{aligned} \limsup \limits _{n\in {\mathbb {N}}}\Vert u_n\Vert _{\lambda }\le T. \end{aligned}$$

Proof

Suppose by contradiction, for any $T>0$, there exists a subsequence of $\{u_n\}$, still denoted by $\{u_n\}$, such that $\Vert u_n\Vert _{\lambda }> T$. Next, we divide the proof into two case:

(i) $\Vert u_n\Vert _{\lambda } > \sqrt{2}T;$

(ii) $T < \Vert u_n\Vert _{\lambda } \le \sqrt{2}T.$

If the case (i) holds, then by Lemma 3.3, (3.5) and ($f_2$) , for n large enough, we have that

$$\begin{aligned} M+1&\ge c_{b,\lambda }^T+1\nonumber \\&\ge I_{b,\lambda }^{T}(u_n)-\frac{1}{\nu }\langle (I_{b,\lambda }^{T})^{'}(u_n), u_n \rangle \\&=\left( \frac{1}{2}-\frac{1}{\nu }\right) \Vert u_n\Vert _{\lambda }^2-\left( \frac{b}{\nu }-\frac{b}{4}\right) \eta \left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) |\nabla u_n|_2^4-\frac{b}{2\nu T^2}\eta ^{'}\left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) |\nabla u_n|_2^{4}\Vert u_n\Vert _{\lambda }^2\\&\quad +\,\int _{{\mathbb {R}}^3}\left( \frac{1}{\nu }f(x,u_n)u_n-F(x,u_n)\right) \,\mathrm{d}x-\int _{{\mathbb {R}}^3}\left( G(x,u_n)-\frac{1}{\nu }g(x,u_n)u_n\right) \,\mathrm{d}x\\&\ge \left( \frac{1}{2}-\frac{1}{\nu }\right) \Vert u_n\Vert _{\lambda }^2-\int _{{\mathbb {R}}^3}\left( G(x,u_n)-\frac{1}{\nu }g(x,u_n)u_n\right) \,\mathrm{d}x. \end{aligned}$$

By $(g_1)$, (2.1) and Hölder’s inequality, we get

$$\begin{aligned} \int _{{\mathbb {R}}^3}\left( G(x,u_n)-\frac{1}{\nu }g(x,u_n)u_n\right) \,\mathrm{d}x\le & {} \int _{{\mathbb {R}}^3}|G(x,u_n)|\,\mathrm{d}x+\frac{1}{\nu }\int _{{\mathbb {R}}^3}|g(x,u_n)u_n|\,\mathrm{d}x\nonumber \\\le & {} \int _{{\mathbb {R}}^3}\left( \frac{1}{q}+\frac{1}{\nu }\right) |h(x)||u_n|^{q}\,\mathrm{d}x\nonumber \\\le & {} \frac{(q+\nu )d_2^{q}}{q\nu }|h|_\frac{2}{2-q}\Vert u_n\Vert _{\lambda }^{q}. \end{aligned}$$

(3.6)

Thus,

$$\begin{aligned} M+1\ge \left( \frac{1}{2}-\frac{1}{\nu }\right) \Vert u_n\Vert _{\lambda }^2-\frac{(q+\nu )d_2^{q}}{q\nu }|h|_\frac{2}{2-q}\Vert u_n\Vert _{\lambda }^{q}, \end{aligned}$$

which is a contradiction, when $T>0$ sufficiently large.

If the case (ii) holds, then

$$\begin{aligned} \eta \left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) \le 1\quad \mathrm {and}\quad {\eta }^{'}\left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) \le \left| \eta ^{'}\left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) \right| \le 2. \end{aligned}$$

(3.7)

It follows from $(f_2)$, (3.6) and (3.7) that

$$\begin{aligned}&\left( \frac{1}{2}-\frac{1}{\nu }\right) \Vert u_n\Vert _{\lambda }^2-\frac{1}{\nu }\Vert (I_{b,\lambda }^T)^{'}(u_n)\Vert _{E_{\lambda }^{'}}\Vert u_n\Vert _{\lambda }\nonumber \\&\quad \le \left( \frac{1}{2}-\frac{1}{\nu }\right) \Vert u_n\Vert _{\lambda }^2+\frac{1}{\nu }\langle (I_{b,\lambda }^{T})^{'}(u_n), u_n \rangle \nonumber \\&\quad =I_{b,\lambda }^{T}(u_n)+\left( \frac{b}{\nu }-\frac{b}{4}\right) \eta \left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) |\nabla u_n|_2^4+\frac{b}{2\nu T^2}\eta ^{'}\left( \frac{\Vert u_n\Vert _{\lambda }^2}{T^2}\right) |\nabla u_n|_2^{4}\Vert u_n\Vert _{\lambda }^2\nonumber \\&\quad \quad -\,\int _{{\mathbb {R}}^3}\left( \frac{1}{\nu }f(x,u_n)u_n-F(x,u_n)\right) \,\mathrm{d}x+\int _{{\mathbb {R}}^3}\left( G(x,u_n)-\frac{1}{\nu }g(x,u_n)u_n\right) \,\mathrm{d}x\nonumber \\&\quad \le I_{b,\lambda }^{T}(u_n)+\left( \frac{b}{\nu }-\frac{b}{4}\right) \Vert u_n\Vert _{\lambda }^4+\frac{(q+\nu )d_2^{q}}{q\nu }|h|_\frac{2}{2-q}\Vert u_n\Vert _{\lambda }^{q}\nonumber \\&\quad \le I_{b,\lambda }^{T}(u_n)+CbT^{4}+CT^{q}. \end{aligned}$$

(3.8)

By virtue of $I_{b,\lambda }^T(u_n)\rightarrow c_{b,\lambda }^T$ and Lemma 3.3, for n large enough, one has

$$\begin{aligned} I_{b,\lambda }^T(u_n)\le 2c_{b,\lambda }^T\le 2\max \limits _{t\in [0,1]}I_{b,\lambda }^T(te)\le 2M. \end{aligned}$$

(3.9)

On the other hand, for n large enough, we have that

$$\begin{aligned} \left( \frac{1}{2}-\frac{1}{\nu }\right) \Vert u_n\Vert _{\lambda }^2-\frac{1}{\nu }\Vert (I_{b,\lambda }^T)^{'}(u_n)\Vert _{E_{\lambda }^{'}}\Vert u_n\Vert _{\lambda }\ge CT^2-T. \end{aligned}$$

(3.10)

Combining with (3.8)–(3.10), we have

$$\begin{aligned} CT^{2}-T-CT^{q} \le 2M+CbT^{4}, \end{aligned}$$

which is a contradiction for T large enough if $b_{T}:=\frac{1}{T^4}>0$ and $b \in (0,min\{b_0,b_T\})$. So the claim follows. $\square$

Remark 3.1

From the above lemma, the sequence $\{u_n\}$ obtained in Lemma 3.4 is also a Cerami sequence at level $c_{b,\lambda }^T>0$ for $I_{b,\lambda }$, that is,

$$\begin{aligned} I_{b,\lambda }(u_n)\rightarrow c_{b,\lambda }^T \quad \mathrm{and} \quad (1+\Vert u_n\Vert _{\lambda })\parallel (I_{b,\lambda })^{'}(u_n)\parallel _{E_{\lambda }^{'}}\rightarrow 0. \end{aligned}$$

Lemma 3.5

Assume that ($f_1$) and ($f_3$) hold. If $u_n\rightharpoonup u$ in $H^1({\mathbb {R}}^3)$, then along a subsequence of $\{u_n\}$,

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\sup \limits _{\varphi \in H^{1}({\mathbb {R}}^3),\Vert \varphi \Vert _{H^{1}}\le 1}\left| \int _{{\mathbb {R}}^3}[f(x,u_n)-f(x,u_n-u)-f(x,u)]\varphi \,\mathrm{d}x\right| =0. \end{aligned}$$

Proof

This lemma has been proved on pages 77–80 of [15] and the appendix of Ackermann and Weth [1]. $\square$

We are now ready to give the compactness condition for $I_{b,\lambda }$. For this, we need to establish the following lemma to prove that $I_{b,\lambda }$ satisfies the Cerami condition by relying on the relevant parameters.

Lemma 3.6

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. If $\{u_n\}\subset E_{\lambda }$ is a sequence satisfying (3.5), then up to a subsequence, there exists $\lambda ^{*}>1$ such that for each $b \in (0,min\{b_{0},b_{T}\})$ and $\lambda \in (\lambda ^{*},\infty )$, $\{u_n\}\subset E_{\lambda }$ contains a convergent subsequence.

Proof

By Lemma 3.4, up to a subsequence, we have that $\Vert u_n\Vert _{\lambda }\le T$. Thus, there exist $u \in E_{\lambda }$ and ${\mathcal {A}} \in {\mathbb {R}}$ such that

$$\begin{aligned} {\left\{ \begin{array}{ll} u_n\rightharpoonup u\ &{}\mathrm {in}\ E_{\lambda },\\ u_n\rightarrow u &{}\mathrm {in}\ L_{loc}^{s}({\mathbb {R}}^3),\quad \forall s\in [2,6),\\ u_n\rightarrow u &{}\mathrm {a.e.\ on}\ {\mathbb {R}}^3,\\ \end{array}\right. } \end{aligned}$$

and

$$\begin{aligned} \int _{{\mathbb {R}}^3}|\nabla u_n|^2\,\mathrm{d}x\rightarrow {\mathcal {A}}^2,\qquad \int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x\le {\mathcal {A}}^2. \end{aligned}$$

It follows from Lemma 2.2 and Hölder’s inequality that

$$\begin{aligned} \int _{{\mathbb {R}}^3}g(x,u_n)v\,\mathrm{d}x\rightarrow \int _{{\mathbb {R}}^3}g(x,u)v\,\mathrm{d}x,\quad \forall v\in E_{\lambda }. \end{aligned}$$

Then, $I_{b,\lambda }^{'}(u_n)\rightarrow 0$ implies that

$$\begin{aligned}&(1+b{\mathcal {A}}^2)\int _{{\mathbb {R}}^3}\nabla u \nabla v\,\mathrm{d}x+\int _{{\mathbb {R}}^3}\lambda V(x)uv\,\mathrm{d}x\nonumber \\&\quad -\,\int _{{\mathbb {R}}^3}f(x,u)v\,\mathrm{d}x-\int _{{\mathbb {R}}^3}g(x,u)v\,\mathrm{d}x=0,\quad \forall v \in E_{\lambda }. \end{aligned}$$

(3.11)

Taking $v=u$ in (3.11), we obtain

$$\begin{aligned}&(1+b{\mathcal {A}}^2)\int _{{\mathbb {R}}^3}|\nabla u|^2\,\mathrm{d}x+\int _{{\mathbb {R}}^3}\lambda V(x)u^2\,\mathrm{d}x\nonumber \\&\quad -\,\int _{{\mathbb {R}}^3}f(x,u)u\,\mathrm{d}x-\int _{{\mathbb {R}}^3}g(x,u)u\,\mathrm{d}x=0. \end{aligned}$$

(3.12)

Next, it is sufficient to prove that $u_n\rightarrow u$ in $E_{\lambda }$. Let $v_n:=u_n-u$, then $v_n\rightharpoonup 0$ in $E_{\lambda }$. Since $u_n\rightharpoonup u$ in $E_{\lambda }$, we obtain

$$\begin{aligned} \Vert u_n\Vert _{\lambda }^2= & {} (v_n,v_n)_{\lambda }+(u,u_n)_{\lambda }+(v_n,u)_{\lambda }\nonumber \\= & {} \Vert v_n\Vert _{\lambda }^2+\Vert u\Vert _{\lambda }^2+o(1). \end{aligned}$$

(3.13)

By the weak lower semi-continuity of norm, we have

$$\begin{aligned} \Vert v_n\Vert _{\lambda }\le \Vert u_n\Vert _{\lambda }+\Vert u\Vert _{\lambda } \le \Vert u_n\Vert _{\lambda }+ \liminf \limits _{n\rightarrow \infty }\Vert u_n\Vert _{\lambda }\le 2T. \end{aligned}$$

(3.14)

It follows from $(V_2)$ that

$$\begin{aligned} |v_n|_2^2=\int _{{{\mathbb {R}}^3}\setminus {{\mathcal {V}}_c}}v_n^2\,\mathrm{d}x+\int _{{\mathcal {V}}_c}v_n^2\,\mathrm{d}x\le \frac{1}{\lambda c}\Vert v_n\Vert _{\lambda }^2+o(1). \end{aligned}$$

Then, by the Hölder and Sobolev inequalities, we have

$$\begin{aligned} |v_n|_p\le |v_n|_2^\sigma |v_n|_6^{1-\sigma }\le S^\frac{\sigma -1}{2}|v_n|_2^\sigma |\nabla v_n|_2^{1-\sigma }\le S^\frac{\sigma -1}{2}(\lambda c)^{-\frac{\sigma }{2}}\Vert v_n\Vert _{\lambda }+o(1), \end{aligned}$$

(3.15)

where $\sigma =\frac{6-p}{2p}>0$. On the one hand, by the definition of the operator norm and Lemma 3.5, we get

$$\begin{aligned}&\left| \int _{{\mathbb {R}}^3}[f(x,u_n)-f(x,v_n)-f(x,u)]u_n\,\mathrm{d}x\right| \nonumber \\&\quad \le \Vert u_n\Vert _{H^{1}({\mathbb {R}}^3)}\sup \limits _{\varphi \in H^{1}({\mathbb {R}}^3),\Vert \varphi \Vert _{H^{1}}\le 1}\nonumber \\&\left| \int _{{\mathbb {R}}^3}[f(x,u_n)-f(x,v_n)-f(x,u)]\varphi \,\mathrm{d}x\right| =o(1). \end{aligned}$$

(3.16)

Since $v_n \rightharpoonup 0$ in $H^1({\mathbb {R}}^3)$ and $v_n\rightarrow 0$ in $L_{loc}^s({\mathbb {R}}^3)$ for $s\in [2, 6)$, then it follows from (3.16) that

$$\begin{aligned} \int _{{\mathbb {R}}^3}f(x,u_n)u_n\,\mathrm{d}x= & {} \int _{{\mathbb {R}}^3}f(x,u)u\,\mathrm{d}x+\int _{{\mathbb {R}}^3}f(x,v_n)v_n\,\mathrm{d}x\nonumber \\&+\,\int _{{\mathbb {R}}^3}f(x,v_n)u\,\mathrm{d}x+\int _{{\mathbb {R}}^3}f(x,u)v_n\,\mathrm{d}x\nonumber \\&+\,\int _{{\mathbb {R}}^3}[f(x,u_n)-f(x,v_n)-f(x,u)]u_n\,\mathrm{d}x\nonumber \\= & {} \int _{{\mathbb {R}}^3}f(x,u)u\,\mathrm{d}x+\int _{{\mathbb {R}}^3}f(x,v_n)v_n\,\mathrm{d}x+o(1). \end{aligned}$$

(3.17)

From (3.3), (3.14), (3.15) and let $\varepsilon =\frac{1}{2d_2^2}$, we get

$$\begin{aligned} \int _{{\mathbb {R}}^3}f(x,v_n)v_n\,\mathrm{d}x&\le \varepsilon |v_n|_2^2+C_{\varepsilon }|v_n|_p^{p-2}|v_n|_p^2\nonumber \\&\le \frac{1}{2}\Vert v_n\Vert _{\lambda }^2+C_{1/2d_2^2}(2Td_p)^{p-2}S^{\sigma -1}(\lambda c)^{-\sigma }\Vert v_n\Vert _{\lambda }^2+o(1). \end{aligned}$$

(3.18)

On the other hand, according to Lemma 2.2, we have

$$\begin{aligned}&\int _{{\mathbb {R}}^3}g(x,u_n)u_n\,\mathrm{d}x-\int _{{\mathbb {R}}^3}g(x,u)u\,\mathrm{d}x\nonumber \\&\quad =\int _{{\mathbb {R}}^3}[g(x,u_n)-g(x,u)]u_n\,\mathrm{d}x+\int _{{\mathbb {R}}^3}g(x,u)(u_n-u)\,\mathrm{d}x\nonumber \\&\quad \le \left( \int _{{\mathbb {R}}^3}|g(x,u_n)-g(x,u)|^2\,\mathrm{d}x\right) ^{\frac{1}{2}}|u_n|_2+\int _{{\mathbb {R}}^3}g(x,u)v_n\,\mathrm{d}x\nonumber \\&\quad \rightarrow 0\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

(3.19)

Combining (3.13) and (3.17)–(3.19), we infer that

$$\begin{aligned} o(1)&=\langle I_{b,\lambda }^{'}(u_n),u_n\rangle \\&=\Vert u_n\Vert _{\lambda }^2+b|\nabla u_n|_2^4-\int _{{\mathbb {R}}^3}f(x,u_n)u_n\,\mathrm{d}x-\int _{{\mathbb {R}}^3}g(x,u_n)u_n\,\mathrm{d}x\\&\quad -\,\Vert u\Vert _{\lambda }^2-b{\mathcal {A}}^2|\nabla u|_2^2+\int _{{\mathbb {R}}^3}f(x,u)u\,\mathrm{d}x+\int _{{\mathbb {R}}^3}g(x,u)u\,\mathrm{d}x\\&=\Vert v_n\Vert _{\lambda }^2-\int _{{\mathbb {R}}^3}f(x,v_n)v_n\,\mathrm{d}x-\int _{{\mathbb {R}}^3}[g(x,u_n)u_n-g(x,u)u]\,\mathrm{d}x+b{\mathcal {A}}^4\\&\quad -b{\mathcal {A}}^2|\nabla u|_2^2+o(1) \ge \Vert v_n\Vert _{\lambda }^2-\int _{{\mathbb {R}}^3}f(x,v_n)v_n\,\mathrm{d}x+o(1)\\&\ge \big (\frac{1}{2}-C_{1/2d_2^2}(2Td_p)^{p-2}S^{\sigma -1}(\lambda c)^{-\sigma }\big )\Vert v_n\Vert _{\lambda }^2+o(1). \end{aligned}$$

Therefore, we can choose

$$\begin{aligned} \lambda ^{*}=\mathrm {max}\left\{ \frac{\left( 2C_{1/2d_2^2}(2Td_p)^{p-2}S^{\sigma -1}\right) ^{1/\sigma }}{c},1\right\} \end{aligned}$$

such that $v_n\rightarrow 0$ in $E_{\lambda }$ for all $\lambda >\lambda ^{*}.$ This completes the proof. $\square$

4 Proof of Theorem 2.1

Proof of Theorem 2.1

Let T be defined as in Lemma 3.4. By Lemmas 3.1 and 3.2, there exists $b_0>0$ such that for every $\lambda \ge 1$ and $b\in (0,b_0)$, $I_{b,\lambda }^T$ possesses a Cerami sequence $\{u_n\}$ at the mountain pass level $c_{b,\lambda }^T$. From Lemmas 3.3, 3.4 and Remark 3.1, we know that there exists $b_T>0$ such that for every $\lambda \ge 1$ and $b\in (0,min\{b_0,b_T\})$, after passing to a subsequence, $\{u_n\}$ is a Cerami sequence of $I_{b,\lambda }$ satisfying $\Vert u_n\Vert _{\lambda }\le T$, that is,

$$\begin{aligned} \sup \limits _{n\in {\mathbb {N}}}\Vert u_n\Vert _{\lambda }\le T,\qquad I_{b,\lambda }(u_n)\rightarrow c_{b,\lambda }^T \end{aligned}$$

and

$$\begin{aligned} (1+\Vert u_n\Vert _{\lambda })\Vert I_{b,\lambda }^{'}(u_n)\Vert _{E_{\lambda }^{'}}\rightarrow 0 \end{aligned}$$

as $n\rightarrow +\infty$. It follows from Lemma 3.6 that there exists $\lambda ^{*}>0$ such that for each $b\in (0,min\{b_0,b_T\})$ and $\lambda \in (\lambda ^*,\infty )$, the sequence $\{u_n\} \subset E_{\lambda }$ contains a convergent subsequence. Without loss of generality, we can assume that there exists $u_{b,\lambda }^{+}\in E_{\lambda }$ such that $u_n \rightarrow u_{b,\lambda }^{+}$ in $E_{\lambda }$ as $n \rightarrow \infty$. Furthermore, we have

$$\begin{aligned} 0<\Vert u_{b,\lambda }^{+}\Vert _{\lambda }\le T,\qquad I_{b,\lambda }(u_{b,\lambda }^{+})=c_{b,\lambda }^T>0\quad \mathrm {and} \quad I_{b,\lambda }^{'}(u_{b,\lambda }^{+})=0. \end{aligned}$$

Consequently, we infer that $u_{b,\lambda }^{+}$ is a nontrivial solution of (${\mathcal {K}}_{b,\lambda }$) for all $b\in (0,min\{b_0, b_T\}$ and $\lambda \in (\lambda ^{*},\infty )$. This ends the proof. $\square$

5 Asymptotic Behavior of Nontrivial Solutions

Proof of Theorem 2.2

We follow the argument in [36] (or see [4, 10]). Let $b\in (0,min\{b_{0},b_T\})$ be fixed. For any sequence $\{\lambda _n\}\subset (\lambda ^{*},+\infty )$ with $\lambda _n \rightarrow +\infty$, where

$$\begin{aligned} \lambda ^{*}=\mathrm {max}\left\{ \frac{\left( 2C_{1/2d_2^2}(2Td_p)^{p-2}S^{\sigma -1}\right) ^{1/\sigma }}{c},1\right\} , \end{aligned}$$

let $u_n:=u_{b,\lambda _n}^{+}$ be the critical point of $I_{b,\lambda _n}$ obtained by Theorem 2.1. By Lemma 3.4, we have

$$\begin{aligned} 0<\Vert u_n\Vert _{\lambda _n}\le T \quad \mathrm {for\ all}\ n. \end{aligned}$$

(5.1)

Thus, up to a subsequence, we may assume that

$$\begin{aligned} {\left\{ \begin{array}{ll} u_n\rightharpoonup u_b^{+} \quad &{}\mathrm {in}\,\,E,\\ u_n\rightarrow u_b^{+} \quad &{}\mathrm {in}\,\,L_{loc}^{s}({\mathbb {R}}^3)\,\,\mathrm {for}\,\,s\in [2,6),\\ u_n\rightarrow u_b^{+} \quad &{}\mathrm {a.e.}\,\mathrm {on}\ {\mathbb {R}}^3. \end{array}\right. } \end{aligned}$$

It follows from (5.1), Fatou’s lemma and $(V_1)$ that

$$\begin{aligned} 0\le \int _{{\mathbb {R}}^3}V(x)|u_{b}^{+}|^2\,\mathrm{d}x\le \liminf \limits _{n\rightarrow \infty }\int _{{\mathbb {R}}^3}V(x)u_{n}^2\,\mathrm{d}x\le \liminf \limits _{n\rightarrow \infty }\frac{\Vert u_n\Vert _{\lambda _n}^2}{\lambda _n}=0. \end{aligned}$$

Hence, $u_b^{+}=0$ a.e. in ${\mathbb {R}}^{3} \setminus V^{-1}(0)$, and so $u_b^{+}\in H_0^1(\Omega )$ by the condition $(V_3)$.

Next, we claim that $u_n\rightarrow u_b^{+}$ in $L^{s}({\mathbb {R}}^3)$ for all $s\in (2,6)$. Contrary to the conclusion, by Lions’ vanishing lemma in [33], there exist $\epsilon , r>0$ and $x_n \in {\mathbb {R}}^3$ such that

$$\begin{aligned} \int _{B_r(x_n)}(u_n-u_b^{+})^2\,\mathrm{d}x\ge \epsilon , \end{aligned}$$

which shows that $|x_n|\rightarrow \infty$ as $n\rightarrow \infty$. Thus $|B_r(x_n)\cap {\mathcal {V}}_c|\rightarrow 0$ as $n\rightarrow \infty$. Moreover, by Hölder inequality, we get

$$\begin{aligned} \int _{B_r(x_n)\cap {\mathcal {V}}_c}(u_n-u_b^{+})^2\,\mathrm{d}x \le |B_r(x_n)\cap {\mathcal {V}}_c|^{\frac{2}{3}} |u_n-u_b^{+}|_6^2\ \rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty$. Consequently, we get

$$\begin{aligned} \Vert u_n\Vert _{\lambda _n}^2&\ge \lambda _{n}c\int _{B_{r}(x_n)\cap \{V\ge c\}}u_{n}^2\,\mathrm{d}x\\&\ge \lambda _{n}c\int _{B_{r}(x_n)\cap \{V\ge c\}}(u_{n}-u_{b}^{+})^2\,\mathrm{d}x\\&=\lambda _{n}c\left( \int _{B_{r}(x_n)}(u_{n}-u_{b}^{+})^2\,\mathrm{d}x-\int _{B_{r}(x_n)\cap {{\mathcal {V}}_c}}(u_{n}-u_{b}^{+})^2\,\mathrm{d}x\right) \\&\rightarrow +\infty \quad \mathrm {as}\ n\rightarrow \infty , \end{aligned}$$

which contradicts (5.1). Thus, we have that $u_n\rightarrow u_b^{+}$ in $L^{s}({\mathbb {R}}^3)$ for all $s\in [2,6)$. Therefore, from (3.17) and (3.19), we obtain

$$\begin{aligned} \int _{{\mathbb {R}}^3}f(x,u_n)u_n\,\mathrm{d}x=\int _{{\mathbb {R}}^3}f(x,u_b^{+})u_b^{+}\,\mathrm{d}x+o(1) \end{aligned}$$

(5.2)

and

$$\begin{aligned} \int _{{\mathbb {R}}^3}g(x,u_n)u_n\,\mathrm{d}x=\int _{{\mathbb {R}}^3}g(x,u_b^{+})u_b^{+}\,\mathrm{d}x+o(1). \end{aligned}$$

(5.3)

Now, we prove that $u_n\rightarrow u_b^{+}$ in E. Indeed, by the fact that

$$\begin{aligned} \langle I_{b,\lambda _n}^{'}(u_n), u_n\rangle =\langle I_{b,\lambda _n}^{'}(u_n), u_b^{+}\rangle =0, \end{aligned}$$

we have

$$\begin{aligned} \Vert u_n\Vert _{\lambda _n}^2+b|\nabla u_n|_2^4=\int _{{\mathbb {R}}^3}g(x,u_n)u_n\,\mathrm{d}x+\int _{{\mathbb {R}}^3}f(x,u_n)u_n\,\mathrm{d}x \end{aligned}$$

(5.4)

and

$$\begin{aligned}&\left( 1+b\int _{{\mathbb {R}}^3}|\nabla u_n|^2\,\mathrm{d}x\right) \int _{{\mathbb {R}}^3}\nabla u_n \nabla u_b^{+}\,\mathrm{d}x+\int _{{\mathbb {R}}^3}\lambda _n V(x)u_{n}u_{b}^{+}\,\mathrm{d}x\nonumber \\&\quad =\int _{{\mathbb {R}}^3}g(x,u_n)u_b^{+}\,\mathrm{d}x+ \int _{{\mathbb {R}}^3}f(x,u_n)u_b^{+}\,\mathrm{d}x. \end{aligned}$$

It follows from $u_b^{+}=0$ almost everywhere in ${\mathbb {R}}^{3} \setminus V^{-1}(0)$ that

$$\begin{aligned} \Vert u_b^{+}\Vert ^2+b|\nabla u_n|_2^{2}|\nabla u_b^{+}|_2^2=\int _{{\mathbb {R}}^3}g(x,u_b^{+})u_b^{+}\,\mathrm{d}x+\int _{{\mathbb {R}}^3}f(x,u_b^{+})u_b^{+}\,\mathrm{d}x+o(1). \end{aligned}$$

(5.5)

Passing to subsequence if necessary, we assume that $\Vert u_n\Vert _{\lambda _n}^2\rightarrow \beta _1$ and $|\nabla u_n|_2^2\rightarrow \beta _2$. By Fatou’s Lemma, we get

$$\begin{aligned} |\nabla u_b^{+}|_2^2\le \liminf \limits _{n\rightarrow \infty }|\nabla u_n|_2^2=\beta _2. \end{aligned}$$

(5.6)

From (5.4) to (5.6), we thus deduce that

$$\begin{aligned} \beta _1+b\beta _2^2=\Vert u_b\Vert ^2+b\beta _2|\nabla u_b^{+}|_2^2\le \Vert u_b^{+}\Vert ^2+b\beta _2^2. \end{aligned}$$

Therefore, $\beta _1\le \Vert u_b^{+}\Vert ^2$. By the weak lower semi-continuity of norm, we have

$$\begin{aligned} \Vert u_b\Vert ^2\le \liminf \limits _{n\rightarrow \infty }\Vert u_n\Vert ^2\le \limsup \limits _{n\rightarrow \infty }\Vert u_n\Vert ^2 \le \limsup \limits _{n\rightarrow \infty }\Vert u_n\Vert _{\lambda _n}^2=\beta _1\le \Vert u_b^{+}\Vert ^2, \end{aligned}$$

(5.7)

which shows that $\Vert u_n\Vert ^2\rightarrow \Vert u_b^{+}\Vert ^2$ as $n\rightarrow \infty$. Since E is a uniformly convex Hilbert space, it yields that $u_n\rightarrow u_b^{+}$ in E.

Next, we shall prove that $u_b^{+}$ is a weak solution of (${\mathcal {K}}_{b,\infty }$). For any $v\in C_0^{\infty }(\Omega )$, since $\langle I_{b,\lambda _n}^{'}(u_n),v\rangle =0$, it is easy to check that

$$\begin{aligned} \left( 1+b\int _{\Omega }|\nabla u_b^{+}|^2\,\mathrm{d}x\right) \int _{\Omega }\nabla u_b^{+} \nabla v\,\mathrm{d}x=\int _{\Omega }g(x,u_b^{+})v\,\mathrm{d}x+\int _{\Omega }f(x,u_b^{+})v\,\mathrm{d}x \end{aligned}$$

i.e., $u_b^{+}$ is a weak solution of (${\mathcal {K}}_{b,\infty }$) by the density of $C_0^{\infty }(\Omega )$ in $H_0^{1}(\Omega )$. Finally, we show that $u_{b}^{+}\not \equiv 0$. If not, we have $u_n\rightarrow 0$ in E which implies that

$$\begin{aligned} 0&\le |I_{b,\lambda _n}(u_n)|\\&\le \frac{1}{2}\Vert u_{n}\Vert _{\lambda _n}^{2}+\frac{b}{4}|\nabla u_n|_2^4+\int _{{\mathbb {R}}^3}F(x,u_n)\,\mathrm{d}x+\int _{{\mathbb {R}}^3}G(x,u_n)\,\mathrm{d}x\\&\le \frac{1}{2}\Vert u_{n}\Vert _{\lambda _n}^{2}+\frac{b}{4}|\nabla u_n|_2^4+\frac{\varepsilon }{2}|u_n|_2^2+\frac{C_\varepsilon }{p}|u_n|_p^p +\frac{1}{q}|h|_{\frac{2}{2-q}}|u_n|_2^{q}\\&\le \frac{1}{2}\Vert u_{n}\Vert _{\lambda _n}^{2}+\frac{b}{4}\Vert u_n\Vert ^4+\frac{\varepsilon d_2^2}{2}\Vert u_n\Vert ^2+\frac{C_\varepsilon d_p^p}{p}\Vert u_n\Vert ^p+\frac{d_2^{q}}{q}|h|_{\frac{2}{2-q}}\Vert u_n\Vert ^{q}\\&\rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty$. Thus, we have that

$$\begin{aligned} I_{b,\lambda _n}(u_n)\rightarrow 0\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

Moreover, by virtue of $u_n$ being the critical point of $I_{b,\lambda _n}$ obtained by Theorem 2.1, we get $I_{b,\lambda _n}(u_n)= c_{b,\lambda }^T$, which is a contradiction. Therefore, $u_b^{+}$ is a nontrivial weak solution of equation (${\mathcal {K}}_{b,\infty }$). The proof is thus finished. $\square$

Proof of Theorem 2.3

Let $\lambda \in (\lambda ^{*}, \infty )$ be fixed. For any subsequence $\{b_n\}\subset (0,min\{b_0,b_T\})$ with $b_{n}\rightarrow 0$, let $u_n:=u_{b_n, \lambda }^{+}$ be the critical point of $I_{b,\lambda _n}$ obtained by Theorem 2.1. It follows from Theorem 2.1 that

$$\begin{aligned} 0<\Vert u_n\Vert _{\lambda }\le T \quad \mathrm {for}\ \mathrm {all}\ n. \end{aligned}$$

Passing to a subsequence if necessary, we may assume that $u_{n}\rightharpoonup u_{\lambda }^{+}$ in $E_{\lambda }$. Note that $I_{b_n, \lambda }^{'}(u_n)=0$, we may deduce that $u_n\rightarrow u_{\lambda }^{+}$ in $E_{\lambda }$ as the proof of Lemma (3.6).

To complete the proof, it suffices to show that $u_{\lambda }^{+}$ is a weak solution of Eq. (${\mathcal {K}}_{0,\lambda }$). Now for any $v\in E_{\lambda }$, since $\langle I_{b_n, \lambda }(u_n), v\rangle =0$, it is easy to check that

$$\begin{aligned} \int _{{\mathbb {R}}^3}(\nabla u_{\lambda }^{+}\nabla v+\lambda V(x) u_{\lambda }^{+}v)\,\mathrm{d}x=\int _{{\mathbb {R}}^3} g(x,u_{\lambda }^{+})v\,\mathrm{d}x+\int _{{\mathbb {R}}^3} f(x,u_{\lambda }^{+})v\,\mathrm{d}x. \end{aligned}$$

Therefore, $u_{\lambda }^{+}$ is a weak solution of (${\mathcal {K}}_{0,\lambda }$). Furthermore, $u_{\lambda }^{+}$ is a nontrivial solution of Equation (${\mathcal {K}}_{0,\lambda }$). The proof is same to the last of part of the proof of Theorem 2.2 and so we omit it. This completes the proof. $\square$

Proof of Theorem 2.4

Similar to the proof of Theorem 2.2, we can easily complete this theorem. $\square$

6 The Second Solution

In this section, we want to prove the existence of a positive solution and a negative solution. To this end, we establish the following Ekeland variational principle, which plays an important role in proving Theorem

Theorem 6.1

(Ekeland variational principle [18]) Let X be a Banach space, $\Psi \in {\mathcal {C}}^{1}$ bounded below, $v\in X$ and $\varepsilon ,\delta >0.$ If

$$\begin{aligned} \Psi (v)\le \inf \limits _{X}\Psi +\varepsilon , \end{aligned}$$

then there exists $u\in X$ such that

$$\begin{aligned} \Psi (u)\le \inf \limits _{X}\Psi +2\varepsilon ,\quad \Vert \Psi ^{'}(u)\Vert <\frac{8\varepsilon }{\delta },\quad \Vert u-v\Vert \le 2\delta . \end{aligned}$$

Lemma 6.1

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, for any $b>0$ and $\lambda \ge 1$, there exist $\alpha ', \rho '>0$ (independent of b) such that for all

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^{q}}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}}, \end{aligned}$$

and $\Vert u\Vert _{\lambda }=\rho '$,

$$\begin{aligned} I_{b,\lambda }(u)\ge \alpha '. \end{aligned}$$

Proof

The proof of this lemma has been completed in Lemma 3.1. So we omit it. $\square$

Lemma 6.2

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, for any $b>0$ and $\lambda \ge 1$, there exists $\phi \in E_{\lambda }$ with $\Vert \phi \Vert _{\lambda }<\rho '$ such that $I_{b,\lambda }(\phi )<0$, where $\rho '>0$ is given in Lemma 6.1.

Proof

By ($g_2$), there exist $C_3,C_4>0$ such that

$$\begin{aligned} G(x,u)\ge C_3|u|^{\theta }-C_4,\quad \forall (x,u)\in {\mathbb {R}}^3\times {\mathbb {R}}. \end{aligned}$$

For any $t>0$ and $u\in C_{0}^{\infty }(\Omega )$, it follows from (3.4) and ($g_2$) that

$$\begin{aligned} I_{b,\lambda }(tu)&=\frac{t^2}{2}\int _{\Omega }|\nabla u|^2\,\mathrm{d}x+\frac{bt^4}{4}(\int _{\Omega }|\nabla u|^2 dx)^2-\int _{\Omega }G(x,tu)dx-\int _{\Omega }F(x,tu)dx\\&\le \frac{t^2}{2}\int _{\Omega }|\nabla u|^2\,\mathrm{d}x+\frac{bt^4}{4}(\int _{\Omega }|\nabla u|^2 dx)^2\\&\quad +C_{4}|\Omega |-C_{3}t^{\theta }\int _{\Omega }|u|^{\theta }\,\mathrm{d}x+Ct^2 \int _{\Omega }|u|^2\,\mathrm{d}x-Ct^{\nu }\int _{\Omega }|u|^{\nu }\,\mathrm{d}x\\&<0 \end{aligned}$$

for $t>0$ enough small. Thus, there exists ${\bar{t}}>0$ and $\phi :={\bar{t}}u$ such that $I_{b,\lambda }(\phi )<0$ when $\Vert \phi \Vert _{\lambda }\le \rho '.$ $\square$

Remark 6.1

By Lemmas 6.1 and 6.2, we can infer that $I_{b,\lambda }$ is bounded below in ${\bar{B}}_{\rho '}(0)$ and

$$\begin{aligned} c^{*}:=\inf \limits _{u\in {\bar{B}}_{\rho '}(0)}I_{b,\lambda }(u)<0, \end{aligned}$$

where $B_{\rho '}(0)=\{u\in E_{\lambda }:\Vert u\Vert _{\lambda }<\rho '\}$.

Next, we show that $I_{b,\lambda }$ satisfies $(PS)_c$ condition in ${\bar{B}}_{\rho '}(0)$ with $c<0$.

Lemma 6.3

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, there exists $\lambda _{*}>1$ such that $I_{b,\lambda }$ satisfies $(PS)_c$ condition in ${\bar{B}}_{\rho '}(0)$ with $c<0$ for all $\lambda >\lambda _{*}.$

Proof

If $\{u_n\}\subset E_{\lambda }$ is a $(PS)_c$ sequence for $I_{b,\lambda }$ in ${\bar{B}}_{\rho '}(0)$ with $c<0$. Thus, we have $\Vert u_n\Vert _{\lambda }<\rho '$. Therefore, there exist $u^{-}\in E_{\lambda }$ and constant ${\mathcal {B}}>0$ such that

$$\begin{aligned} u_n\rightharpoonup u^{-}\ \mathrm {in}\ E_{\lambda },\quad |\nabla u_n|_2^2\rightarrow {\mathcal {B}}^2,\quad |\nabla u^{-}|_2^2\le {\mathcal {B}}^2. \end{aligned}$$

The rest of the proof in this lemma is similar to Lemma 3.6. So we omit it. $\square$

Proposition 6.1

Suppose that $(V_1)-(V_3)$, $(f_1)-(f_3)$ and $(g_1)-(g_2)$ are satisfied. Then, there exists $\lambda _{*}>0$ such that for each

$$\begin{aligned} |h|_{\frac{2}{2-q}}\le \left( \frac{8 d_2^{q}}{q}\left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}} d_p^{p}q(p-2)}\right) ^{\frac{q-2}{p-q}}+\frac{8C_{1/{2d_{2}^{2}}}d_p^{p}}{p} \left( \frac{d_{2}^{q}p(2-q)}{C_{1/{2d_{2}^{2}}}d_p^{p}q(p-2)}\right) ^\frac{p-2}{p-q}\right) ^{-\frac{p-q}{p-2}}, \end{aligned}$$

and $\lambda >\lambda _{*}$, the functional $I_{b,\lambda }$ has a local minimizer $u_{b,\lambda }^{-}\in E_{\lambda }$. Furthermore, $u_{b,\lambda }^{-}$ is a nontrivial solution of Eq. (${\mathcal {K}}_{b,\lambda }$)and $I_{b,\lambda }(u_{b,\lambda }^{-})<0$ with $\Vert u_{b,\lambda }^{-}\Vert _{\lambda }<\rho '$.

Proof

On the one hand, by Theorem 6.1 and Remark 6.1, there exists a sequence $\{u_n\}\subset {\bar{B}}_{\rho '}(0)$ such that

$$\begin{aligned} I_{b,\lambda }(u_n)\rightarrow c^{*}\quad \mathrm {and}\quad (I_{b,\lambda })^{'}(u_n)\rightarrow 0\quad \mathrm {as}\ n\rightarrow \infty . \end{aligned}$$

On the other hand, by Lemma 6.3, there exists $\lambda _{*}>0$ such that $I_{b,\lambda }$ satisfies $(PS)_{c^{*}}$ condition in ${\bar{B}}_{\rho '}(0)$ with $c^{*}<0$ for $\lambda >\lambda _{*}$. Thus, up to a subsequence, there exists a subsequence $\{u_n\}$ and $u_{b,\lambda }^{-}\in {\bar{B}}_{\rho '}(0)$ such that $u_n\rightarrow u_{b,\lambda }^{-}$ in $E_{\lambda }$ for $\lambda >\lambda _{*}$. Thus, $u_{b,\lambda }^{-}$ is a local minimizer on ${\bar{B}}_{\rho '}(0)$ satisfying

$$\begin{aligned} I_{b,\lambda }(u_{b,\lambda }^{-})=c^{*}<0,\quad (I_{b,\lambda })^{'}(u_{b,\lambda }^{-})=0\quad \mathrm {and}\quad \Vert u_{b,\lambda }^{-}\Vert _{\lambda }<\rho '. \end{aligned}$$

Obviously, $u_{b,\lambda }^{-}$ is a nontrivial solution of Equation (${\mathcal {K}}_{b,\lambda }$) with $I_{b,\lambda }(u_{b,\lambda }^{-})<0$ and $\Vert u_{b,\lambda }^{-}\Vert _{\lambda }<\rho '$. The proof is completed. $\square$

Proof of Theorem 2.5

By Proposition 6.1, we can complete the proof of Theorem 2.5. $\square$