1 Introduction

In this paper we consider the following quasilinear Schrödinger equations

$$ i\frac{\partial z}{\partial t}=-\Delta z+W(x) z-\kappa \Delta \sqrt{1+|z|^{2}}\frac{z}{2\sqrt{1+|z|^{2}}}-\rho \bigl(|z|^{2}\bigr)z, \quad x\in \mathbb{R}^{N}, $$
(1)

where \(z(x,t):\mathbb{R}^{N}\times \mathbb{R}\rightarrow \mathbb{C}\), \(N\geq 2\), \(W(x):\mathbb{R}^{N}\rightarrow \mathbb{R}\) is a given potential, \(\kappa \) is a parameter and \(\rho \) is a real function. If we set \(\rho (s)=1-\frac{1}{\sqrt{1+s}}\), then (1) is known to describe propagation of high-power ultrashort laser pulse in a medium [1]. For this special case, in [2], Bouard et al. studied the Cauchy problem associated to (1) and they proved global existence and uniqueness of small solutions in transverse space dimensions 2 and 3, and local existence without any smallness condition in transverse space dimension 1. In [3], the locally well-posed of (1) on an interval \([0,T]\) was proved.

Here our interest is in the existence of standing wave solutions, that is, solutions of type \(z(x,t)=\exp (-iE t)u(x)\), where \(E \in \mathbb{R}\) and \(u>0\) is a real function. It is well known that \(z\) satisfies (1) if and only if the function \(u\) solves the following equations of elliptic type

$$ -\Delta u+V(x)u-\kappa \Delta \sqrt{1+u^{2}} \frac{u}{2 \sqrt{1+u ^{2}}}=f(u),\quad x\in \mathbb{R}^{N}, $$
(2)

where \(V(x)=W(x)-E\) is the new potential function and \(f(s)=\rho (|s|^{2})s\). For \(N=2\), Colin [4] studied the existence of ground states of (2) with \(V(x)=2\omega \), \(\kappa =-2\) and \(f(s)=s-\frac{s}{\sqrt{1+s^{2}}}\), where \(\omega \) is a fixed positive parameter. For \(N\geq 3\) and \(\kappa =1\), Shen and Wang [10] obtained the existence of nontrivial solutions for (2) if \(f(s)\) involves subcritical growth. Moreover, they assumed that \(f(s)\) satisfies the following generalized (AR) condition: there exists \(\mu >2 \) such that

$$ 0< \mu g(s)F(s)\leq G(s)f(s), \quad \forall s>0, $$
(3)

where \(g(s)=\sqrt{1+\frac{s^{2}}{2(1+s^{2})}}\), \(G(s)=\int _{0} ^{s}g(t)dt \) and \(F(s)=\int _{0}^{s}f(t)dt\). From the definition of \(g(s)\), we get \(\frac{s g(s)}{G(s)}\leq 6-2\sqrt{6}\) for \(s>0\) (Note that \(6-2\sqrt{6}\) is supremum) and (3) holds clearly if

$$ 0< \mu (6-2\sqrt{6})F(s)\leq sf(s), \quad \forall s>0. $$
(4)

Note that (4) implies that \(F(s)\geq C_{1}|s|^{\mu (6-2 \sqrt{6})}\) for \(s>C_{1}>0\). For a particular case, i.e., \(f(s)=|s|^{r-1}s\) with \(2(6-2\sqrt{6})< r+1<2^{*}\), the existence of a nontrivial solution of (2) was obtained in [12]. However, (4) is invalid if \(F(s)=|s|^{\beta -1}s\) with \(\beta \leq 2(6-2\sqrt{6})\). Therefore, a question is: whether Eq. (2) exists solutions if \(f(s)=|s|^{\beta -2}s\) with \(\beta \in (0,2(6-2\sqrt{6})]\)?

The main purpose of this paper is to extend the result in [10] to general \(\kappa \) and improve the generalized (AR) condition (3). It is well known that the (AR) condition in bounded domains was introduced by Ambrosetti and Rabinowitz [6] as follows: there exist \(\mu >2\) and \(t_{0}>0\) such that for \(|t|>t_{0}\),

$$\begin{aligned} 0< \mu F(x,t)\leq tf(x,t), \end{aligned}$$
(5)

where \({F(x,t)=\int _{0}^{t}f(x,s)dx}\). Since then, condition (5) is used frequently to study the existence of nontrivial solutions for elliptic problems. Simultaneously, some authors devoted themselves to improve (5). For example, condition (5) was improved by Shen, Guo in [9] as: there exist \(C>0\) and \(t_{0}>0\) such that for \(|t|>t_{0}\),

$$\begin{aligned} t f(x,t)-2F(x,t)\geq C|t|^{\mu _{0}}-f_{0}(x), \end{aligned}$$
(6)

where \(f_{0}(x)\in L(\varOmega )\), either \(\mu _{0}>\frac{N}{2}(p-2)\) for \(p\in (2+\frac{2}{N},2^{*})\) or \(\mu _{0}=1\) for \(p\in (2, 2+ \frac{2}{N}]\). Obviously, \(F(x,t)=\frac{1}{2}t^{2}\log (1+|t|)^{ \alpha }\) with \(\alpha \geq 1\) satisfies (6). For a similar improved result, we refer to Costa, Magalhães [8]. We remark that condition (6) was used to prove the boundedness of the Cerami consequence in [8, 9].

If condition (5) holds for all \(x\in \mathbb{R} ^{N}\) and \(t\in \mathbb{R}\setminus \{0\}\), then it was also adopted to study elliptic problems in unbounded domains, see e.g. [7]. Later, Jeanjean in [13] (see also [14]) introduced the following conditions:

$$ \lim_{t\rightarrow \infty }\frac{F(x,t)}{t^{2}}=+\infty $$

and there exists a constant \(D\in [1,+\infty )\) such that

$$\begin{aligned} \mathscr{F}(x,t)\leq D\mathscr{F}(x,s), \quad \forall 0< s \leq t , \end{aligned}$$
(7)

where \(\mathscr{F}(x,t)=tf(x,t)-2F(x,t)\). It is noted that if \(\frac{f(x,t)}{t}\) is non-decreasing in \(t\), then (7) holds with \(D=1\). As is pointed out in Tang [21] that conditions (5) and (7) are complementary to each other, in the sense that, the function \(F(x,t)=C|t|^{3}\int _{0}^{t}|s|^{1+ \sin s} sds\) meets (5) with \(\mu =3\) but not (7). For more results about the improvement of (5), we refer to [16,17,18,19,20] and references therein.

In what follows, we always assume that \(N\geq 3\), \(g(t)=\sqrt{1+\frac{ \kappa t^{2}}{2(1+t^{2})}}\) and \(G(t)=\int _{0}^{t}g(s)ds\). Clearly, \(g(t)\) is increasing for \(\kappa >0\) and is decreasing for \(\kappa <0\) in \(|t|\).

We also assume that the potential \(V:\mathbb{R}^{N}\rightarrow \mathbb{R}\) is continuous and satisfies the following conditions:

\((V)\) :

\(0< V_{0}\leq V(x)\leq V_{\infty }:=\lim _{|x|\rightarrow +\infty }V(x)<\infty , \text{for all } x \in \mathbb{R}^{N}\).

The function \(f(t) \) satisfies the following conditions:

\((f_{1})\) :

\(f(t)=o(t)\) as \(t\rightarrow 0^{+}\) and \(f(t)=0\) for \(t\leq 0\);

\((f_{2})\) :

there exists \(p\in (2,2^{*})\) such that \(f(t)\leq Cg(t)(G(t)+G(t)^{p-1}) \) for some positive constant \(C>0\) and all \(t\geq 0\);

\((f_{3})\) :

\(\lim _{t\rightarrow +\infty } \frac{F(t)}{G(t)^{2}} = +\infty \), where \(F(t)=\int _{0}^{t}f(s)ds\);

\((f_{4})\) :

there is \(D\in [1,+\infty )\) such that

$$ \mathscr{F}_{g}(t) \leq D\mathscr{F}_{g}(s), \quad \text{for }0 \leq t\leq s, $$
(8)

where \(\mathscr{F}_{g}(s) =\frac{1}{2} \frac{f(s)G(s)}{g(s)}-F(s)\).

Besides, we make the following assumption:

\((f_{4}')\) :

for \(\kappa >0\), \(\mathscr{F}_{g}(t)\geq 0\) for all \(t\geq 0\) and there exist \(C, t_{0}>0\) such that for \(|t|\geq t_{0}\), either

$$ \mathscr{F}_{g}(t) \geq C \biggl[\frac{F(t)}{t^{2}} \biggr]^{\lambda }, \quad \lambda >\frac{N}{2}, $$
(9)

or

$$ \mathscr{F}_{g}(t) \geq C|t|^{\mu }, \quad \mu >\frac{N}{2}(p-2), \quad p>\frac{2}{N}+2; $$
(10)
\((f_{4}'')\) :

for \(\kappa <0\), \(\mathscr{F}(t):=\frac{1}{2}f(t)t-F(t) \geq 0\) for all \(t\geq 0\) and there exist \(C_{1}, t_{1}>0\) such that for \(|t|\geq t_{1}\), either

$$ \mathscr{F}(t) \geq C_{1} \biggl[\frac{F(t)}{t^{2}} \biggr]^{\lambda }, \quad \lambda >\frac{N}{2}, $$
(11)

or

$$ \mathscr{F}(t) \geq C_{1}|t|^{\mu }, \quad \mu >\frac{N}{2}(p-2), \quad p>\frac{2}{N}+2. $$
(12)

Remark 1

Clearly, \(\mathscr{F}_{g}(s)\geq 0\) for all \(s\geq 0\). Condition (11) was introduced in [21].

Remark 2

If function \(f_{g}(s):=\frac{f(s)}{g(s)G(s)} \) is increasing in \((0,+\infty )\), then \(\mathscr{F}_{g}(s) \) is also increasing in \((0,+\infty )\). In fact, for \(s>t>0\), we get

$$\begin{aligned} {\mathscr{F}}_{g}(s)-{ \mathscr{F}}_{g}(t) = &\frac{f(s)G(s)}{2g(s)}- \frac{f(t)G(t)}{2g(t)}- \int _{t}^{s} f(\tau )d\tau \\ = & \int _{0}^{s}f_{g}(s)G(\tau )g(\tau )d \tau - \int _{0}^{t}f_{g}(t)G( \tau )g(\tau )d \tau - \int _{t}^{s}f_{g}(\tau )G(\tau )g(\tau )d \tau \\ = & \int _{t}^{s} \bigl[f_{g}(s)-f_{g}( \tau ) \bigr]G(\tau )g(\tau )d\tau + \int _{0}^{t} \bigl[f_{g}(s)- f_{g}(t) \bigr]G(\tau )g(\tau )d\tau \\ \geq & 0. \end{aligned}$$

Remark 3

Let \(F(s)=s^{\beta }\)\((\beta >2)\) or \(F(s)=\frac{1}{2}G^{2}(s)\log (1+s)\) for \(s>0\), then direct calculations show that (10) holds for \(\mu >2\).

The main results are the following:

Theorem 4

Assume that\(\kappa \geq 0\), \((V_{0})-(V_{1})\), \((f_{1})\)\((f_{3})\)and\((f_{4})\) (or\((f_{4}')\)) hold. Then Eq. (2) possesses a positive solution\(u\).

Theorem 5

Assume that\(-2<\kappa <0\), \((V_{0})-(V_{1})\), \((f_{1})\)\((f_{3})\), \((f_{4}'')\)hold. Then Eq. (2) possesses a positive solution\(u\).

As we will see later, the different monotonicity of \(g(s)\) for \(\kappa > 0\) and \(\kappa <0\) leads to opposite properties of \(G^{-1}(t)\) (the inverse function of \(G(t)\)). Thus, we should deal with them separately, particularly, the different test function will be used.

Finally, we remark that this paper also revises the wrong usage of Resonance and the Hahn-Banach Theorems to prove the boundedness of Cerami sequence in our recent paper [11].

Thought this paper, we denote \(X\) by the completion of the space \(C_{0}^{\infty }(\mathbb{R}^{N})\) with respect to the norm

$$ \|u\|= \biggl[ \int _{\mathbb{R}^{N}}\bigl(|\nabla u|^{2}+V(x)u^{2} \bigr)dx \biggr]^{ \frac{1}{2}}. $$

From \((V)\), \(X\) is equivalent to \(H^{1}(\mathbb{R}^{N})\).

\(C\) denotes positive (possibly different) constants.

2 Adjustment of the Variational Setting

We observe that (2) is the Euler-Lagrange equation associated with the energy functional

$$ I(u)= \frac{1}{2} \int _{\mathbb{R}^{N}} \bigl[g^{2}(u)|\nabla u|^{2}+ V(x)u ^{2} \bigr]dx- \int _{\mathbb{R}^{N}} F(u)dx, $$
(13)

where \(g(t)=\sqrt{1+\frac{\kappa t^{2}}{2(1+t^{2})}}\). As in [10], we make a change of variable as follows:

$$ v=G(u)= \int _{0}^{u}g(t)dt. $$

Then we can write \(I\) as:

$$ J(v)= \frac{1}{2} \int _{\mathbb{R}^{N}} \bigl[|\nabla v|^{2}+ V(x)|G ^{-1}(v)|^{2} \bigr]dx- \int _{\mathbb{R}^{N}} F\bigl(G^{-1}(v)\bigr)dx. $$
(14)

From Lemma 2.1 below, we see that \(J\) is well defined in \(X\) and by a standard argument, \(J\in C^{1}\).

Lemma 1

If\(u\in X\)is a critical point of\(I\), then\(v\in X\)is a critical point of\(J\), and vice versa.

Proof

If \(u\in X\) is a critical point of \(I\), then for \(\forall \varphi \in C^{\infty }_{0}(\mathbb{R}^{N})\), it satisfies

$$\begin{aligned} \bigl\langle I'(u),\varphi \bigr\rangle = \int _{\mathbb{R}^{N}} \bigl[g^{2}(u) \nabla u\nabla \varphi +g(u)g'(u)|\nabla u|^{2} \varphi + V(x) u \varphi -f(u) \varphi \bigr]dx=0. \end{aligned}$$
(15)

On the other hand, since \(u=G^{-1}(v)\), for \(\forall \psi \in C^{ \infty }_{0}(\mathbb{R}^{N})\), if we choose \(\varphi =\frac{1}{g(u)} \psi \) in (15), we get

$$\begin{aligned} \bigl\langle J'(v),\psi \bigr\rangle = \int _{\mathbb{R}^{N}} \biggl[\nabla v \nabla \psi + V(x)\frac{G^{-1}(v)}{g(G^{-1}(v))} \psi -\frac{f( G^{-1}(v))}{g(G ^{-1}(v))}\psi \biggr]dx=0. \end{aligned}$$
(16)

Conversely, if we let \(\psi =g(u)\varphi \) in (16), we get (15). □

From Lemma 2.1 and variational method, in order to find the nontrivial solutions of (2), it suffices to study the existence of the nontrivial solutions of the following equations

$$\begin{aligned} -\Delta v+ V(x)\frac{G^{-1}(v)}{g(G^{-1}(v))}=\frac{f(G^{-1}(v))}{g(G ^{-1}(v))},\quad x\in \mathbb{R}^{N}. \end{aligned}$$
(17)

Now, we collect some properties of \(g(s)\) and the change of variable \(G^{-1}(s)\).

Lemma 2

([11, Lemma 2.1])

  1. (1)

    \(g(s)\)is increasing for\(\kappa >0\)and is decreasing for\(-2<\kappa <0\)with respect to\(|s|\). Moreover,

    $$\begin{aligned} \min \biggl\{ 1, \sqrt{\frac{\kappa +2}{2}} \biggr\} \leq g(s)\leq \max \biggl\{ 1, \sqrt{\frac{\kappa +2}{2}} \biggr\} , \quad s\in \mathbb{R} \end{aligned}$$
    (18)

    and

    $$\begin{aligned} \min \biggl\{ 1, \sqrt{\frac{\kappa +2}{2}} \biggr\} \leq \bigg|\frac{s}{G ^{-1}(s)} \bigg|\leq \max \biggl\{ 1, \sqrt{\frac{\kappa +2}{2}} \biggr\} , \quad s\in \mathbb{R}, \end{aligned}$$
    (19)
  2. (2)

    \(\lim _{s\rightarrow 0}\frac{G^{-1}(s)}{s}=1\)for\(\kappa >-2\);

  3. (3)

    \(\lim _{s\rightarrow \infty }\frac{G^{-1}(s)}{s}=\sqrt{\frac{2}{ \kappa +2}}\)for\(\kappa >-2\);

  4. (4)

    \(|G^{-1}(s)'|\leq \max \{1, \sqrt{\frac{2}{\kappa +2}} \}\)for all\(s\in \mathbb{R}\).

Lemma 3

([11, Lemma 2.2])

For\(\kappa >-2\), there hold

$$ \min \biggl\{ 0, 1+\frac{4-2\sqrt{2(2+\kappa )}}{\kappa } \biggr\} \leq \sup_{s\in \mathbb{R}} \frac{sg'(s)}{g(s)}\leq \max \biggl\{ 0, 1+\frac{4-2\sqrt{2(2+ \kappa )}}{\kappa } \biggr\} $$

and

$$\begin{aligned} \min \biggl\{ 1, \frac{4+2\kappa -2\sqrt{2(2+\kappa )}}{\kappa } \biggr\} |s| \leq & \big| G^{-1}(s)g\bigl(G^{-1}(s)\bigr) \big| \\ \leq & \max \biggl\{ 1, \frac{4+2\kappa -2\sqrt{2(2+\kappa )}}{ \kappa } \biggr\} |s|,\quad s\in \mathbb{R}. \end{aligned}$$
(20)

3 Proof of Theorems

First, we establish the geometric hypotheses of the mountain pass theorem.

Lemma 1

Assume that\((V)\)and\((f_{1})\)\((f_{3})\).

  1. (1)

    There exist\(\rho _{0}, a_{0}>0\)such that\(J(v)\geq a _{0}\)for all\(\|v\|=\rho _{0}\).

  2. (2)

    There exists\(e\in X\)such that\(J(e)<0\).

Proof

(1) Let

$$ K(x,s):=-\frac{1}{2}V(x)G^{-1}(s)^{2}+F \bigl(G^{-1}(s)\bigr). $$

Then, by Lemma 2.2(2) and \((f_{1})\), we have

$$\begin{aligned} \lim _{s\rightarrow 0}\frac{K(x,s)}{s^{2}}=\lim _{s\rightarrow 0} \biggl[-\frac{1}{2}V(x) \biggl( \frac{G^{-1}(s)}{s} \biggr)^{2}+ \frac{F(G^{-1}(s))}{G^{-1}(s)^{2}} \cdot \frac{G^{-1}(s)^{2}}{s^{2}} \biggr]=-\frac{1}{2}V(x). \end{aligned}$$
(21)

From Lemma 2.2(3) and (\(f_{2}\)),

$$\begin{aligned} \lim _{s\rightarrow \infty }\frac{K(x,s)}{|s|^{p}}=\lim _{s\rightarrow \infty } \biggl[-\frac{1}{2}V(x) \biggl( \frac{G^{-1}(s)}{s} \biggr)^{2} \frac{1}{|s|^{p-2}} + \frac{F(G^{-1}(s))}{|s|^{p}} \biggr]=C_{1}, \end{aligned}$$
(22)

where \(C_{1}\) is a nonnegative constant. Thus, by (21) and (22), for \(\varepsilon >0\) sufficiently small, there exists a constant \(C_{\varepsilon }>0\) such that

$$ K(x,s)\leq \biggl[-\frac{1}{2}V(x)+\varepsilon \biggr]s^{2}+C_{\varepsilon }|s|^{p}. $$
(23)

Then, by (23), choosing \(\varepsilon =\frac{1}{2}V_{0}\), we have

$$\begin{aligned} J(v) \geq & \frac{1}{2} \int _{\mathbb{R}^{N}} \biggl[|\nabla v|^{2}+ \biggl(V(x)- \frac{1}{2}V_{0} \biggr)v^{2} \biggr]dx-C_{2} \int _{\mathbb{R}^{N}}|v|^{p}dx \\ \geq &\frac{1}{4}\|v\|^{2} -C_{3}\|v \|^{p}. \end{aligned}$$
(24)

Since \(p>2\), we prove (1) when \(\|v\|=\rho _{0}\) if choosing \(\rho _{0}\) small.

(2) We choose some \(\varphi \in C_{0}^{\infty }(\mathbb{R}^{N},[0,1])\). It follows from \((f_{3})\) and (19) that

$$\begin{aligned} J(t \varphi ) \leq &t^{2} \biggl[\frac{1}{2} \int _{\mathbb{R}^{N}} \bigl(| \nabla \varphi |^{2}+CV_{\infty } \varphi ^{2} \bigr)dx- \int _{\mathbb{R} ^{N}} \frac{F(G^{-1}(t\varphi ))}{G(G^{-1}(t\varphi ))^{2}}\varphi ^{2} dx \biggr] \\ \rightarrow & -\infty ,\quad \text{as } t\rightarrow \infty , \end{aligned}$$

which yields the result if we take \(e=t\varphi \) with \(t\) large enough. □

In consequence of Lemma 3.1 and of Ambrosetti-Rabinowitz Mountain Pass Theorem [23], for the constant

$$ c=\inf _{\gamma \in \varGamma }\sup _{t\in [0,1]}J\bigl(\gamma (t)\bigr)>0, $$

where \(\varGamma =\{\gamma \in C([0,1],X): \gamma (0)=0, \gamma (1) \neq0, J(\gamma (1))<0\}\), there exists a Cerami sequence at level \(c\), that is, \(J(v_{n})\rightarrow c\) and \((1+\|v_{n}\|)J'(v_{n}) \rightarrow 0\) as \(n\rightarrow \infty \).

Now, we prove that the Cerami sequence \(\{v_{n}\}\) is bounded. From \((f_{1})\), we may assume that \(v_{n}(x)\geq 0\) a.e. in \(\mathbb{R} ^{N}\). By contradiction, we assume that \(\|v_{n}\|\rightarrow +\infty \). We set \(w_{n}:=\frac{v_{n}}{\|v_{n}\|}\), then \(\|w_{n}\|=1\) and thus there exists a subsequence (still denote \(\{w_{n}\}\)) such that \(w_{n}\rightharpoonup w\) in \(X\), \(w_{n}\rightarrow w\) in \(L^{p}_{loc}( \mathbb{R}^{N})\) for \(p \in [1,2^{*})\). Moreover, up to a subsequence, \(\{w_{n}\}\) satisfies the alternative:

  1. (I)

    (non-vanishing) There exist \(\alpha >0\), \(R<+\infty \) and \(\{y_{n}\}\subset \mathbb{R}^{N}\) such that

    $$ \lim_{n\rightarrow \infty } \int _{B_{R}(y_{n})}w_{n}^{2}dx\geq \alpha >0. $$
  2. (II)

    (vanishing) For all \(R>0\), there holds

    $$ \lim_{n\rightarrow \infty }\sup_{y\in \mathbb{R}^{N}} \int _{B_{R}(y)}w _{n}^{2}dx=0. $$

We will prove that none of the two cases can occur, getting so the desired contradiction.

Lemma 2

([11, Lemma 4.1])

The non-vanishing of\(\{w_{n}\}\)is impossible.

Lemma 3

The vanishing of\(\{w_{n}\}\)is impossible.

Proof

(i) Suppose that \(\kappa >0\) and \((f_{1})\)\((f _{4})\).

If (II) holds, then by Lions’ Lemma [5], we get

$$\begin{aligned} \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}} |w_{n}|^{p}dx=0, \quad 2< p< 2^{*}. \end{aligned}$$
(25)

Define

$$ J(t_{n}v_{n}):=\max_{t\in [0,1]} J(tv_{n}). $$

For any \(m>0\), let \(\tilde{w}_{n}=2\sqrt{m}w_{n}=2\sqrt{m}\frac{v _{n}}{\|v_{n}\|}\). Then from \((f_{1})\), \((f_{2})\) and Lemma 2.2(1), (2), (3), we get

$$ \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}} F\bigl(G^{-1}(\tilde{w} _{n}) \bigr)dx=0. $$

Thus, for sufficiently large \(n\), from Lemma 2.2(1), we have

$$\begin{aligned} J(t_{n}v_{n})\geq J(\tilde{w}_{n})\geq \frac{4}{\kappa +2}m- \int _{\mathbb{R}^{N}} F\bigl(G^{-1}(\tilde{w}_{n}) \bigr)dx\geq \frac{8}{\kappa +2}m. \end{aligned}$$

Since \(m>0\) is arbitrary, we have \(\lim_{n\rightarrow \infty } J(t _{n}v_{n})=+\infty \).

On the other hand, from the definition of \(t_{n}\), we have

$$\begin{aligned} \bigl\langle J'(t_{n}v_{n}), t_{n}v_{n}\bigr\rangle = &t_{n} \frac{d}{dt}J(tv_{n}) |_{t=t_{n}} \\ = & \int _{\mathbb{R}^{N}} \biggl[|\nabla (t_{n} v_{n})|^{2}+V(x)\frac{G ^{-1}(t_{n}v_{n})t_{n}v_{n}}{g(G^{-1}(t_{n}v_{n}))} \biggr]dx \\ &{}- \int _{\mathbb{R}^{N}}\frac{f(G^{-1}(t_{n}v_{n}))t_{n}v_{n}}{g(G ^{-1}(t_{n}v_{n}))}dx \\ = &0. \end{aligned}$$
(26)

Recalling \(v_{n}(x)\geq 0\) a.e. in \(\mathbb{R}^{N}\) and \(t_{n}\in [0,1]\), thus \(t_{n} v_{n}(x)\leq v_{n}(x)\) a.e. in \(\mathbb{R}^{N}\). By \((f_{4})\), we get \(\mathscr{F}_{g}(G^{-1}(t_{n}v _{n})) \leq D\mathscr{F}_{g}(G^{-1}(v_{n}))\) a.e. in \(\mathbb{R}^{N}\). Consequently, it follows from (26) that

$$\begin{aligned} \begin{aligned} J(t_{n}v_{n})&= \int _{\mathbb{R}^{N}}\mathscr{F}_{g}\bigl(G^{-1}(t_{n}v_{n}) \bigr)dx -\frac{1}{2} \int _{\mathbb{R}^{N}}V(x) \biggl[\frac{G^{-1}(t_{n}v_{n})t _{n}v_{n}}{g(G^{-1}(t_{n}v_{n}))}-|{G^{-1}(t_{n}v_{n})}|^{2} \biggr]dx \\ &\leq D \int _{\mathbb{R}^{N}}\mathscr{F}_{g}\bigl(G^{-1}(v_{n}) \bigr)dx - \frac{1}{2} \int _{\mathbb{R}^{N}}V(x) \biggl[\frac{G^{-1}(t_{n}v_{n})t _{n}v_{n}}{g(G^{-1}(t_{n}v_{n}))}-|{G^{-1}(t_{n}v_{n})}|^{2} \biggr]dx . \end{aligned} \end{aligned}$$

Therefore, we have

$$\begin{aligned} c+o(1) = &J(v_{n})- \frac{1}{2}\bigl\langle J'(v_{n}),v_{n} \bigr\rangle \\ = &\frac{1}{2} \int _{\mathbb{R}^{N}}V(x) \biggl[|G^{-1}(v_{n})|^{2}- \frac{G ^{-1}(v_{n})v_{n}}{g(G^{-1}(v_{n}))} \biggr]dx+ \int _{\mathbb{R}^{N}} \mathscr{F}_{g}\bigl(G^{-1}(v_{n}) \bigr)dx \\ \geq &\frac{1}{D}J(t_{n}v_{n})+\frac{1}{2D} \int _{\mathbb{R}^{N}}V(x) \biggl[\frac{G^{-1}(t_{n}v_{n})t_{n}v_{n}}{g(G^{-1}(t_{n}v_{n}))}-| {G^{-1}(t_{n}v_{n})}|^{2} \biggr]dx \\ &{}-\frac{1}{2} \int _{\mathbb{R}^{N}}V(x) \biggl[\frac{G^{-1}(v_{n})v_{n}}{g(G ^{-1}(v_{n}))}-|{G^{-1}(v_{n})}|^{2} \biggr]dx. \end{aligned}$$
(27)

Let \(h(t)=\frac{G^{-1}(t)t}{g(G^{-1}(t))}-|{G^{-1}(t)}|^{2} \), then \(h(t)\leq 0\) for \(t\geq 0\) (note that \(G(t)\leq g(t)t\) for \(t\geq 0\) and \(\kappa >0\)) and

$$\begin{aligned} h'(t)=\frac{g(G^{-1}(t)) [t-g(G^{-1}(t))G^{-1}(t) ]-G^{-1}(t)tg'(G ^{-1}(t))}{g^{3}(G^{-1}(t))}\leq 0,\quad t \geq 0. \end{aligned}$$

Thus, from (27), we have

$$\begin{aligned} \begin{aligned} c+o(1)&\geq \frac{1}{D}J(t_{n}v_{n})+ \frac{1}{2D} \int _{\mathbb{R} ^{N}}V(x)\bigl[h(t_{n}v_{n})-h(v_{n}) \bigr]dx \\ &\rightarrow +\infty , \end{aligned} \end{aligned}$$

which is a contradiction.

(ii) Suppose that \(\kappa >0\) and \((f_{1})\)\((f_{3})\) and \((f_{4}')\).

By (19), we get

$$\begin{aligned} \begin{aligned} \int _{\mathbb{R}^{N}}\bigl[|\nabla v_{n}|^{2}+V(x)|G^{-1}(v_{n})|^{2} \bigr]dx \geq \frac{2}{\kappa +2}\|v_{n}\|^{2}. \end{aligned} \end{aligned}$$
(28)

Dividing \(J(v_{n})=c+o(1)\) by \(\|v_{n}\|^{2}\) in both sides and from (28), we get

$$\begin{aligned} \begin{aligned} \frac{c+o(1)}{\|v_{n}\|^{2}}\geq \frac{1}{\kappa +2}-\frac{1}{\|v_{n} \|^{2}} \int _{\mathbb{R}^{N}}F\bigl(G^{-1}(v_{n})\bigr)dx, \end{aligned} \end{aligned}$$
(29)

that is,

$$\begin{aligned} \begin{aligned} \lim_{n\rightarrow \infty } \int _{\mathbb{R}^{N}}\frac{F(G^{-1}(v_{n}))}{v _{n}^{2}}w_{n}^{2}dx \geq \frac{1}{\kappa +2}. \end{aligned} \end{aligned}$$
(30)

By \((f_{1})\) and Lemma 2.2(2), for any \(\varepsilon >0\), there exists \(\delta >0\) such that

$$\begin{aligned} \begin{aligned} \int _{\{x\in \mathbb{R}^{N}:0\leq G^{-1}(v_{n}(x))\leq \delta \}}\frac{F(G ^{-1}(v_{n}))}{v_{n}^{2}}w_{n}^{2}dx \leq \frac{\varepsilon }{4} \int _{\{x\in \mathbb{R}^{N}:0\leq G^{-1}(v_{n}(x))\leq \delta \}}w _{n}^{2}dx\leq \frac{\varepsilon }{4V_{0}}. \end{aligned} \end{aligned}$$
(31)

On the other hand, since \(\mathscr{F}_{g}(G^{-1}(v_{n}(x)))\geq 0\) a.e. in \(\mathbb{R}^{N}\), by (27), we get

$$\begin{aligned} \begin{aligned} c+o(1)\geq \int _{\mathbb{R}^{N}}\mathscr{F}_{g}\bigl(G^{-1}(v_{n}) \bigr)dx \geq \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}} \mathscr{F}_{g}\bigl(G^{-1}(v_{n}) \bigr)dx. \end{aligned} \end{aligned}$$
(32)

Thus, if (9) holds (we may assume that \(\delta \geq t_{0}\) since \(0<\frac{F(G^{-1}(v_{n}(x)))}{v_{n}^{2}(x)}\leq C\) for \(\delta \leq G^{-1}(v_{n}(x)) < t_{0}\)), by Lemma 2.2(1), (25) and (32), we have

$$\begin{aligned} \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}\frac{F(G ^{-1}(v_{n}))}{v_{n}^{2}}w_{n}^{2}dx \leq &C \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}| \mathscr{F}_{g}\bigl(G^{-1}(v_{n}) \bigr)|^{\frac{1}{\lambda }}w_{n}^{2}dx \\ \leq &C \biggl( \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}w_{n}^{\frac{2\lambda }{\lambda -1}}dx \biggr)^{\frac{\lambda -1}{ \lambda }} \\ \rightarrow &0, \end{aligned}$$
(33)

where \(\frac{2\lambda }{\lambda -1}\in (2,2^{*})\). If (10) holds, by (25) and (32), we have

$$\begin{aligned} & \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}\frac{F(G ^{-1}(v_{n}))}{v_{n}^{2}}w_{n}^{2}dx \\ &\quad \leq C \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}|v _{n}|^{p-2}w_{n}^{2}dx \\ &\quad \leq C \biggl( \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}|v_{n}|^{\mu }dx \biggr)^{\frac{p-2}{\mu }} \biggl( \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}w_{n}^{\frac{2 \mu }{\mu -p+2}}dx \biggr)^{\frac{\mu -p+2}{\mu }} \\ &\quad \leq C \biggl( \int _{\{x\in \mathbb{R}^{N}: G^{-1}(v_{n}(x))\geq \delta \}}w_{n}^{\frac{2\mu }{\mu -p+2}}dx \biggr)^{\frac{\mu -p+2}{\mu }} \\ &\quad \rightarrow 0, \end{aligned}$$
(34)

where \(\frac{2\mu }{\mu -p+2}\in (2,2^{*})\). Since \(\varepsilon >0\) is arbitrary, combining (32) and (33) (or (34)), we get a contradiction to (30).

(iii) Suppose that \(\kappa <0\) and \((f_{1})\)\((f_{3})\) and \((f_{4}'')\).

Let \(\psi _{n}=g(G^{-1}(v_{n}))G^{-1}(v_{n})\), from Lemmas 2.2 and 2.3, we get \(\psi _{n}\in X\). So, we choose \(\psi =\psi _{n}\) as a test function in \(\langle J'(v_{n}), \psi \rangle =o(1)\) to get

$$\begin{aligned} c+o(1) = &J(v_{n})- \frac{1}{2}\bigl\langle J'(v_{n}),\psi _{n}\bigr\rangle \\ = &- \int _{\mathbb{R}^{N}}\frac{G^{-1}(v_{n})g'(G^{-1}(v_{n}))}{g(G^{-1}(v _{n}))}|\nabla v_{n}|^{2}dx+ \int _{\mathbb{R}^{N}}\mathscr{F}_{g}(v _{n})dx \\ \geq & \int _{\mathbb{R}^{N}}\mathscr{F}_{g}(v_{n})dx. \end{aligned}$$
(35)

Now, by (11) and (12), analogous to the proof of (ii)–(33) and (34), we get the result. □

Thanks to Lemmas 3.2 and 3.3, we establish that the Cerami sequence \(\{v_{n}\}\) is bounded in \(X\). Up to a subsequence if necessary, we assume that there exists \(v\in X\) such that \(v_{n}\rightharpoonup v\) in \(X\), \(v_{n}\rightarrow v\) in \(L^{p}_{loc}( \mathbb{R}^{N})\) for \(p\in [1,2^{*})\) and \(v_{n}(x)\rightarrow v(x)\) a.e. in \(\mathbb{R}^{N}\). By [22], we have \(|v_{n}(x)|\leq |z(x)|\) for every \(n\) with \(z(x)\in L^{p}(\varOmega )\), where \(\varOmega :=\mathrm{supp} \psi \), \(\psi \in C_{0}^{\infty }(\mathbb{R}^{N})\). Consequently, from \((f_{1})\)\((f_{3})\), Lemma 2.2 and the Lebesgue Dominated Theorem, we have \(\langle J'(v_{n}),\psi \rangle +o(1)=\langle J'(v),\psi \rangle \). Thus, \(J'(v)=0\), which implies that \(v\) is a weak solution.

We next show \(v\not \equiv 0\). Since the proof is similar to [11], so we give a brief sketch of the proof here. We suppose, by contradiction, that \(v\equiv 0\). Then, under condition \((V)\), \(\{v_{n}\}\) is also a Cerami sequence for the functional \(J_{\infty }:X\rightarrow \mathbb{R}\) defined by

$$ J_{\infty }(v_{n})=\frac{1}{2} \int _{\mathbb{R}^{N}} \bigl[|\nabla v _{n}|^{2}+V_{\infty }|G^{-1}(v_{n})|^{2} \bigr]dx- \int _{\mathbb{R}^{N}} F\bigl(G^{-1}(v_{n})\bigr)dx. $$

Moreover, \(J_{\infty }(v_{n})= c+o(1)\).

Next, we claim that the vanishing for \(\{v_{n}\}\) cannot occur. Suppose by contradiction, \(\{v_{n}\}\) vanish, then by Lions’ Lemma [5], \((f_{1})\) and \((f_{2})\), we have

$$\begin{aligned} \lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}}\frac{f(G^{-1}(v _{n}))}{g(G^{-1}(v_{n}))}v_{n} dx=0 \end{aligned}$$
(36)

and

$$\begin{aligned} \lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} F\bigl(G^{-1}(v _{n}) \bigr)dx=0. \end{aligned}$$
(37)

Then, by Lemma 2.3, we have

$$\begin{aligned} 0 = &\lim _{n\rightarrow \infty }\bigl\langle J'(v_{n}),v_{n}\bigr\rangle \\ = &\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \biggl[| \nabla v_{n}|^{2}+ V(x)\frac{G^{-1}(v_{n})}{g(G^{-1}(v_{n}))}v_{n} -\frac{f(G ^{-1}(v_{n}))}{g(G^{-1}(v_{n}))}v_{n} \biggr]dx \\ = &\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \biggl[| \nabla v_{n}|^{2}+ V(x)\frac{G^{-1}(v_{n})}{g(G^{-1}(v_{n}))}v_{n} \biggr]dx \\ \geq &\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \biggl[| \nabla v_{n}|^{2}+\min \biggl\{ 1,\frac{\kappa }{4+2\kappa -2\sqrt{2(2+ \kappa )}}\biggr\} V(x)|G^{-1}(v_{n})|^{2} \biggr]dx . \end{aligned}$$
(38)

Thus,

$$\begin{aligned} \lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \bigl[|\nabla v_{n}|^{2}+ V(x)|G^{-1}(v_{n})|^{2} \bigr]dx=0. \end{aligned}$$
(39)

Combining (37) and \(\text{(39)}\), we get \(J(v_{n})\rightarrow 0\), which is a contradiction since \(J(v_{n}) \rightarrow c>0\). Thus, \(\{v_{n}\}\) does not vanish and there exist \(\alpha , R>0\), and \(\{y_{n}\}\subset \mathbb{R}^{N}\) such that

$$ \lim _{n\rightarrow \infty } \int _{B_{R}(y_{n})}v_{n}^{2}dx \geq \alpha >0. $$
(40)

Define \(\widetilde{v}_{n}(x)=v_{n}(x+y_{n})\). Since \(\{v_{n}\}\) is a Cerami sequence for \(J_{\infty }\), \(\{\widetilde{v}_{n}\}\) is also a Cerami sequence for \(J_{\infty }\). Arguing as in the case of \(\{v_{n}\}\), up to a subsequence, we get that \(\widetilde{v}_{n} \rightharpoonup \widetilde{v}\) in \(X\) with \(J_{\infty }'( \widetilde{v})=0\). By (40), we have \(\widetilde{v}\neq0\). Therefore, if \(\kappa \geq 0\), using Fatou’s Lemma, we deduce that

$$\begin{aligned} 2c = &\lim _{n\rightarrow \infty }\bigl[2 J_{\infty }(\widetilde{v} _{n})-\bigl\langle J_{\infty }'( \widetilde{v}_{n}),\widetilde{v}_{n}\bigr\rangle \bigr] \\ = &\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}}V(x)G^{-1}( \widetilde{v}_{n}) \biggl[ G^{-1}(\widetilde{v}_{n})-\frac{1}{g(G^{-1}( \widetilde{v}_{n}))} \widetilde{v}_{n} \biggr]dx+2\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}} \mathscr{F}_{g}\bigl(G ^{-1}(v_{n})\bigr) dx \\ \geq & \int _{\mathbb{R}^{N}}V(x)G^{-1}(\widetilde{v}) \biggl[ G^{-1}( \widetilde{v})-\frac{1}{g(G^{-1}(\widetilde{v}))}\widetilde{v} \biggr]dx+2 \int _{\mathbb{R}^{N}}\mathscr{F}_{g}\bigl(G^{-1}(v) \bigr)dx \\ = &2 J_{\infty }(\widetilde{v})-\bigl\langle J_{\infty }'( \widetilde{v}), \widetilde{v}\bigr\rangle \\ = &2 J_{\infty }(\widetilde{v}). \end{aligned}$$

If \(-2<\kappa <0\), let \(H(x,v,\nabla v)=-\frac{G^{-1}(v)g'(G^{-1}(v))}{g(G ^{-1}(v))}|\nabla v|^{2}\), we consider the functional

$$ \phi (v):= \int _{\mathbb{R}^{N}}H(x,v,\nabla v)dx. $$

Since \(g'(v)\leq 0\) for \(v\geq 0\), we get \(H(x,v,p)\geq 0\). Moreover, \(H(x,v,\cdot )\) is convex in \(p\). Thus, by Theorem 1.6 in [24], \(\phi (v)\) is lower semi-continuous. Choosing \(\psi =G ^{-1}(v_{n})g(G^{-1}(v_{n}))\) as a test function, we have

$$\begin{aligned} 2c = &\lim _{n\rightarrow \infty }\bigl[2 J_{\infty }(\widetilde{v} _{n})-\bigl\langle J_{\infty }'( \widetilde{v}_{n}),G^{-1}(v_{n})g \bigl(G^{-1}( \widetilde{v}_{n})\bigr)\bigr\rangle \bigr] \\ = &-\lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}}\frac{G^{-1}(v _{n})g'(G^{-1}(v_{n}))}{g(G^{-1}(v_{n}))}|\nabla v_{n}|^{2}dx+2 \lim _{n\rightarrow \infty } \int _{\mathbb{R}^{N}}\mathscr{F}\bigl(G ^{-1}(v_{n}) \bigr)dx \\ \geq &- \int _{\mathbb{R}^{N}} \frac{G^{-1}(v)g'(G^{-1}(v))}{g(G^{-1}(v))}|\nabla v|^{2}+2 \int _{\mathbb{R}^{N}}\mathscr{F}\bigl(G^{-1}(v)\bigr)dx \\ = &2 J_{\infty }(\widetilde{v})-\bigl\langle J_{\infty }'( \widetilde{v}),G ^{-1}(v)g\bigl(G^{-1}(\widetilde{v})\bigr) \bigr\rangle \\ = &2 J_{\infty }(\widetilde{v}). \end{aligned}$$

Thus, \(\widetilde{v}\) is a critical point of \(J_{\infty }\) satisfying \(J_{\infty }(\widetilde{v})\leq c\). By the strong Maximum Principle, \(\widetilde{v}>0\) on \(\mathbb{R}^{N}\). We stress that if \(V(x)\equiv V_{\infty }\) for \(x\in \mathbb{R}^{N}\), the proof is done.

Now, we assume that \(V(x)\leq V_{\infty }\), \(V(x)\not \equiv V_{ \infty }\) for \(x\in \mathbb{R}^{N}\). It follows from the arguments in [15], we get a path \(\gamma :[0,1]\rightarrow X\) such that

$$\begin{aligned} \textstyle\begin{cases} \gamma (0)=0,\quad J_{\infty }(\gamma (1))< 0, \quad \widetilde{v}\in \gamma ([0,1]), \\ \gamma (t)(x)>0,\quad \forall x\in \mathbb{R}^{N},\quad t\in [0,1], \\ \max _{t\in [0,1]}J_{\infty }(\gamma (t))=J_{\infty }( \widetilde{v}). \end{cases}\displaystyle \end{aligned}$$
(41)

Let \(\varGamma _{\infty }=\{\gamma \in C([0,1],X): \gamma (0)=0, \gamma (1) \neq0, J_{\infty }(\gamma (1))<0\}\) and taking the path \(\gamma \) given by (41). Since \(\gamma \in \varGamma _{\infty }\subset \varGamma \), we have

$$ c\leq \max _{t\in [0,1]}J\bigl(\gamma (t)\bigr):=J\bigl(\gamma (\bar{t})\bigr)< J _{\infty }\bigl(\gamma (\bar{t})\bigr)\leq \max _{t\in [0,1]}J_{\infty } \bigl( \gamma (t)\bigr)=J_{\infty }(\widetilde{v})\leq c, $$

which is a contradiction and the result is proved. Finally, \(v\) is a positive solution by the strong Maximum Principle.