1 Introduction

Consider the following kind of Klein–Gordon–Maxwell system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+V(x)u-(2\omega +\phi )\phi u=f(x,u),&{}\quad x\in {{\mathbb {R}}}^{3},\\ \Delta \phi =(\omega +\phi )u^{2},&{}\quad x\in {{\mathbb {R}}}^{3}, \end{array}\right. \end{aligned}$$
(1.1)

where \(\omega >0\) is a constant, \(V:{{\mathbb {R}}}^{3}\rightarrow {{\mathbb {R}}}\), \(\phi , u:{{\mathbb {R}}}^{3}\rightarrow {{\mathbb {R}}}\), \(f:{{\mathbb {R}}}^{3}\times {{\mathbb {R}}}\rightarrow {{\mathbb {R}}}\). We assume that the following basic conditions hold:

  1. (A1)

    \(f\in C({{\mathbb {R}}}^{3}\times {{\mathbb {R}}},{{\mathbb {R}}})\), and there are constants \(p\in (2,6)\) and \(c_{0}>0\) such that

    $$\begin{aligned} |f(x,t)| \le c_{0}(1+|t|^{p-1}),\quad \forall \ (x,t)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}; \end{aligned}$$
  2. (A2)

    \(f(x,t)/|t|\rightarrow 0\) as \(|t|\rightarrow 0\) uniformly in \(x\in {{\mathbb {R}}}^{3}\), and \(F(x,t)\ge 0\) for all \((x,t)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}\), where \(F(x,t):=\int _{0}^{t}f(x,s)ds\).

Benci and Fortunato [1] first introduced the Klein–Gordon–Maxwell (we use KGM for short from now on) equations to simulate the Klein–Gordon equation interacting with the electromagnetic field. Specifically speaking, the model represents standing waves \(\psi =u(x)e^{iwt}\) in equilibrium with a purely electrostatic field E\(=-\nabla \phi (x)\), where \(\phi \) is the gauge potential. By applying a well known equivariant version of mountain pass theorem, Benci and Fortunato [1, 2] first studied the following special KGM system with constant potential \(m_{0}^{2}-\omega ^{2}\),

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+\left[ m_{0}^{2}-(\omega +\phi )^{2}\right] u=|u|^{q-2}u,&{}\quad x\in {{\mathbb {R}}}^{3},\\ \Delta \phi =(\omega +\phi )u^{2},&{}\quad x\in {{\mathbb {R}}}^{3}, \end{array}\right. \end{aligned}$$
(1.2)

where \(q\in (4,6)\), \(m_{0}>0\) and \(\omega >0\) are constants. When \(|\omega |<|m_{0}|\) and \(q\in (4,6)\), Benci and Fortunato acquired the existence and multiplicity of solitary wave solutions for system (1.2).

In [3], D’Aprile and Mugnai also obtained multiplicity of solitary wave solutions for system (1.2) if one of the following assumptions is satisfied:

  1. (i)

    \(0<\omega <\sqrt{(q-2)/2}m_{0}\) and \(q \in (2, 4)\);

  2. (ii)

    \(q \in (4, 6)\) and \(0<\omega <m_{0}\).

    The results obtained by D’Aprile and Mugnai filled the gap for \(q\in (2, 4)\). In [4], by a Pohozaev-type argument, nonexistence of nontrivial solution of system (1.2) for \(0<q\le 2\) or \(q\ge 6\) is established by D’Aprile and Mugnai. Afterwards, by minimizing the functional of system (1.2), a least energy solution of system (1.2) was obtained by Azzollini and Pomponio [5] if one of the following assumptions is satisfied:

  3. (iii)

    \(q \in (4, 6)\) and \(0<\omega <m_{0}\);

  4. (iv)

    \(q \in (2, 4)\) and \(0<\omega <a_{1}(q)m_{0}\), where \(a_{1}(q)=\sqrt{(q-2)/(6-q)}\).

Later, the existence range of \((m_{0},\omega )\) for \(q\in (2, 4)\) was improved by Azzollini, Pisani and Pomponio [6] as follows:

$$\begin{aligned} 0<\omega <m_{0}a_{2}(q)\ \ \text{ with } \ \ a_{2}(q) =\left\{ \begin{array}{ll} \sqrt{(q-2)(4-q)} &{}\quad \text{ if }\ \ q\in (2, 3),\\ 1&{}\quad \text{ if }\ \ q\in [3, 4). \end{array}\right. \end{aligned}$$
(1.3)

In [7], Wang also obtained similar existence result by relaxing the range of \((m_{0},\omega )\) for \(q\in (2, 4)\) as follows:

$$\begin{aligned} 0<\omega <m_{0}a_{3}(q)\ \ \text{ with } \ \ a_{3}(q)=\sqrt{\frac{4(q-2)}{(4-q)^{2}+4(q-2)}}. \end{aligned}$$
(1.4)

It is easy to see that \(a_{2}(q)\) is larger than \(a_{3}(q)\), so the range of \(\omega \) in (1.3) is wider than that in (1.4). If system (1.2) is added by a lower order perturbation, in the year 2004, Cassani [8] studied this kind of KGM system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+\left[ m_{0}^{2}-(\phi +\omega )^{2}\right] u=\mu |u|^{q-2}u+|u|^{4}u,&{}\quad x\in {{\mathbb {R}}}^{3},\\ \Delta \phi =(\phi +\omega )u^{2},&{}\quad x\in {{\mathbb {R}}}^{3}, \end{array}\right. \end{aligned}$$
(1.5)

where \(m_{0}, \mu >0\) and \(q\in [4, 6)\). The author proved that : (1) system (1.5) has trivial solution when \(q=6\); (2) system (1.5) has at least a radial symmetric solution when \(q\in (4,6)\) and \(0<\omega <m_{0}\); (3) system (1.5) admits a nontrivial solution when \(q=4\) and \(\mu \) is large enough. Soon, in [9], Wang considered a kind of nonlinear KGM system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+\left[ m_{0}^{2}-(e\phi -\omega )^{2}\right] u=\mu |u|^{q-2}u+|u|^{4}u,&{}\quad x\in {{\mathbb {R}}}^{3},\\ \Delta \phi =e(e\phi -\omega )u^{2},&{}\quad x\in {{\mathbb {R}}}^{3}, \end{array}\right. \end{aligned}$$
(1.6)

where \(m_{0}, e, \omega , \mu >0\) and \(q\in (2, 6)\). By studying system (1.6) on the constraint space \(H^{1}_{r}({{\mathbb {R}}}^{3})=\{u\in H^{1}({{\mathbb {R}}}^{3}):u(x)=u(|x|)\}\) and by applying the reduction method, Wang proved that system (1.6) has at least a radially symmetric nontrivial solution.

In [10], Carriao et al. investigated the following KGM system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+V(x)u-(2\omega +\phi )\phi u=\mu |u|^{q-2}u+|u|^{4}u,&{}\quad x\in {{\mathbb {R}}}^{3},\\ \Delta \phi =(\phi +\omega )u^{2},&{}\quad x\in {{\mathbb {R}}}^{3}, \end{array}\right. \end{aligned}$$
(1.7)

where \(\omega , \mu >0\), \(q\in (2, 6)\) and V(x) is periodic potential. By minimizing the corresponding functional associated with problem (1.7) on some Nehari manifold with the so called Brézis-Nirenberg technique, Carriao, Cunha and Miyagaki obtained that problem (1.7) possesses positive ground state solutions. Later, Chen et al. [11] improved the results in [8, 10] under weaker conditions.

In [12], Colin and Watanabe investigated the following type of KGM system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta u+\left[ m_{0}^{2}-(e\phi -\omega )^{2}\right] u=|u|^{q-2}u,&{}\quad x\in {{\mathbb {R}}}^{3},\\ \Delta \phi =-e(e\phi +\omega )u^{2},&{}\quad x\in {{\mathbb {R}}}^{3}, \end{array}\right. \end{aligned}$$
(1.8)

where \(m_{0}>0\), \(e\in {{\mathbb {R}}}\), \(\omega \in {{\mathbb {R}}}\), and \(q\in (2, 6)\). Unlike the results aforementioned, Colin and Watanabe did not reduce the functional associated with (1.8) to a single variable action and did not consider the minimization problem on the Nehari manifold, so the result obtained by them requires no restriction on q and \(\omega \).

Replacing the nonlinear term \(|u|^{q-2}u\) by a more general function f(u), Benci and Fortunato [13] studied system (1.2) with nonlinear term f(u) and established the existence of three dimensional vortex solutions under suitable conditions. If a solitary wave \(\psi \) satisfies a non-vanishing angular momentum, it is called a vortex. Later, Mugnai and Rinaldi [14] studied the existence of cylindrically symmetric electro-magneto-static solitary waves for (1.2) with a positive mass and a nonnegative nonlinear potential. They also obtained nonexistence results. The results obtained in [14] improve the results in [13]. As point out in [13], the nonlinear KGM equations are the models for the interaction between the matter and the electromagnetic field. For more physical background, please see [13, 14]. For more related results of KGM equations, we refer the readers to [15,16,17,18,19,20,21,22,23,24] and the references therein.

Using symmetric mountain pass theorem and variant fountain theorem in critical point theory, the multiplicity of solutions for (1.1) were first obtained by He [25] if (A1) and the following conditions hold:

  1. (V1’)

    \(V\in C({{\mathbb {R}}}^{3},{{\mathbb {R}}})\), \(\inf _{{{\mathbb {R}}}^{3}}V(x)>0\) and there exists a constant \(r>0\) such that

    $$\begin{aligned} \lim _{|y|\rightarrow \infty }\text{ meas }\{x\in {{\mathbb {R}}}^{3}:|x-y|\le r, V(x)\le M\}=0, \ \forall \ M>0; \end{aligned}$$
  2. (A0)

    \(f(x,t)=-f(x,-t),\ \forall \ (x,t)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}\);

  3. (AR)

    there exists \(\mu >4\) such that

    $$\begin{aligned} \mu F(x,t)\le f(x,t)t,\ \forall \ (x,t)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}; \end{aligned}$$

    or

  4. (AR’)

    \(\lim _{|t|\rightarrow \infty }\frac{f(x,t)}{|t|^{3}}=+\infty \) uniformly in \(x\in {{\mathbb {R}}}^{3}\), and

    $$\begin{aligned} f(x,t)t-4F(x,t)\rightarrow \infty \ \ \text{ as } \ |t|\rightarrow \infty \ \ \text{ uniformly } \text{ in } \ x\in {{\mathbb {R}}}^{3}. \end{aligned}$$

Condition (AR) or (AR’) is very important since it plays a role both in achieving the mountain pass geometry of the functional associated with system (1.1) and in obtaining the boundedness of Palais-Smale (PS) sequence or Cerami sequence. In the recent years, many authors devoted to replacing (AR) (or (AR’)) and (V1’) by weaker conditions. For example, Ding and Li [26] and Li and Tang [27] used the following weaker conditions instead of (V1’), (AR) and (AR’) to investigate system (1.1):

  1. (V1)

    \(V\in C({{\mathbb {R}}}^{3},{{\mathbb {R}}})\), \(\inf _{{{\mathbb {R}}}^{3}}V(x)>-\infty \) and there exists a constant \(r>0\) such that

    $$\begin{aligned} \lim _{|y|\rightarrow \infty }\text{ meas }\{x\in {{\mathbb {R}}}^{3}:|x-y|\le r, V(x)\le M\}=0, \ \forall \ M>0; \end{aligned}$$
  2. (SQ)

    \(\lim _{|t|\rightarrow \infty }\frac{F(x,t)}{|t|^{4}}=+\infty \) uniformly in \(x\in {{\mathbb {R}}}^{3}\);

  3. (SQ’)

    there is \(\theta _{1}\ge 0\) such that

    $$\begin{aligned} f(x,t)t-4F(x,t)+\theta _{1}t^{2}\ge 0,\ \ \text{ uniformly } \text{ in } \ (x,t)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}. \end{aligned}$$

Recently, Chen and Tang [28] used the following weaker conditions to relax (AR), (AR’), (SQ) and (SQ’):

  1. (A3’)
    $$\begin{aligned} \lim _{|t|\rightarrow \infty }\frac{F(x,t)}{|t|^{2}}=+\infty \ \ \text{ uniformly } \text{ in }\ x\in {{\mathbb {R}}}^{3}, \end{aligned}$$
    (1.9)

    and there is \(r_{1}>0\) such that \(F(x,t)\ge 0\), \(\forall \ x\in {{\mathbb {R}}}^{3}, \ |t|\ge r_{1}\);

  2. (A3”)

    there exists \(\mu >2\) and \(\theta >0\) such that

    $$\begin{aligned} f(x,t)t-\mu F(x,t)+\theta t^{2}\ge 0, \ \ \forall \ (x,t)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}. \end{aligned}$$
    (1.10)

It is well known that (1.9) was first introduce by Liu and Wang [29]. Subsequently, it has been commonly used in obtaining nontrivial solutions for system (1.2) in all literature. Recently, Tang, Lin and Yu [30] used the following local super-quadratic condition to study Schrödinger equation.

  1. (A3)

    there exists a domain \(A\subset {{\mathbb {R}}}^{3}\) such that

    $$\begin{aligned} \lim _{|t|\rightarrow \infty }\frac{F(x,t)}{|t|^{2}}=+\infty \ \ \text{ a.e. }\ x\in A. \end{aligned}$$
    (1.11)

(1.11) is also used in the very recent paper [31] for seeking the existence of ground state solutions and infinitely many geometrically distinct solutions for a kind of Schrödinger equations. For more new works about Schrödinger equation, please see [32, 33] and references therein.

As is known, KGM system is different from the Schrödinger equation because of the presence of the solitary wave \(\psi =u(x)e^{iwt}\) in equilibrium which linked with a purely electrostatic field E\(=-\nabla \phi (x)\), that is the term \((2\omega +\phi )\phi u\) presented in KGM system. The appearance of \((2\omega +\phi )\phi u\) brings some difficulties not only in showing the link geometry of the functional of system (1.1) but also in verifying the boundedness of Cerami sequences. A natural question is whether the local super-quadratic condition is applicable for system (1.1). The purpose of this paper is to solve this problem. We will generalize and improve the results which obtained in [25,26,27,28] in another direction under (A3) and other conditions. To state our conclusions, in addition to (A1)-(A3) and (V1), we also need the following conditions:

  1. (V)

    \(V\in C({{\mathbb {R}}}^{3}, {{\mathbb {R}}})\), V(x) is 1-periodic in each of \(x_{1}\), \(x_{2}\), \(x_{3}\) and

    $$\begin{aligned} \sup [\sigma (-\Delta +V)\cap (-\infty ,0)]<0<\Theta :=\inf [\sigma (-\Delta +V)\cap (0,\infty )]; \end{aligned}$$
  2. (A4)

    there exists a constant \(c_{1}>0\) such that \({\mathscr {F}}(x,t):=\frac{1}{2}f(x,t)t-F(x,t)\ge \left( \frac{\omega ^{2}}{8}+c_{1}\right) t^{2}\ge 0\), and there are \(c_{2}>0\), \(\delta _{0}\in (0,\Theta )\) and \(\varrho \in (0,1)\) such that

    $$\begin{aligned} \frac{f(x,t)}{t}\ge \Theta -\delta _{0}\ \ \text{ implies } \ \left[ \frac{|f(x,t)|}{|t|^{\varrho }}\right] ^{\frac{6}{5-\varrho }}\le c_{2}{\mathscr {F}}(x,t); \end{aligned}$$
  3. (A5)

    \({\mathscr {F}}(x,t)\ge \left( \frac{\omega ^{2}}{8}+c_{1}\right) t^{2}\ge 0\), and there are \(c_{3}>0\), \(R_{0}>0\) and \(\varrho \in (0,1)\) such that

    $$\begin{aligned} \left[ \frac{|f(x,t)|}{|t|^{\varrho }}\right] ^{\frac{6}{5-\varrho }}\le c_{3}{\mathscr {F}}(x,t),\ \ |t|\ge R_{0}; \end{aligned}$$

We state the following two main theorems.

Theorem 1.1

Suppose that (V), (A1)–(A4) hold. Assume that f(x, t) is 1-periodic in \(x_{1}\), \(x_{2}\) and \(x_{3}\). Then problem (1.1) has at least one nontrivial solution.

Theorem 1.2

Suppose that (V1), (A1)–(A3) and (A5) hold. Then problem (1.1) has at least one nontrivial solution.

Remark 1.3

We must point out that (A3), (A4) and (A5) are used to obtain nontrivial solutions for Schrödinger equation in [30, 31]. In this paper, the periodic case and non-periodic case for KGM systems are investigated, respectively. As far as we known, there are only three papers [10, 18, 28] considering the periodic case for KGM systems. Besides, the potential V(x) is allowed to be sign-changing. From this point, the results in this paper seem new. When V is periodic and \(f(x,t)\equiv f(t)\) satisfies some other super-linear conditions, Cunha [18] obtained the existence of a least energy solution for system (1.1).

Remark 1.4

It is pointed out that (A3) and (A4) (or (A5)) are much weaker than (AR), (AR’), (SQ), (SQ’), (A3’) and (A3”). (A3) is said to be local super-quadratic condition. As far as we known, it is first used by Tang et. al. [30] to obtain nontrivial solutions for Schrödinger equation. Tang et. al used new skills to conquer the difficulties arose in proving the existence of solutions for the functional of Schrödinger equation under the local super-quadratic condition. Following the strategy of [30], in the present paper, we use (A3) and other suitable conditions to obtain nontrivial solutions for KGM systems.

Now, we give two examples which satisfy (A3), (A4) and (A5), but not (AR), (AR’), (SQ), (SQ’), (A3’) and (A3”).

Example 1.5

Let \(F(x,t)=\frac{\omega ^{2}+1}{2}[|\cos (2\pi x_{1})|+\cos (2\pi x_{1})]t^{2}\ln (e+t^{2})\). Then

$$\begin{aligned} f(x,t)= & {} \frac{\omega ^{2}+1}{2}\left[ |\cos (2\pi x_{1})|+\cos (2\pi x_{1})\right] \left[ 2t\ln (e+t^{2})+\frac{2t^{3}}{e+t^{2}}\right] ,\\ {\mathscr {F}}(x,t)= & {} \frac{(\omega ^{2}+1)|t|^{4}[|\cos (2\pi x_{1})|+\cos (2\pi x_{1})]}{2(e+t^{2})}. \end{aligned}$$

It is not difficult to see that f satisfies (A1)-(A5) with \(0<\varrho <1\) and \(A=(-1/6,1/6)\times {{\mathbb {R}}}^{2}\), but f does not satisfy any of (AR), (AR’), (SQ), (SQ’), (A3’) and (A3”).

Example 1.6

Let \({\mathcal {B}}\) be a closed set of \({{\mathbb {R}}}^{3}\), and \(F(x,t)=\frac{\omega ^{2}+a}{2}\left[ 2-\frac{1}{\ln (e+t^{2})}\right] |t|^{2+b(x)}\), where \(a>0\) is a constant, \(b\in C({{\mathbb {R}}}^{3}, {{\mathbb {R}}})\), \(b(x)=0\) for \(x\in {\mathcal {B}}\) and \(0<b(x)<2\) for \(x\in {{\mathbb {R}}}^{3}\setminus {\mathcal {B}}\). Then,

$$\begin{aligned} f(x,t)= & {} \frac{(\omega ^{2}+a)(2+b(x))}{2}|t|^{b(x)}t\left[ 2-\frac{1}{\ln (e+t^{2})}\right] +\frac{(\omega ^{2}+a)|t|^{2+b(x)}t}{(e+t^{2})[\ln (e+t^{2})]^{2}},\\ {\mathscr {F}}(x,t)= & {} \frac{(\omega ^{2}+a)b(x)|t|^{2+b(x)}}{4}\left[ 2-\frac{1}{\ln (e+t^{2})}\right] +\frac{(\omega ^{2}+a)|t|^{4+b(x)}}{2(e+t^{2})[\ln (e+t^{2})]^{2}}. \end{aligned}$$

It is not difficult to see that f satisfies (A1)-(A5) with \(0<\varrho <1\) and \(A\subset {\bar{A}}\subset {{\mathbb {R}}}^{3}\backslash {\mathcal {B}}\), but neither of (AR), (AR’), (SQ), (SQ’), (A3’) and (A3”). Moreover, f(x, t) is allowed to be asymptotically linear when \(x\in {\mathcal {B}}\) and to be super-linear when \(x\in {{\mathbb {R}}}^{3}\backslash {\mathcal {B}}\)

The remainder of this paper is organized as follows. We present the variational setting for system (1.1) and give some preliminaries in the next section. The proof of Theorem 1.1 is given in Sect. 3 and the proof of Theorem 1.2 is given in Sect. 4. In the following, for convenient, \(C_{i}(i=1,2,\ldots )\) are different positive constants in different places.

2 The Variational Setting and Preliminary Results

Let \({\mathcal {A}}=-\Delta +V\). Then \({\mathcal {A}}\) is self-adjoint in \(L^{2}({{\mathbb {R}}}^{3})\) with domain \({\mathcal {D}}=H^{1}({{\mathbb {R}}}^{3})\) (see [34], Theorem 4.26). Let \(\{\xi (\lambda ):-\infty \le \lambda \le +\infty \}\) be the spectral family of \({\mathcal {A}}\), and \(|{\mathcal {A}}|\) is the absolute value of \({\mathcal {A}}\). \(|{\mathcal {A}}|^{1/2}\) denotes the square root of \(|{\mathcal {A}}|\). Set \({\mathcal {U}}=id-\xi (0)-\xi (0_{-})\). Then, \({\mathcal {U}}\) commutes with \({\mathcal {A}}\), \(|{\mathcal {A}}|\) and \(|{\mathcal {A}}|^{1/2}\), and \({\mathcal {A}}={\mathcal {U}}|{\mathcal {A}}|\) is the polar decomposition of \({\mathcal {A}}\) (see [35], Theorem IV 3.3). Let

$$\begin{aligned} E={\mathcal {D}}(|{\mathcal {A}}|^{1/2}), \ E^{-}=\xi (0_{-})E,\ E^{0}=[\xi (0)-\xi (0_{-})]E,\ E^{+}=[id-\xi (0)]E. \end{aligned}$$
(2.1)

For any \(u\in E\), one has \(u=u^{-}+u^{0}+u^{+}\), where

$$\begin{aligned} u^{-}:=\xi (0_{-})u\in E^{-}, \ u^{0}:=[\xi (0)-\xi (0_{-})]u\in E^{0},\ u^{+}:=[id-\xi (0)]u\in E^{+}, \end{aligned}$$
(2.2)

and

$$\begin{aligned} {\mathcal {A}}u=-|{\mathcal {A}}|u,\ \forall \ u\in E^{-};\ {\mathcal {A}}u=0,\ \forall \ u\in E^{0}; {\mathcal {A}}u=|{\mathcal {A}}|u,\ \forall \ u\in E^{+}\cap {\mathcal {D}}({\mathcal {A}}). \end{aligned}$$
(2.3)

Define an inner product

$$\begin{aligned} (u,v)=(|{\mathcal {A}}|^{1/2}u,|{\mathcal {A}}|^{1/2}v)_{L^{2}}+(u^{0},z^{0})_{L^{2}}, \ \ \forall \ u,v\in E, \end{aligned}$$
(2.4)

and the corresponding norm is

$$\begin{aligned} \Vert u\Vert =(\Vert |{\mathcal {A}}|^{1/2}u\Vert ^{2}_{2}+\Vert u^{0}\Vert ^{2}_{2})^{1/2},\ \ \forall \ u\in E, \end{aligned}$$
(2.5)

where \((\cdot ,\cdot )_{L^{2}}\) is the inner product of \(L^{2}({{\mathbb {R}}}^{3})\), \(\Vert \cdot \Vert _{s}\) denote the norm of \(L^{s}({{\mathbb {R}}}^{3})\), \(2\le s\le 6\). Since \(E=H^{1}({{\mathbb {R}}}^{3})\) with equivalent norms under (V) and \(E\subset H^{1}({{\mathbb {R}}}^{3})\) under (V1), E embeds continuously in \(L^{s}({{\mathbb {R}}}^{3})\) for all \(s\in [2, 6]\), hence there is a constant \(\gamma _{s}>0\) such that

$$\begin{aligned} \Vert u\Vert _{s}\le \gamma _{s}\Vert u\Vert ,\ \ \forall \ u\in E. \end{aligned}$$
(2.6)

We have the orthogonal decomposition \(E=E^{-}\oplus E^{0}\oplus E^{+}\) with respect to both \((\cdot ,\cdot )_{L^{2}}\) and \((\cdot ,\cdot )\), and E is a Hilbert space with the inner product and the norm given by (2.4) and (2.5), respectively. From (2.3) and (2.5), one has

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}}(|\nabla u|^{2}+V(x)u^{2})\mathrm{{d}}x=\Vert u^{+}\Vert ^{2}-\Vert u^{-}\Vert ^{2}. \end{aligned}$$
(2.7)

If (V) (or (V1)) and (A1) hold, then the weak solutions of problem (1.1), named \((u,\phi _{u})\in E\times {\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3})\) are critical points of the functional given by

$$\begin{aligned} \Upsilon (u,\phi )=\frac{1}{2}\int _{{{\mathbb {R}}}^{3}}[|\nabla u|^{2}+V(x)u^{2}-|\nabla \phi |^{2}-(2\omega +\phi )\phi u^{2}]\mathrm{{d}}x-\int _{{{\mathbb {R}}}^{3}}F(x,u)\mathrm{{d}}x,\ \ u\in E, \end{aligned}$$
(2.8)

where \({\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3}):=\{u\in L^{6}({{\mathbb {R}}}^{3}):|\nabla u|\in L^{2}({{\mathbb {R}}}^{3})\}\). The functional \(\Upsilon \) is strongly indefinite, that is both unbounded from below and from above on infinitely dimensional spaces. To overcome this difficulty, we borrowed the idea from [2, 3] to reduce the study of (2.8) to the study of \(\Upsilon \) with only one variable u, which has been used by most authors. The following technical results obtained in [3,4,5] will be used in our proofs.

Lemma 2.1

[3, 4] For any \(u\in H^{1}({{\mathbb {R}}}^{3})\), there is a unique \(\phi =\phi _{u}\in {\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3})\) which solves equation

$$\begin{aligned} -\Delta \phi +\phi u^{2}=-\omega u^{2}. \end{aligned}$$
(2.9)

Moreover, the map \(J:u\in H^{1}({{\mathbb {R}}}^{3})\mapsto \phi _{u}\in {\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3})\) is continuously differentiable, and \(-\omega \le \phi _{u}\le 0\) on the set \(\{x\in {{\mathbb {R}}}^{3}|u(x)\ne 0\}\).

Lemma 2.2

[5] If \(u_{n}\rightharpoonup u\in H^{1}({{\mathbb {R}}}^{3})\), then up to subsequences, \(\phi _{u_{n}}\rightharpoonup \phi _{u}\) in \({\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3})\). Moreover, \(J'(u_{n})\rightarrow J'(u)\) in the sense of distributions, where J is the same as that in Lemma 2.1.

Multiplying (2.9) by \(\phi _{u}\) and integrating by parts, one has

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}} |\nabla \phi _{u}|^{2}\mathrm{{d}}x=-\int _{{{\mathbb {R}}}^{3}}\omega \phi _{u}u^{2}\mathrm{{d}}x-\int _{{{\mathbb {R}}}^{3}} \phi _{u}^{2}u^{2}\mathrm{{d}}x=-\int _{{{\mathbb {R}}}^{3}}(\omega +\phi _{u})\phi _{u}u^{2}\mathrm{{d}}x. \end{aligned}$$
(2.10)

Using (2.7), (2.8) and (2.10), the functional \(\Phi (u):=\Upsilon (u,\phi )\) reduces to the following form

$$\begin{aligned} \Phi (u)= & {} \frac{1}{2}(\Vert u^{+}\Vert ^{2}-\Vert u^{-}\Vert ^{2}) -\int _{{{\mathbb {R}}}^{3}}\left[ \frac{1}{2}\omega \phi _{u}u^{2}+F(x,u)\right] \mathrm{{dx}}\nonumber \\= & {} \frac{1}{2}\Vert u\Vert ^{2}-\int _{{{\mathbb {R}}}^{3}}\left[ \frac{1}{2}\omega \phi _{u}u^{2}+F(x,u)\right] \mathrm{{d}}x,\ \ \forall \ u=u^{-}+u^{0}+u^{+}\in E. \end{aligned}$$
(2.11)

By Lemmas 2.1 and 2.2, if (V) (or (V1)) and (A1) hold, then one has \(\Phi \in C^{1}(E,{{\mathbb {R}}})\), and

$$\begin{aligned} \langle \Phi '(u),v\rangle = \int _{{{\mathbb {R}}}^{3}}(\nabla u\cdot \nabla v+V(x)uv)\mathrm{{d}}x-\int _{{{\mathbb {R}}}^{3}}[(2\omega +\phi _{u})\phi _{u}u+f(x,u)]v\mathrm{{d}}x,\ \forall \ v\in E, \end{aligned}$$
(2.12)

moreover,

$$\begin{aligned} \langle \Phi '(u),u\rangle = \Vert u^{+}\Vert ^{2}-\Vert u^{-}\Vert ^{2}-\int _{{{\mathbb {R}}}^{3}}[(2\omega +\phi _{u})\phi _{u}u^{2}+f(x,u)u]\mathrm{{d}}x,\ \forall \ u\in E,. \end{aligned}$$
(2.13)

Furthermore, as in [6], the pair \((u,\phi _{u})\in E\times {\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3})\) is a solution of system (1.1) if and only if u is a critical point of \(\Phi \) and \(\phi =\phi _{u}\) which is unique. For simplicity, in the following, we just say that \(u\in E\) is a weak solution of system (1.1) instead of \((u,\phi _{u})\in E\times {\mathfrak {D}}^{1,2}({{\mathbb {R}}}^{3})\). The following two lemmas are very useful in our proofs.

Lemma 2.3

[36, 37] Let \((X,\Vert \cdot \Vert )\) be a real Hilbert space with \(X=X^{-}\oplus X^{+}\) and \(X^{-}\bot X^{+}\), and let \(I\in C^{1}(X,{{\mathbb {R}}})\) of the form

$$\begin{aligned} I(u)=\frac{1}{2}(\Vert u^{+}\Vert ^{2}-\Vert u^{-}\Vert ^{2})-\psi (u),\ \ u=u^{-}+u^{+}\in X^{-}\oplus X^{+}. \end{aligned}$$
(2.14)

Suppose that the following conditions are satisfied:

  1. (S1)

    \(\psi \in C^{1}(X,{{\mathbb {R}}})\) is bounded from below and weakly sequentially lower semi-continuous;

  2. (S2)

    \(\psi '\) is weakly sequentially continuous;

  3. (S3)

    there exists \(r>\rho >0\) and \(e\in X^{+}\) with \(\Vert e\Vert =1\) such that

    $$\begin{aligned} k:=\inf I(S^{+}_{\rho })>\sup I(\partial Q), \end{aligned}$$

    where

    $$\begin{aligned} S^{+}_{\rho }=\{u\in X^{+}:\Vert u\Vert =\rho \},\ \ Q=\{v+se:v\in X^{-}, s\ge 0,\ \Vert v+se\Vert \le r\}. \end{aligned}$$

    Then there exist a constant \(c\in [k,\sup I(Q)]\) and a sequence \(\{u_{n}\}\subset X\) satisfying

    $$\begin{aligned} I(u_{n})\rightarrow c, \ \ \Vert I'(u_{n})\Vert (1+\Vert u_{n}\Vert )\rightarrow 0,\ \ \text{ as }\ n\rightarrow \infty . \end{aligned}$$

As is known, a functional \(I\in C^{1}(X,{{\mathbb {R}}})\) is said to be weakly sequentially lower semi-continuous if \(I(u)\le \liminf _{n\rightarrow \infty }I(u_{n})\) for any \(u_{n}\rightharpoonup u\) in X, and \(I'\) is said to be weakly sequentially continuous if \(\lim _{n\rightarrow \infty }\langle I'(u_{n}),v\rangle =\langle I(u),v\rangle \) for each \(v\in X\).

Lemma 2.4

[38] If assumption (V1) holds, then the embedding from E into \(L^{s}({{\mathbb {R}}}^{3})\) is compact for \(s\in [2, 6)\).

Let

$$\begin{aligned} \Psi (u)=\int _{{{\mathbb {R}}}^{3}}\left[ \frac{1}{2}\omega \phi _{u}u^{2}+F(x,u)\right] \mathrm{{d}}x,\ \ u\in E. \end{aligned}$$
(2.15)

By Lemmas 2.1 and 2.2, we can easily obtain and prove the following lemma by employing a standard argument.

Lemma 2.5

Assume that (V) (or (V1)), (A1), (A2) and (A3) are satisfied. Then \(\Psi \) is bounded from below, weakly sequentially lower semi-continuous, and \(\Psi '\) is weakly sequentially continuous.

3 Proof of Theorem 1.1

The periodic case for KGM system is considered and the proof of Theorem 1.1 is given in this section. To do this, assume that (V) holds and V(x) and f(x, t) are both 1-periodic in each of \(x_{1}\), \(x_{2}\) and \(x_{3}\). Hence, \(E^{0}=\{0\}\), and then \(E=E^{-}\oplus E^{+}\).

Lemma 3.1

Assume that (V), (A1) and (A2) are satisfied. Then there exists \(\rho >0\) such that

$$\begin{aligned} k:=\inf \{\Phi (u):u\in E^{+}, \Vert u\Vert =\rho \}>0. \end{aligned}$$
(3.1)

Proof

From (A1) and (A2), for \(\varepsilon =\frac{1}{3\gamma _{2}^{2}}\), there exists a constant \(C_{1}>0\) such that

$$\begin{aligned} f(x,u)\le \frac{1}{3\gamma _{2}^{2}}|u|+C_{1}|u|^{p-1},\ \ \forall \ (x,u)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}. \end{aligned}$$
(3.2)

From (3.2), we have

$$\begin{aligned} F(x,u)\le \frac{1}{6\gamma _{2}^{2}}|u|^{2}+\frac{C_{1}}{p}|u|^{p},\ \ \forall \ (x,u)\in {{\mathbb {R}}}^{3}\times {{\mathbb {R}}}. \end{aligned}$$
(3.3)

From Lemma 2.1, we know that \(-\omega \le \phi _{u}\le 0\) on the set \(\{x\in {{\mathbb {R}}}^{3}|u(x)\ne 0\}\), then we have \(0\le -\omega \phi _{u}\le \omega ^{2}\) on the set \(\{x\in {{\mathbb {R}}}^{3}|u(x)\ne 0\}\). Hence, for all \(\omega >0\) and \(u\in E\), from (2.6), (2.11) and (3.3), one has

$$\begin{aligned} \Phi (u)= & {} \frac{1}{2}\Vert u\Vert ^{2}-\frac{1}{2}\int _{{{\mathbb {R}}}^{3}}\omega \phi _{u}u^{2}\mathrm{{d}}x-\int _{{{\mathbb {R}}}^{3}}F(x,u)\mathrm{{d}}x\nonumber \\\ge & {} \frac{1}{2}\Vert u\Vert ^{2}-\int _{{{\mathbb {R}}}^{3}}F(x,u)\mathrm{{d}}x\nonumber \\\ge & {} \frac{1}{2}\Vert u\Vert ^{2}-\int _{{{\mathbb {R}}}^{3}}\left( \frac{1}{6\gamma _{2}^{2}}|u|^{2}+\frac{C_{1}}{p}|u|^{p}\right) \mathrm{{d}}x\nonumber \\\ge & {} \Vert u\Vert \left( \frac{1}{3}\Vert u\Vert -\frac{C_{1}}{p}\gamma _{p}^{p}\Vert u\Vert ^{p-1}\right) . \end{aligned}$$
(3.4)

Set

$$\begin{aligned} h(t)=\frac{1}{3}t-\frac{C_{1}}{p}\gamma _{p}^{p}t^{p-1},\ t\ge 0. \end{aligned}$$

Since \(p\in (2,6)\), one can easily obtain that

$$\begin{aligned} \max _{t\ge 0}h(t)=h(\rho )=\frac{p-2}{3(p-1)}\rho >0, \end{aligned}$$

where \(\rho =\left( \frac{p}{3C_{1}(p-1)\gamma _{p}^{p}}\right) ^{1/(p-2)}\). Hence, there exists \(\alpha =\rho h(\rho )>0\) such that

$$\begin{aligned} \Phi |_{\Vert u\Vert =\rho }(u)\ge \alpha >0. \end{aligned}$$

The proof Lemma 3.1 is complete. \(\square \)

From (A3), one can assume that \(A\subset {{\mathbb {R}}}^{3}\) is a bounded domain without loss of generality. Choose \(w\in C_{0}^{\infty }(A,{{\mathbb {R}}}^{+})\cap C_{0}^{\infty }({{\mathbb {R}}}^{3},{{\mathbb {R}}}^{+})\) such that

$$\begin{aligned} \Vert w^{+}\Vert ^{2}-\Vert w^{-}\Vert ^{2}=\int _{{{\mathbb {R}}}^{3}}(|\nabla w|^{2}+V(x)w^{2})\mathrm{{d}}x=\int _{A}(|\nabla w|^{2}+V(x)w^{2})\mathrm{{d}}x\ge 1, \end{aligned}$$

which shows that \(w^{+}\ne 0\).

We need Lemma 3.2 to show the mountain pass geometry of \(\Phi \).

Lemma 3.2

Assume that (V), (A1), (A2) and (A3) hold. Then \(\sup \Phi (E^{-}\oplus {{\mathbb {R}}}^{+}w^{+})<\infty \) and there exists \(R_{w}>0\) such that

$$\begin{aligned} \Phi (u)\le 0,\ \ u\in E^{-}\oplus {{\mathbb {R}}}^{+}w^{+},\ \Vert u\Vert \ge R_{w}. \end{aligned}$$
(3.5)

Proof

Arguing by contradiction, one can assume that there is a sequence \(\{a_{n}+b_{n}w^{+}\}\subset E^{-}\oplus {{\mathbb {R}}}^{+}w^{+}\) with \(\Vert a_{n}+b_{n}w^{+}\Vert \rightarrow +\infty \) as \(n\rightarrow \infty \), \(\Phi (a_{n}+b_{n}w^{+})>0\) for all \(n\in {{\mathbb {N}}}\). Let \(z_{n}=(a_{n}+b_{n}w^{+})/\Vert a_{n}+b_{n}w^{+}\Vert =z_{n}^{-}+t_{n}w^{+}\). Then \(\Vert z_{n}^{-}+t_{n}w^{+}\Vert =1\). Without loss of generality, one may assume that \(t_{n}\rightarrow t\), \(z_{n}^{-}\rightharpoonup z^{-}\), \(z_{n}^{-}\rightarrow z^{-}\) in \(L^{s}(A)\) for \(s\in [2, 6)\) and \(z_{n}^{-}\rightarrow z^{-}\) a.e. on \({{\mathbb {R}}}^{3}\) passing to a subsequence. From (2.11), one gets

$$\begin{aligned} 0< \frac{\Phi (a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}= & {} \frac{t_{n}^{2}}{2}\Vert w^{+}\Vert ^{2}-\frac{1}{2}\Vert z_{n}^{-}\Vert ^{2} -\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\nonumber \\&-\int _{{{\mathbb {R}}}^{3}}\frac{\frac{1}{2}\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x. \end{aligned}$$
(3.6)

From (A2) and (A3), there are constants \(C_{2}\ge \frac{\omega ^{2}}{2}\) and \(C_{3}\) such that

$$\begin{aligned} F(x,u)\ge C_{2}|u|^{2}-C_{3},\ \ \forall \ x\in {{\mathbb {R}}}^{3},\ \forall \ u\in {{\mathbb {R}}}. \end{aligned}$$
(3.7)

If \(t=0\), then from \(w^{+}\ne 0\), (3.6), (3.7) and Lemma 2.1, as \(n\rightarrow \infty \), one gets

$$\begin{aligned} 0\le & {} \frac{1}{2}\Vert z_{n}^{-}\Vert ^{2}+\int _{{{\mathbb {R}}}^{3}}\frac{C_{2}|a_{n}+b_{n}w^{+}|^{2}-C_{3}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x +\int _{{{\mathbb {R}}}^{3}}\frac{-\frac{1}{2}\omega ^{2}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\nonumber \\\le & {} \frac{1}{2}\Vert z_{n}^{-}\Vert ^{2}+\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x +\int _{{{\mathbb {R}}}^{3}}\frac{\frac{1}{2}\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\nonumber \\= & {} \frac{1}{2}\Vert z_{n}^{-}\Vert ^{2}+\int _{{{\mathbb {R}}}^{3}} \left[ \frac{1}{2}\omega \phi _{a_{n}+b_{n}w^{+}}+\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}\right] (z_{n}^{-}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\\le & {} \frac{t_{n}^{2}}{2}\Vert w^{+}\Vert ^{2}\rightarrow 0, \end{aligned}$$
(3.8)

which implies that \(\Vert z_{n}^{-}\Vert \rightarrow 0\) as \(n\rightarrow \infty \), and we have \(1=\Vert z_{n}^{-}+t_{n}w^{+}\Vert \rightarrow 0\) as \(n\rightarrow \infty \), a contradiction. Therefore, \(t\ne 0\).

Now, we prove that

$$\begin{aligned} (z^{-}+tw^{+})|_{A}\ne 0. \end{aligned}$$
(3.9)

The proof of (3.9) is similar to that of [30], for the reader’s convenience, the details are given here. Suppose that (3.9) is not true, then we have

$$\begin{aligned} (z^{-}+tw^{+})|_{A}=0. \end{aligned}$$
(3.10)

It follows from supp \(w\subset A\), (2.7), (2.11), (3.6), (3.7) and (3.10) that

$$\begin{aligned} 0\le & {} 2\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x +\int _{{{\mathbb {R}}}^{3}}\frac{\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\nonumber \\\le & {} t_{n}^{2}\Vert w^{+}\Vert ^{2}-\Vert z_{n}^{-}\Vert ^{2}\nonumber \\= & {} \int _{{{\mathbb {R}}}^{3}}[|\nabla (z_{n}^{-}+t_{n}w^{+})|^{2}+V(x)(z_{n}^{-}+t_{n}w^{+})^{2}]\mathrm{{d}}x\nonumber \\= & {} \int _{A}[|\nabla (z_{n}^{-}+t_{n}w^{+})|^{2}+V(x)(z_{n}^{-}+t_{n}w^{+})^{2}]\mathrm{{d}}x\nonumber \\&+\int _{{{\mathbb {R}}}^{3}\backslash A}[|\nabla (z_{n}^{-}+t_{n}w^{+})|^{2}+V(x)(z_{n}^{-}+t_{n}w^{+})^{2}]\mathrm{{d}}x\nonumber \\= & {} \int _{A}[|\nabla (z_{n}^{-}+t_{n}w^{+})|^{2}+V(x)(z_{n}^{-}+t_{n}w^{+})^{2}]\mathrm{{d}}x\nonumber \\&+\int _{{{\mathbb {R}}}^{3}\backslash A}[|\nabla (z_{n}^{-}-t_{n}w^{-})|^{2}+V(x)(z_{n}^{-}-t_{n}w^{-})^{2}]\mathrm{{d}}x\nonumber \\= & {} \int _{A}[|\nabla z_{n}^{-}|^{2}-|\nabla z^{-}|^{2}]\mathrm{{d}}x+o(1) +\int _{{{\mathbb {R}}}^{3}\backslash A}[|\nabla (z_{n}^{-}-t_{n}w^{-})|^{2}+V(x)(z_{n}^{-}-t_{n}w^{-})^{2}]\mathrm{{d}}x\nonumber \\= & {} \int _{A}[|\nabla z_{n}^{-}|^{2}-|\nabla z^{-}|^{2}]\mathrm{{d}}x+o(1) +\int _{{{\mathbb {R}}}^{3}}[|\nabla (z_{n}^{-}-t_{n}w^{-})|^{2}+V(x)(z_{n}^{-}-t_{n}w^{-})^{2}]\mathrm{{d}}x\nonumber \\&-\int _{A}[|\nabla (z_{n}^{-}-t_{n}w^{-})|^{2}+V(x)(z_{n}^{-}-t_{n}w^{-})^{2}]\mathrm{{d}}x\nonumber \\= & {} -\Vert z_{n}^{-}-t_{n}w^{-}\Vert ^{2}-\int _{A}[|\nabla (z_{n}^{-}-t_{n}w^{-})|^{2}+V(x)(z_{n}^{-}-t_{n}w^{-})^{2}]\mathrm{{d}}x+o(1)\nonumber \\= & {} -\Vert z_{n}^{-}-t_{n}w^{-}\Vert ^{2}-t^{2}\int _{A}[|\nabla w|^{2}+V(x)w^{2}]\mathrm{{d}}x+o(1)\nonumber \\\le & {} -t^{2}+o(1), \end{aligned}$$
(3.11)

which is a contradiction and implies that (3.9) holds. From (2.6), (3.6), (3.9), (A2), (A3), Lemma 2.1 and Fatou’s Lemma, we have

$$\begin{aligned} 0\le & {} \limsup _{n\rightarrow \infty }\left[ \frac{t_{n}^{2}}{2}\Vert w^{+}\Vert ^{2}-\frac{1}{2}\Vert z_{n}^{-}\Vert -\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\right. \nonumber \\&\left. -\,\frac{1}{2}\int _{{{\mathbb {R}}}^{3}}\frac{\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\right] \nonumber \\\le & {} \frac{t^{2}}{2}\Vert w^{+}\Vert ^{2}-\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z_{n}^{-}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\&+\,\frac{\omega ^{2}}{2}\lim _{n\rightarrow \infty }\int _{{{\mathbb {R}}}^{3}}(z_{n}^{-}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\= & {} \frac{t^{2}}{2}\Vert w^{+}\Vert ^{2}+\frac{\omega ^{2}}{2}\lim _{n\rightarrow \infty }\Vert z_{n}^{-}+t_{n}w^{+}\Vert ^{2}_{2}\nonumber \\&-\,\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z_{n}^{-}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\\le & {} \frac{t^{2}}{2}\Vert w^{+}\Vert ^{2}+\frac{\omega ^{2}\gamma _{2}^{2}}{2} -\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z^{-}+tw^{+})^{2}\mathrm{{d}}x\nonumber \\= & {} -\infty , \end{aligned}$$
(3.12)

which is a contradiction. We now complete the proof of Lemma 3.2. \(\square \)

Corollary 3.3

Assume that (V), (A1), (A2) and (A3) hold. Then there is \(r>\rho \) such that \(\sup \Phi (\partial Q)\le 0\), where \(\rho \) is the same as that in Lemma 3.1 and

$$\begin{aligned} Q=\{a+bw^{+}:a\in E^{-}, b\ge 0, \Vert a+bw^{+}\Vert \le r\}. \end{aligned}$$
(3.13)

From Lemmas 2.3, 2.5, 3.1 and Corollary 3.3, one can obtain Lemma 3.4.

Lemma 3.4

Assume that (V), (A1), (A2) and (A3) hold. Then there are a constant \(c>0\) and a sequence \(\{u_{n}\}\subset E\) satisfying

$$\begin{aligned} \Phi (u_{n})\rightarrow c,\ \Vert \Phi '(u_{n})\Vert (1+\Vert u_{n}\Vert )\rightarrow 0\ \ \text{ as }\ n\rightarrow \infty . \end{aligned}$$
(3.14)

Lemma 3.5

Assume that (V), (A1), (A2), (A3) and (A4) hold. Then any sequence \(\{u_{n}\}\subset E\) satisfying

$$\begin{aligned} \Phi (u_{n})\rightarrow c\ge 0,\ \langle \Phi '(u_{n}),u_{n}^{\pm }\rangle \rightarrow 0\ \ \text{ as }\ n\rightarrow \infty \end{aligned}$$
(3.15)

is bounded in E.

Proof

From (2.11), (2.12), (3.15), Lemma 2.1 and (A4), there exists a constant \(C_{4}>0\) such that

$$\begin{aligned} C_{4}\ge & {} \Phi (u_{n})-\frac{1}{2}\langle \Phi '(u_n), u_n\rangle \nonumber \\= & {} \int _{{{\mathbb {R}}}^{3}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x+\int _{{{\mathbb {R}}}^{3}}\frac{1}{2}(\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}^{2}\mathrm{{d}}x\nonumber \\\ge & {} \int _{{{\mathbb {R}}}^{3}}\left( \frac{\omega ^{2}}{8}+c_{1}\right) u_{n}^2\mathrm{{d}}x-\frac{\omega ^{2}}{8}\int _{{{\mathbb {R}}}^{3}}u_{n}^{2}\mathrm{{d}}x\nonumber \\= & {} \int _{{{\mathbb {R}}}^{3}}c_{1}u_{n}^2\mathrm{{d}}x. \end{aligned}$$
(3.16)

It follows from (3.16) that there is a positive constant \(C_{5}\) such that

$$\begin{aligned} \Vert u_{n}\Vert _{2}\le C_{5}. \end{aligned}$$
(3.17)

From (3.17) and Lemma 2.1, we have

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x\le & {} C_{4}-\int _{{{\mathbb {R}}}^{3}}\frac{1}{2}(\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}^{2}\mathrm{{d}}x\nonumber \\\le & {} C_{4}+\frac{\omega ^{2}}{8}\int _{{{\mathbb {R}}}^{3}}u_{n}^{2}\mathrm{{d}}x\nonumber \\\le & {} C'_{5}, \end{aligned}$$
(3.18)

where \(C'_{5}\) is a positive constant. To prove the boundedness of \(\{u_{n}\}\), arguing by contradiction, we assume that \(\Vert u_{n}\Vert \rightarrow \infty \) as \(n\rightarrow \infty \). Let \(z_{n}=u_{n}/\Vert u_{n}\Vert \). Therefore, \(\Vert z_{n}\Vert ^{2}=1\). Set

$$\begin{aligned} \Omega _{n}:=\left\{ x\in {{\mathbb {R}}}^{3}:\frac{f(x,u_{n})}{u_{n}}\le \Theta -\delta _{0}\right\} . \end{aligned}$$
(3.19)

Since \(\Theta \Vert z_{n}^{+}\Vert _{2}^{2}\le \Vert z_{n}^{+}\Vert ^{2}\), from (3.19), we get

$$\begin{aligned} \int _{\Omega _{n}}\frac{f(x,u_{n})}{u_{n}}(z_{n}^{+})^{2}\mathrm{{d}}x\le (\Theta -\delta _{0})\Vert z_{n}^{+}\Vert _{2}^{2}\le 1-\frac{\delta _{0}}{\Theta }. \end{aligned}$$
(3.20)

From (A4), (3.18) and Hölder inequality, one has

$$\begin{aligned}&\int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}\frac{|f(x,u_{n})|}{\Vert u_{n}\Vert }|z_{n}^{+}-z_{n}^{-}|\mathrm{{d}}x\nonumber \\&\quad = \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|z_{n}^{+}-z_{n}^{-}|\mathrm{{d}}x\nonumber \\&\quad \le \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\left[ \int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}\left( \frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}\right) ^{\frac{6}{5-\varrho }}\mathrm{{d}}x\right] ^{\frac{5-\varrho }{6}}\left( \int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}|z_{n}|^{\frac{6\varrho }{1+\varrho }}|z_{n}^{+}-z_{n}^{-}|^{\frac{6}{1+\varrho }}\mathrm{{d}}x\right) ^{\frac{1+\varrho }{6}}\nonumber \\&\quad \le \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\left[ \int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}\left( \frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}\right) ^{\frac{6}{5-\varrho }}\mathrm{{d}}x\right] ^{\frac{5-\varrho }{6}}\Vert z_{n}\Vert ^{\varrho }_{6}\Vert z_{n}^{+}-z_{n}^{-}\Vert _{6}\nonumber \\&\quad \le \frac{C_{6}}{\Vert u_{n}\Vert ^{1-\varrho }}\left( \int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x\right) ^{\frac{5-\varrho }{6}}=o(1). \end{aligned}$$
(3.21)

Let \(m(s)=2\omega s+s^{2}\), \(-\omega \le s\le 0\). By a direct calculation, we see that

$$\begin{aligned} -\omega ^{2}\le m(s)\le 0, \ \ -\omega \le s\le 0. \end{aligned}$$
(3.22)

From (3.17), (3.22), Lemma 2.1 and Hölder inequality, one obtains

$$\begin{aligned}&\frac{1}{\Vert u_{n}\Vert ^{2}}\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}(u_{n}^{+}-u_{n}^{-})|\mathrm{{d}}x\nonumber \\&\quad = \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}||u_{n}||z_{n}^{+}-z_{n}^{-}|\mathrm{{d}}x\nonumber \\&\quad \le \frac{\omega ^{2}}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|u_{n}||z_{n}^{+}-z_{n}^{-}|\mathrm{{d}}x\nonumber \\&\quad \le \frac{\omega ^{2}}{\Vert u_{n}\Vert }\Vert u_{n}\Vert _{2}\Vert z_{n}^{+}-z_{n}^{-}\Vert _{2}\nonumber \\&\quad \le \frac{\omega ^{2}C_{7}}{\Vert u_{n}\Vert }=o(1), \end{aligned}$$
(3.23)

From (A4), we have \(uf(x,u)\ge 0\). Therefore, from (2.11), (3.15), (3.20), (3.21) and (3.23), one obtains

$$\begin{aligned} 1+o(1)= & {} \frac{\Vert u_{n}\Vert ^{2}-\langle \Phi '(u_{n}),u_{n}^{+}-u_{n}^{-}\rangle }{\Vert u_{n}\Vert ^{2}}\nonumber \\= & {} \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}f(x,u_{n})(z_{n}^{+}-z_{n}^{-})\mathrm{{d}}x\nonumber \\&+\, \frac{1}{\Vert u_{n}\Vert ^{2}}\int _{{{\mathbb {R}}}^{3}}(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}(u_{n}^{+}-u_{n}^{-})\mathrm{{d}}x\nonumber \\= & {} \int _{\Omega _{n}}\frac{f(x,u_{n})}{u_{n}}[(z_{n}^{+})^{2}-(z_{n}^{-})^{2}]\mathrm{{d}}x\nonumber \\&+\,\frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}\frac{f(x,u_{n})}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }(z_{n}^{+}-z_{n}^{-})\mathrm{{d}}x\nonumber \\&+\,\frac{1}{\Vert u_{n}\Vert ^{2}}\int _{{{\mathbb {R}}}^{3}}(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}(u_{n}^{+}-u_{n}^{-})\mathrm{{d}}x\nonumber \\\le & {} \int _{\Omega _{n}}\frac{f(x,u_{n})}{u_{n}}(z_{n}^{+})^{2}\mathrm{{d}}x +\frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\int _{{{\mathbb {R}}}^{3}\backslash \Omega _{n}}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|z_{n}^{+}-z_{n}^{-}|\mathrm{{d}}x\nonumber \\&+\,\frac{1}{\Vert u_{n}\Vert ^{2}}\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}(u_{n}^{+}-u_{n}^{-})|\mathrm{{d}}x\nonumber \\\le & {} 1-\Theta +o(1). \end{aligned}$$
(3.24)

This is a contradiction which implies that \(\{u_{n}\}\) is bounded. The proof is complete. \(\square \)

Proof of Theorem 1.1

From Lemmas 3.4 and 3.5, we know that there is a bounded sequence \(\{u_{n}\}\subset E\) satisfying (3.15). Passing to a subsequence if necessary, one has \(u_{n}\rightharpoonup u\) in E, \(u_{n}\rightarrow u\) in \(L^{s}({{\mathbb {R}}}^{3})\) for \(s\in [2, 6)\) and \(u_{n}\rightarrow u\) a.e. on \({{\mathbb {R}}}^{3}\). Together with Lemma 2.2, up to a subsequence, one can show that \(u_{n}\rightarrow u\) by a standard argument (see [26], Lemma 3.1). Jointly with Lemmas 3.1, 3.2 and Corollary 3.3, one can obtain that system (1.1) possesses at least one nontrivial solution. We complete the proof of Theorem 1.1 now. \(\square \)

4 Proof of Theorem 1.2

The non-periodic case for system (1.1) is considered now, that is V(x) is coercive. One supposes that (V1) holds. Hence \(E=E^{-}\oplus E^{0}\oplus {E^{+}}\), and one has that the embedding from E into \(L^{s}({{\mathbb {R}}}^{3})\) is compact for \(s\in [2, 6)\).

Lemma 4.1

Assume that (V1), (A1) and (A2) hold. Then there is \(\rho >0\) such that

$$\begin{aligned} k:=\inf \{\Phi (u):u\in E^{+}, \Vert u\Vert =\rho \}>0. \end{aligned}$$
(4.1)

The proof of the above lemma is very similar to that of Lemma 3.1, so we do not give the detail here. From unique continuation theorem [39], the following lemma holds which comes from [30].

Lemma 4.2

[30] Assume that (V1) hold. If \({\mathcal {A}}z=-\Delta z+V(x)z=0\) and \(z|_{A}=0\), then \(z=0\).

We have \(\dim (E^{-}\oplus E^{0}):=m<\infty \) under (V1). Let \(w_{1}, w_{2}, \cdots , w_{m}\) be an orthogonal basis. We can assume that \(A\subset {{\mathbb {R}}}^{3}\) is a bounded domain without loss of generality. Choose \(w\in C_{0}^{\infty }({{\mathbb {R}}}^{3},{{\mathbb {R}}}^{+})\cap C_{0}^{\infty }(A,{{\mathbb {R}}}^{+})\) such that \(w|_{A}, w_{1}|_{A}, \cdots , w_{m}|_{A}\) are linearly independent and

$$\begin{aligned} \Vert w^{+}\Vert ^{2}-\Vert w^{-}\Vert ^{2}= & {} \int _{{{\mathbb {R}}}^{3}}(|\nabla w|^{2}+V(x)w^{2})\mathrm{{d}}x\\= & {} \int _{A}(|\nabla w|^{2}+V(x)w^{2})\mathrm{{d}}x\ge 1, \end{aligned}$$

which shows that \(w^{+}\ne 0\).

Lemma 4.3

Assume that (V1), (A1), (A2) and (A3) hold. Then \(\sup \Phi (E^{-}\oplus E^{0}\oplus {{\mathbb {R}}}^{+}w^{+})<\infty \) and there is \(R_{w}>0\) such that

$$\begin{aligned} \Phi (u)\le 0,\ u\in E^{-}\oplus E^{0}\oplus {{\mathbb {R}}}^{+}w^{+},\ \Vert u\Vert \ge R_{w}. \end{aligned}$$
(4.2)

Proof

Arguing by contradiction, one can assume that there is a sequence \(\{a_{n}+b_{n}w^{+}\}\subset E^{-}\oplus E^{0}\oplus {{\mathbb {R}}}^{+}w^{+}\) with \(\Vert a_{n}+b_{n}w^{+}\Vert \rightarrow \infty \) as \(n\rightarrow \infty \), \(\Phi (a_{n}+b_{n}w^{+})>0\) for all \(n\in {{\mathbb {N}}}\). Let \(z_{n}=(a_{n}+b_{n}w^{+})/\Vert a_{n}+b_{n}w^{+}\Vert =z_{n}^{-}+z_{n}^{0}+t_{n}w^{+}\), then \(\Vert z_{n}^{-}+z_{n}^{0}+t_{n}w^{+}\Vert =1\). Since the dimension of \(E^{-}\oplus E^{0}\) is finite, passing to a subsequence if neccessary, one may assume that \(z_{n}^{-}\rightarrow z^{-}\), \(z_{n}^{0}\rightarrow z^{0}\) in \(L^{s}({{\mathbb {R}}}^{3})\) for \(2\le s<6\), \(t_{n}\rightarrow t\). Thus, it follows from (2.11) that

$$\begin{aligned} 0<\frac{\Phi (a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}= & {} \frac{t_{n}}{2}\Vert w^{+}\Vert ^{2}-\frac{1}{2}\Vert z_{n}^{-}\Vert ^{2} -\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\nonumber \\&-\int _{{{\mathbb {R}}}^{3}}\frac{\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{2\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x. \end{aligned}$$
(4.3)

We need to consider three cases:

Case 1\(t=0\) and \(z^{0}=0\). From (3.7), (4.3) and Lemma 2.1, as \(n\rightarrow \infty \), we have

$$\begin{aligned} 0\le & {} \frac{1}{2}\Vert z_{n}^{-}\Vert ^{2}+\int _{{{\mathbb {R}}}^{3}}\frac{C_{2}|a_{n}+b_{n}w^{+}|^{2}-C_{3}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x +\int _{{{\mathbb {R}}}^{3}}\frac{-\frac{1}{2}\omega ^{2}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\nonumber \\\le & {} \frac{1}{2}\Vert z_{n}^{-}\Vert ^{2}+\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x -\int _{{{\mathbb {R}}}^{3}}\frac{\omega ^{2}(a_{n}+b_{n}w^{+})^{2}}{2\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\\\le & {} \frac{1}{2}\Vert z_{n}^{-}\Vert ^{2}+\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x +\int _{{{\mathbb {R}}}^{3}}\frac{\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{2\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\\\le & {} \frac{t_{n}}{2}\Vert w^{+}\Vert ^{2}\rightarrow 0\ \ \text{ as }\ n\rightarrow \infty , \end{aligned}$$

which implies that \(\Vert z_{n}^{-}\Vert \rightarrow 0\) as \(n\rightarrow \infty \). Hence, \(1=\Vert z_{n}^{-}+z_{n}^{0}+t_{n}w^{+}\Vert \rightarrow 0\) as \(n\rightarrow \infty \), this is a contradiction.

Case 2\(t=0\) and \(z^{0}\ne 0\). We have \(z^{-}=0\) and \({\mathcal {A}}z^{0}=0\) in this case. It follows from Lemma 4.2 that \(z^{0}|_{A}\ne 0\). From Lemma 2.1, (2.6), (4.3) and (A3), we obtain

$$\begin{aligned} 0\le & {} \limsup _{n\rightarrow \infty }\left[ \frac{t_{n}^{2}}{2}\Vert w^{+}\Vert ^{2}-\frac{1}{2}\Vert z_{n}^{-}\Vert -\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\right. \nonumber \\&\left. -\,\frac{1}{2}\int _{{{\mathbb {R}}}^{3}}\frac{\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\right] \nonumber \\\le & {} -\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z_{n}^{-}+z_{n}^{0}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\&+\,\frac{\omega ^{2}}{2}\lim _{n\rightarrow \infty }\int _{{{\mathbb {R}}}^{3}}(z_{n}^{-}+z_{n}^{0}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\= & {} \frac{\omega ^{2}}{2}\lim _{n\rightarrow \infty }\Vert z_{n}^{-}+z_{n}^{0}+t_{n}w^{+}\Vert ^{2}_{2}\nonumber \\&-\,\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z_{n}^{-}+z_{n}^{0}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\\le & {} \frac{\omega ^{2}\gamma _{2}^{2}}{2} -\int _{A}\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z^{0})^{2}\mathrm{{d}}x=-\infty , \end{aligned}$$
(4.4)

which is a contradiction.

Case 3\(t\ne 0\). We should prove that

$$\begin{aligned} (z^{-}+z^{0}+tw^{+})|_{A}\ne 0. \end{aligned}$$
(4.5)

Arguing by indirection, one may suppose that

$$\begin{aligned} (z^{-}+z^{0}+tw^{+})|_{A}=0. \end{aligned}$$
(4.6)

Since \(z^{-}+z^{0}-t(w^{-}+w^{0})\in E^{-}\oplus E^{0}\), there exist \(y_{1}, y_{2},\ldots , y_{m}\) such that

$$\begin{aligned} z^{-}+z^{0}-t(w^{-}+w^{0})=y_{1}w_{1}+y_{2}w_{2}+\cdots +y_{m}w_{m}, \end{aligned}$$

which together with (4.6) implies that

$$\begin{aligned} 0=(z^{-}+z^{0}+tw^{+})|_{A}=\left( \sum _{i=1}^{m}y_{i}w_{i}+tw\right) \bigg |_{A}=\sum _{i=1}^{m}a_{i}w_{i}\big |_{A}+tw|_{A}. \end{aligned}$$
(4.7)

It follows from (4.7) that \(w|_{A}, w_{1}|_{A},\cdots , w_{m}|_{A}\) are linearly dependent, a contradiction, which shows that (4.5) holds. From (2.6), (4.3), (4.5), (A3), Lemma 2.1 and Fatou’s lemma, one obtains

$$\begin{aligned} 0\le & {} \limsup _{n\rightarrow \infty }\left[ \frac{t_{n}^{2}}{2}\Vert w^{+}\Vert ^{2}-\frac{1}{2}\Vert z_{n}^{-}\Vert -\int _{{{\mathbb {R}}}^{3}}\frac{F(x,a_{n}+b_{n}w^{+})}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\right. \nonumber \\&\left. -\,\frac{1}{2}\int _{{{\mathbb {R}}}^{3}}\frac{\omega \phi _{a_{n}+b_{n}w^{+}}(a_{n}+b_{n}w^{+})^{2}}{\Vert a_{n}+b_{n}w^{+}\Vert ^{2}}\mathrm{{d}}x\right] \nonumber \\\le & {} \frac{t^{2}}{2}\Vert w^{+}\Vert ^{2} -\,\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z_{n}^{-}+z_{n}^{0}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\&+\,\frac{\omega ^{2}}{2}\lim _{n\rightarrow \infty }\int _{{{\mathbb {R}}}^{3}}(z_{n}^{-}+z_{n}^{0}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\= & {} \frac{t^{2}}{2}\Vert w^{+}\Vert ^{2}+\frac{\omega ^{2}}{2}\lim _{n\rightarrow \infty }\Vert z_{n}^{-}+z_{n}^{0}+t_{n}w^{+}\Vert ^{2}_{2}\nonumber \\&-\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z_{n}^{-}+z_{n}^{0}+t_{n}w^{+})^{2}\mathrm{{d}}x\nonumber \\\le & {} \frac{t^{2}}{2}\Vert w^{+}\Vert ^{2}+\frac{\omega ^{2}\gamma _{2}^{2}}{2} -\int _{A}\liminf _{n\rightarrow \infty }\int _{A}\frac{F(x,a_{n}+b_{n}w^{+})}{(a_{n}+b_{n}w^{+})^{2}}(z^{-}+z^{0}+tw^{+})^{2}\mathrm{{d}}x\nonumber \\= & {} -\infty , \end{aligned}$$
(4.8)

a contradiction. We now finish the proof of Lemma 4.3. \(\square \)

Corollary 4.4

Assume that (V1), (A1), (A2) and (A3) hold. Then there is \(r>\rho \) such that \(\sup \Phi (\partial Q)\le 0\), where \(\rho \) is the same as that in Lemma 4.1 and

$$\begin{aligned} Q=\{a+bw^{+}:a\in E^{-}\oplus E^{0}, b\ge 0, \Vert a+bw^{+}\Vert \le r\}. \end{aligned}$$

From Lemmas 2.3, 2.5, 4.1 and Corollary 4.4, we obtain Lemma 4.5.

Lemma 4.5

Assume that (V1), (A1), (A2) and (A3) hold. Then there exists a constant \(c>0\) and a sequence \(\{u_{n}\}\subset E\) satisfying

$$\begin{aligned} \Phi (u_{n})\rightarrow c,\ \Vert \Phi '(u_{n})\Vert (1+\Vert u_{n}\Vert )\rightarrow 0\ \ \text{ as }\ n\rightarrow \infty . \end{aligned}$$
(4.9)

Lemma 4.6

Under assupmtions (V1), (A1)–(A3) and (A5), any sequence \(\{u_{n}\}\subset E\) satisfying (4.9) is bounded in E

Proof

In order to obtain the boundedness of \(\{u_{n}\}\), we argue by indirection, assume that \(\Vert u_{n}\Vert \rightarrow \infty \) as \(n\rightarrow \infty \). Let \(z_{n}=u_{n}/\Vert u_{n}\Vert \). Then \(\Vert z_{n}\Vert =1\). Up to a subsequence, one may assume that \(z_{n}\rightharpoonup z\) in E. Then by Lemma 2.4, \(z_{n}\rightarrow z\) in \(L^{s}({{\mathbb {R}}}^{3})\) for \(s\in [2, 6)\), and \(z_{n}\rightarrow z\) a.e. on \({{\mathbb {R}}}^{3}\) as \(n\rightarrow \infty \). From (4.9), we obtain

$$\begin{aligned} c+o(1)\ge & {} \Phi (u_{n})-\frac{1}{2}\langle \Phi '(u_{n}),u_{n}\rangle \nonumber \\= & {} \int _{{{\mathbb {R}}}^{3}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x+\int _{{{\mathbb {R}}}^{3}}\frac{1}{2}(\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}^{2}\mathrm{{d}}x \ \ \text{ as }\ n\rightarrow \infty . \end{aligned}$$
(4.10)

Similar to (3.17) and (3.18), there exists positive constants \(C_{8}\) and \(C'_{8}\) such that

$$\begin{aligned} \Vert u_{n}\Vert _{2}\le C_{8}, \end{aligned}$$
(4.11)

and

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x\le C'_{8}. \end{aligned}$$
(4.12)

From (2.6), (3.22), (4.11), Lemma 2.1 and Hölder inequality, we get

$$\begin{aligned} \frac{1}{\Vert u_{n}\Vert ^{2}}\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}u_{n}^{+}|\mathrm{{d}}x= & {} \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}||u_{n}||z_{n}^{+}|\mathrm{{d}}x\nonumber \\\le & {} \frac{\omega ^{2}}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|u_{n}||z_{n}^{+}|\mathrm{{d}}x\nonumber \\= & {} \frac{\omega ^{2}}{\Vert u_{n}\Vert }\Vert u_{n}\Vert _{2}\Vert z_{n}^{+}\Vert _{2}\nonumber \\\le & {} \frac{\omega ^{2}C_{9}}{\Vert u_{n}\Vert }=o(1). \end{aligned}$$
(4.13)

In the following, we consider two possible cases.

Case 1\(z=0\). In this case, \(\Vert z_{n}^{-}\Vert +\Vert z_{n}^{0}\Vert \rightarrow 0\), \(z_{n}\rightarrow 0\) in \(L^{s}({{\mathbb {R}}}^{3})\) for \(s\in [2, 6)\), \(z_{n}\rightarrow 0\) a.e on \({{\mathbb {R}}}^{3}\) as \(n\rightarrow \infty \). Hence, from (A1) and (A2), we get

$$\begin{aligned} \int _{0<|u_{n}|<R_{0}}\frac{|f(x,u_{n})|}{|u_{n}|}|z_{n}z_{n}^{+}|\mathrm{{d}}x\le C_{10}|z_{n}|_{2}|z_{n}^{+}|_{2}\rightarrow 0\ \ \text{ as }\ n\rightarrow \infty . \end{aligned}$$
(4.14)

From (2.6), (4.12), (A5) and Hölder inequality, we obtain

$$\begin{aligned}&\frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\int _{|u_{n}|\ge R_{0}}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|z_{n}^{+}|\mathrm{{d}}x\nonumber \\&\quad \le \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\left[ \int _{|u_{n}|\ge R_{0}}\left( \frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}\right) ^{\frac{6}{5-\varrho }}\mathrm{{d}}x\right] ^{\frac{5-\varrho }{6}}\Vert z_{n}\Vert _{6}^{\varrho }\Vert z_{n}^{+}\Vert _{6}\nonumber \\&\quad \le \frac{C_{11}}{\Vert u_{n}\Vert ^{1-\varrho }}\left[ \int _{|u_{n}|\ge R_{0}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x\right] ^{\frac{5-\varrho }{6}}=o(1). \end{aligned}$$
(4.15)

From (2.6), (4.13), (4.14) and (4.15), we get

$$\begin{aligned} 1+o(1)= & {} \frac{\Vert u_{n}\Vert ^{2}-\Vert u_{n}^{-}\Vert ^{2}-\Vert u_{n}^{0}\Vert ^{2}-\langle \Phi '(u_{n}),u_{n}^{+}\rangle }{\Vert u_{n}\Vert ^{2}}\nonumber \\= & {} \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}f(x,u_{n})z_{n}^{+}\mathrm{{d}}x+ \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}z_{n}^{+}\mathrm{{d}}x\nonumber \\\le & {} \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}f(x,u_{n})z_{n}^{+}\mathrm{{d}}x+\frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}z_{n}^{+}|\mathrm{{d}}x\nonumber \\\le & {} \int _{0<|u_{n}|<R_{0}}\frac{|f(x,u_{n})|}{|u_{n}|}|z_{n}z_{n}^{+}|\mathrm{{d}}x +\frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\int _{|u_{n}|\ge R_{0}}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|z_{n}^{+}|\mathrm{{d}}x\nonumber \\&+\frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}z_{n}^{+}|\mathrm{{d}}x\nonumber \\= & {} o(1), \end{aligned}$$
(4.16)

this is a contradiction.

Case 2. \(z\ne 0\). For any \(v\in C_{0}^{\infty }({{\mathbb {R}}}^{3})\), from (2.12) and (4.9), we get

$$\begin{aligned} o(1)= & {} \langle \Phi '(u_{n}),\Vert u_{n}\Vert v\rangle \nonumber \\= & {} \Vert u_{n}\Vert \int _{{{\mathbb {R}}}^{3}}(\nabla u_{n}\cdot \nabla v+V(x)u_{n}v)\mathrm{{d}}x-\Vert u_{n}\Vert \int _{{{\mathbb {R}}}^{3}}[(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}+f(x,u_{n})]v \mathrm{{d}}x\nonumber \\= & {} \Vert u_{n}\Vert ^{2}\int _{{{\mathbb {R}}}^{3}}(\nabla z_{n}\cdot \nabla v+V(x)z_{n}v)\mathrm{{d}}x-\Vert u_{n}\Vert \int _{{{\mathbb {R}}}^{3}}[(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}+f(x,u_{n})]v \mathrm{{d}}x,\nonumber \\ \end{aligned}$$
(4.17)

which implies that

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}}(\nabla z_{n}\cdot \nabla v+V(x)z_{n}v)\mathrm{{d}}x-\frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}[(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}+f(x,u_{n})]v \mathrm{{d}}x=o(1).\nonumber \\ \end{aligned}$$
(4.18)

From Lemma 2.1, (3.22), (4.11) and Hölder inequality, we get

$$\begin{aligned} \frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|(2\omega +\phi _{u_{n}})\phi _{u_{n}}u_{n}v| \mathrm{{d}}x\le & {} \frac{\omega ^{2}}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|u_{n}||v| \mathrm{{d}}x\nonumber \\\le & {} \frac{\omega ^{2}}{\Vert u_{n}\Vert }\left[ \int _{{{\mathbb {R}}}^{3}}|u_{n}|^{2}\mathrm{{d}}x\right] ^{\frac{1}{2}}\left( \int _{{{\mathbb {R}}}^{3}}|v|^{2} \mathrm{{d}}x\right) ^{\frac{1}{2}}\nonumber \\= & {} \frac{\omega ^{2}}{\Vert u_{n}\Vert }\Vert u_{n}\Vert _{2}\Vert v\Vert _{2}\nonumber \\\le & {} \frac{C_{8}\omega ^{2}}{\Vert u_{n}\Vert }\Vert v\Vert _{2}=o(1). \end{aligned}$$
(4.19)

From (A1), (A2), (A5), (2.6), (4.12) and Hölder inequality, one obtains

$$\begin{aligned}&\frac{1}{\Vert u_{n}\Vert }\int _{{{\mathbb {R}}}^{3}}|f(x,u_{n})v|\mathrm{{d}}x\nonumber \\&\quad \le \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\int _{u_{n}\ne 0}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|v|\mathrm{{d}}x\nonumber \\&\quad = \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\left[ \int _{0<|u_{n}|<R_{0}}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|v|\mathrm{{d}}x +\int _{|u_{n}|\ge R_{0}}\frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}|z_{n}|^{\varrho }|v|\mathrm{{d}}x\right] \nonumber \\&\quad \le \frac{1}{\Vert u_{n}\Vert ^{1-\varrho }}\left\{ C_{12}\Vert z_{n}\Vert _{2}^{\varrho }\Vert v\Vert _{2/(2-\varrho )}+\left[ \int _{|u_{n}|\ge R_{0}}\left( \frac{|f(x,u_{n})|}{|u_{n}|^{\varrho }}\right) ^{\frac{6}{5-\varrho }}\mathrm{{d}}x\right] ^{\frac{5-\varrho }{6}}\Vert z_{n}\Vert _{6}^{\varrho }\Vert v\Vert _{6}\right\} \nonumber \\&\quad \le \frac{C_{13}}{\Vert u_{n}\Vert ^{1-\varrho }}\left\{ \Vert v\Vert _{2/(2-\varrho )}+\Vert v\Vert _{6}\left[ \int _{|u_{n}|\ge R_{0}}{\mathscr {F}}(x,u_{n})\mathrm{{d}}x\right] ^{\frac{5-\varrho }{6}}\right\} \nonumber \\&\quad \le \frac{C_{14}}{\Vert u_{n}\Vert ^{1-\varrho }}(\Vert v\Vert _{2/(2-\varrho )}+\Vert v\Vert _{6})=o(1). \end{aligned}$$
(4.20)

From (4.18), (4.19) and (4.20), we have

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}}(\nabla z_{n}\cdot \nabla v+V(x)z_{n}v)\mathrm{{d}}x=o(1), \ \ v\in C_{0}^{\infty }({{\mathbb {R}}}^{3}). \end{aligned}$$
(4.21)

Since \(z_{n}\rightharpoonup z\) as \(n\rightarrow \infty \), from (4.21), one obtains

$$\begin{aligned} \int _{{{\mathbb {R}}}^{3}}(\nabla z\cdot \nabla v+V(x)zv)\mathrm{{d}}x=0, \ \ v\in C_{0}^{\infty }({{\mathbb {R}}}^{3}), \end{aligned}$$
(4.22)

which shows that \({\mathcal {A}}z=-\triangle z+V(x)z=0\). It follows from Lemma 4.2 that \(z|_{A}\ne 0\). From (2.6), (2.11), (A3), Lemma 2.1 and Fatou’s Lemma, one has

$$\begin{aligned} 0= & {} \lim _{n\rightarrow \infty }\frac{c+o(1)}{\Vert u_{n}\Vert ^{2}}=\lim _{n\rightarrow \infty }\frac{\Phi (u_{n})}{\Vert u_{n}\Vert ^{2}}\nonumber \\\le & {} \lim _{n\rightarrow \infty }\left[ \frac{1}{2}-\int _{{{\mathbb {R}}}^{3}}\frac{F(x,u_{n})}{u_{n}^{2}}z_{n}^{2}\mathrm{{d}}x-\frac{1}{2}\int _{{{\mathbb {R}}}^{3}}\omega \phi _{u_{n}}z_{n}^{2}\mathrm{{d}}x\right] \nonumber \\\le & {} \lim _{n\rightarrow \infty }\left[ \frac{1}{2}-\int _{A}\frac{F(x,u_{n})}{u_{n}^{2}}z_{n}^{2}\mathrm{{d}}x+\frac{\omega ^{2}}{2}\Vert z_{n}\Vert ^{2}_{2}\right] \nonumber \\\le & {} \lim _{n\rightarrow \infty }\left[ \frac{1}{2}+\frac{\omega ^{2}\gamma _{2}^{2}}{2}-\int _{A}\frac{F(x,u_{n})}{u_{n}^{2}}z^{2}\mathrm{{d}}x\right] \nonumber \\= & {} \frac{1}{2}+\frac{\omega ^{2}\gamma _{2}^{2}}{2}-\lim _{n\rightarrow \infty }\int _{A}\frac{F(x,u_{n})}{u_{n}^{2}}z^{2}\mathrm{{d}}x\nonumber \\= & {} -\infty ,\ \ \text{ as }\ n\rightarrow \infty , \end{aligned}$$
(4.23)

this is a contradiction. Hence \(\{u_{n}\}\) is bounded in E. We complete the proof Lemma 4.6 now. \(\square \)

Proof of Theorem 1.2

From Lemmas 4.5 and 4.6, there is a bounded sequence \(\{u_{n}\}\subset E\) satisfying (4.9). Passing to a subsequence if necessary, one has \(u_{n}\rightharpoonup u\) in E, \(u_{n}\rightarrow u\) in \(L^{s}({{\mathbb {R}}}^{3})\) for \(s\in [2, 6)\) and \(u_{n}\rightarrow u\) a.e. on \({{\mathbb {R}}}^{3}\). Together with Lemma 2.2, up to a subsequence, we can show that \(u_{n}\rightarrow u\) in E by a standard argument (see [26], Lemma 3.1). Thus, jointly with Lemmas 4.1, 4.3 and Corollary 4.4, we can obtain that system (1.1) possesses at least one nontrivial solution. The proof of Theorem 1.2 is complete.