Multiplicity and concentration behavior of solutions for magnetic Choquard equation with critical growth

Abstract

In this paper, we consider the following nonlinear Choquard equation with magnetic field

$$\begin{aligned} \begin{aligned} \left\{ \begin{array}{l} \displaystyle \bigg (\frac{\varepsilon }{i}\nabla -A(x)\bigg )^{2}u+V(x)u=\varepsilon ^{\mu -N}\left( \,\,\int \limits _{{\mathbb {R}}^{N}}\frac{|u(y)|^{2_{\mu }^{*}}+F(|u(y)|^{2})}{|x-y|^{\mu }}\text {d}y\right) \left( |u|^{2_{\mu }^{*}-2}u+\frac{1}{2_{\mu }^{*}}f(|u|^{2})u\right) \hspace{1.14mm}\text{ in }\hspace{1mm} {\mathbb {R}}^{N},\\ \displaystyle u\in H^{1}({\mathbb {R}}^{N},{\mathbb {C}})\\ \end{array} \right. \end{aligned} \end{aligned}$$

where $\varepsilon >0$ is a small parameter, $N\ge 3$, $0<\mu <N$, $2_{\mu }^{*}=\frac{2N-\mu }{N-2}$, $V(x):{\mathbb {R}}^{N}\rightarrow {\mathbb {R}}^{N}$ and $A(x):{\mathbb {R}}^{N}\rightarrow {\mathbb {R}}^{N}$ is a continuous potential, f is a continuous subcritical term, and F is the primitive function of f. Under a local assumption on the potential V, by the variational methods, the penalization techniques and the Ljusternik–Schnirelmann theory, we prove the multiplicity and concentration properties of nontrivial solutions of the above problem for $\varepsilon >0$ small enough.

1 Introduction and main results

In this paper, we study the following nonlinear Choquard equation with magnetic fields

$$\begin{aligned} \begin{aligned} \left( \frac{\varepsilon }{i}\nabla -A(x)\right) ^{2}u+V(x)u =\varepsilon ^{\mu -N}\left( \,\,\int \limits _{{\mathbb {R}}^{N}}\frac{|u(y)|^{2_{\mu }^{*}}+F(|u(y)|^{2})}{|x-y|^{\mu }}\text {d}y\right) \left( |u|^{2_{\mu }^{*}-2}u+\frac{1}{2_{\mu }^{*}}f(|u|^{2})u\right) , \end{aligned} \end{aligned}$$

(1.1)

where $x\in {\mathbb {R}}^{N}$, $\varepsilon >0$ is a parameter, $N\ge 3$, $0<\mu <N$, $2_{\mu }^{*}=\frac{2N-\mu }{N-2}$, $V(x)\in C({\mathbb {R}}^{N},{\mathbb {R}})$ is an electric potential, and $A(x)\in C({\mathbb {R}}^{N},{\mathbb {R}}^{N})$ is a magnetic potential. The operator $\left( \frac{1}{i}\nabla -A(x)\right) ^{2}$, called magnetic Laplacian, is defined by

$$\begin{aligned} -\Delta _{A} u:=\left( \frac{1}{i}\nabla -A(x)\right) ^{2}u =-\Delta u-\frac{2}{i}A(x)\cdot \nabla u+|A(x)|^{2}u -\frac{1}{i}u{\text {div}}(A(x)). \end{aligned}$$

For problem (1.1), there are vast literature concerning the existence and multiplicity of nonlinear Choquard equation without magnetic field, namely $A \equiv 0$. Then, (1.1) turns to the Choquard equation

$$\begin{aligned} -\varepsilon ^{2}\Delta u+V(x) u=\varepsilon ^{\mu -N}\left( \,\,\int \limits _{{\mathbb {R}}^{N}} \frac{G(u(y))}{|x-y|^{\mu }}\text {d}y\right) g(u) \text{ in } {\mathbb {R}}^{N}. \end{aligned}$$

(1.2)

When $\varepsilon =1$, Eq. (1.2) reduces to the generalized Choquard equation:

$$\begin{aligned} -\Delta u +V(x)u =\Big (\frac{1}{|x|^{\mu }}*|u|^{p}\Big )|u|^{p-2}u \quad \text{ in } \quad {\mathbb {R}}^3. \end{aligned}$$

(1.3)

The case of (1.3) $N=3, p=2$, $V(x)=1$ and $\mu =1$ describes the quantum mechanics of a polaron at rest by Pekar [36]. In 1976, Choquard [26] described an approximation to Hartree–Fock theory of a one component plasma by (1.3). Thus, Eq. (1.3) is also called the nonlinear Schrödinger–Newton equation. By using critical point theory, Lions [28] obtained the existence of infinitely many radially symmetric solutions in $H^{1}\left( {\mathbb {R}}^{N}\right)$ and Ackermann [1] proved the existence of infinitely many geometrically distinct weak solutions for a general case.

Moroz and Van Schaftingen [31, 33] eliminated this restriction and showed the regularity, positivity and radial symmetry of the ground states for the optimal range of parameters, and also derived that these solutions decay asymptotically at infinity.

On the other hand, the semiclassical limit problem attracts people’s attention, namely $\varepsilon \rightarrow 0$ in (1.2). The question of the existence of semiclassical solutions for the nonlocal problem (1.2) has been posed in [6]. Moroz and Van Schaftingen [32] develop a novel nonlocal penalization by variational methods to show that equation (1.2) with $G(u)=|u|^{q}$ has a family of solutions concentrated at the local minimum of V, with V satisfying some additional assumptions at infinity. Other results see [5, 12, 25, 30, 43] and references therein.

The magnetic nonlinear Schrödinger equation (1.1) has been extensively investigated by many authors by applying suitable variational and topological methods (see [3, 9, 13, 14, 16, 17, 20, 23, 40, 41] and references therein). To the best of our knowledge, the first result involving the magnetic field was obtained by Esteban and Lions [20]. They used the concentration–compactness principle and minimization arguments to obtain solutions for $\varepsilon >0$ fixed and $N=2,3$. For the nonlinear magnetic Schrödinger equation

$$\begin{aligned} \bigg (\frac{\varepsilon }{i}\nabla -A(x)\bigg )^{2}u+V(x)u =f(|u|^{2})u,\quad x\in {\mathbb {R}}^{N}, \end{aligned}$$

(1.4)

Alves [3] used the method of the Nehari manifold, the penalization method and the Ljusternik–Schnirelmann category theory to relate the number of solutions with the topology of the set for a subcritical nonlinearity $f \in C^{1}$. Ji and R$\breve{a}$dulescu [23] showed that the arguments developed in [3] fail if f is only continuous, and they improved the methods to study the multiplicity and concentration results for magnetic Schrödinger equation in which the subcritical nonlinearity f is only continuous. Later, Ji and R$\breve{a}$dulescu [24] used the same methods to study the multiplicity and concentration results for magnetic Schrödinger equation with critical growth.

About the Choquard equations with magnetic potential like

$$\begin{aligned} \bigg (\frac{\varepsilon }{i}\nabla -A(x)\bigg )^{2}u+V(x)u =\varepsilon ^{\mu -N}\left( \,\,\int \limits _{{\mathbb {R}}^{N}} \frac{F(|u(y)|^{2})}{|x-y|^{\mu }}\text {d}y\right) f(|u|^{2})u. \end{aligned}$$

(1.5)

For $\varepsilon =1$, $\mu \in (0,N)$, $\frac{2N-\mu }{N}<p<\frac{2N-\mu }{N-2}$, $f(u)=|u|^{p}$, Cingolani et al. [14] established the existence of multiple complex-valued solutions. Alves et al. [4] proved the existence and multiplicity of solutions by using variational methods, penalization techniques and Ljusternik–Schnirelmann theory under the suitable conditions of V and f.

It is quite natural to consider the multiplicity and concentration phenomena of nontrivial solutions for problem (1.1) with critical growth. Inspired by [4, 24], the main purpose of this paper is to investigate multiplicity and concentration of nontrivial solutions for problem (1.1) by proposing a local assumption on V(x) and adapting the penalization technique and Ljusternik–Schnirelmann category theory.

To go on studying the problem (1.1), we recall an important inequality, namely:

Lemma 1.1

(Hardy–Littlewood–Sobolev inequality) [27] Let $t,r>1$ and $0<\mu <N$ with $1/t+\mu /N+1/r=2$, $f\in L^{t}({\mathbb {R}}^N)$ and $h\in L^{r}({\mathbb {R}}^N)$. There exists a sharp constant $C(t,N,\mu ,r)$, independent of f, h, such that

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{f(x)h(y)}{|x-y|^{\mu }}\mathrm{{d}}x\mathrm{{d}}y\le C(t,N,\mu ,r) |f|_{t}|h|_{r}. \end{aligned}$$

(1.6)

If $t=r=2N/(2N-\mu )$, then there is equality in (1.6) if and only if $f\equiv Ch$ and

$$\begin{aligned} h(x)=A(\gamma ^{2}+|x-a|^{2})^{-(2N-\mu )/2} \end{aligned}$$

for some $A\in {\mathbb {C}}$, $0\ne \gamma \in {\mathbb {R}}$ and $a\in {\mathbb {R}}^{N}$.

Notice that, by the Hardy–Littlewood–Sobolev inequality, the integral

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{|u(x)|^{q}|u(y)|^{q}}{|x-y|^{\mu }}\mathrm{{d}}x\mathrm{{d}}y \end{aligned}$$

is well defined if $|u|^{q}\in L^{t}({\mathbb {R}}^N)$ for some $t>1$ satisfying

$$\begin{aligned} \frac{2}{t}+\frac{\mu }{N}=2. \end{aligned}$$

Thus, for $u\in H^{1}({\mathbb {R}}^N)$, by Sobolev embedding Theorems, we know

$$\begin{aligned} 2\le tq\le \frac{2N}{N-2}, \end{aligned}$$

that is

$$\begin{aligned} \frac{2N-\mu }{N}\le q\le \frac{2N-\mu }{N-2}. \end{aligned}$$

Thus, $\frac{2N-\mu }{N}$ is called the lower critical exponent and $2_{\mu }^{*}=\frac{2N-\mu }{N-2}$ is the upper critical exponent in the sense of the Hardy–Littlewood–Sobolev inequality and the Laplace operator.

Before giving our main result, we present the following hypothesis on the potential $V(x)\in C({\mathbb {R}}^{N},{\mathbb {R}})$:

$(V_{1})$ there exists $V_{1}>0$ such that $V_{1}=\inf _{x\in {\mathbb {R}}^{N}}V(x)$;

$(V_{2})$ there exists a bounded open set $\Lambda \subset {\mathbb {R}}^{N}$such that

$$\begin{aligned} 0<V_{0}=\inf _{x\in \Lambda }V(x)<\min _{x\in \partial \Lambda }V(x). \end{aligned}$$

Observe that

$$\begin{aligned} M:=\left\{ x \in \Lambda : V(x)=V_{0}\right\} \ne \emptyset . \end{aligned}$$

Moreover, the nonlinearity $f \in C({\mathbb {R}},{\mathbb {R}})$ be a function satisfying:

$(f_{1})$:

$f(t)=0$ if $t \le 0$;

$(f_{2})$:

there exists $p,q \in \left( \frac{2N-\mu }{N},2_{\mu }^{*}\right)$ and $\lambda >0$ such that

$$\begin{aligned} f(t)\ge \lambda t^{(p-2)/2} \quad \forall t>0, \quad \lim _{t\rightarrow +\infty } \frac{f(t^{2}) t}{t^{q-1}}=0; \end{aligned}$$

$(f_{3})$:

f(t) is strictly increasing in $(0,\infty )$.

The main result of this paper is the following:

Theorem 5.5

Suppose that condictions $(V_{1})$–$(V_{2})$ and $(f_{1})$–$(f_{3})$ are satisfied. Then, for any $\delta >0$ such that

$$\begin{aligned} M_{\delta }:=\{x\in {\mathbb {R}}^{N}:{\text {dist}}(x, M)<\delta \} \subset \Lambda , \end{aligned}$$

there exists $\varepsilon _{\delta }>0$ such that, for any $0<\varepsilon <\varepsilon _{\delta }$, problem (1.1) has at least ${\text {cat}}_{M_{\delta }}(M)$ nontrivial solutions. Moreover, for every sequence $\{\varepsilon _{n}\}$ such that $\varepsilon _{n}\rightarrow 0^{+}$ as $n \rightarrow \infty$, if we denote by $u_{\varepsilon _{n}}$ one of these solutions of (1.1) for $\varepsilon =\varepsilon _{n}$ and $\eta _{\varepsilon _{n}}\in {\mathbb {R}}^{N}$ the global maximum point of $|u_{\varepsilon _{n}}|$, then

$$\begin{aligned} \lim _{n} V\left( \eta _{\varepsilon _{n}}\right) =V_{0}. \end{aligned}$$

We shall use the variational methods, penalization techniques and the Ljusternik–Schnirelmann theory to prove Theorem 1.2. There are three main difficulties to overcome. The first difficulty is the presence of the magnetic field A(x), problem (1.1) became a complex-valued problem. We use the diamagnetic inequality to transform complex values into real values, while corresponding embedding theorem also holds. The second one is that the nonlocal terms appearing in the equation, the commonly used methods and techniques may be ineffective. We can use the Hardy–Littlewood–Sobolev inequality to overcome this difficulty. The last one is that critical exponent occurs the right of problem (1.1), we need more accurate estimate to overcome the lack of compactness.

The paper is organized as follows. In Sect. 2, we give some preliminaries. In Sect. 3, we introduce a modified problem. In Sect. 4, we study the associated autonomous problem. In Sect. 5, we obtain the multiplicity of solutions for problem (1.1). In Sect. 6, we complete the proof Theorem 1.2.

2 Preliminaries

In this section, we collect some notations and some useful preliminary lemmas. Recall that the Sobolev space with magnetic potential $H_{A}^{1}({\mathbb {R}}^{N},{\mathbb {C}})$ is defined by

$$\begin{aligned} H_{A}^{1}({\mathbb {R}}^{N},{\mathbb {C}})=\{u\in L^{2}({\mathbb {R}}^{N},{\mathbb {C}}): |\nabla _{A}u|\in L^{2}({\mathbb {R}}^{N},{\mathbb {R}})\}, \end{aligned}$$

where $\nabla _{A}u=(i^{-1}\nabla -A)u$. The space $H_{A}^{1}({\mathbb {R}}^{N},{\mathbb {C}})$ is an Hilbert space endowed with the scalar product

$$\begin{aligned} \langle u,v\rangle _{\varepsilon }=\Re \left( \,\, \int \limits _{{\mathbb {R}}^{N}}\nabla _{A}u\overline{\nabla _{A}v}+u{\overline{v}}\text {d}x\right) , \end{aligned}$$

where $\Re$ and the bar denote the real part of a complex number and the complex conjugation, respectively.

Using the change of variable $x\rightarrow \varepsilon x$, problem (1.1) is equivalent to the following one

$$\begin{aligned} \left\{ \begin{array}{l} \displaystyle \bigg (\frac{1}{i}\nabla -A_{\varepsilon }(x)\bigg )^{2}u+V_{\varepsilon }(x)u=\left( \,\,\int \limits _{{\mathbb {R}}^{N}}\frac{|u(y)|^{2_{\mu }^{*}}+F(|u(y)|^{2})}{|x-y|^{\mu }}\text {d}y\right) \left( |u|^{2_{\mu }^{*}-2}u+\frac{1}{2_{\mu }^{*}}f(|u|^{2})u\right) \hspace{1.14mm}\text{ in }\hspace{0.5mm} {\mathbb {R}}^{N},\\ \displaystyle u\in H^{1}({\mathbb {R}}^{N},{\mathbb {C}})\\ \end{array} \right. \end{aligned}$$

(2.1)

where $A_{\varepsilon }(x)=A(\varepsilon x)$ and $V_{\varepsilon }(x)=V(\varepsilon x)$.

Owing to the presence of potential $V_{\varepsilon }(x)$, we introduce the subspace

$$\begin{aligned} H_{\varepsilon }=\{u\in H_{A}^{1}({\mathbb {R}}^{N},{\mathbb {C}}): \int \limits _{{\mathbb {R}}^{N}}V_{\varepsilon }(x)|u|^{2}\text {d}x<\infty \}. \end{aligned}$$

The space $H_{\varepsilon }$ is an Hilbert space endowed with the scalar product

$$\begin{aligned} (u,v)_{\varepsilon }=\Re \left( \,\, \int \limits _{{\mathbb {R}}^{N}}\nabla _{\varepsilon }u\overline{\nabla _{\varepsilon }v}+V(\varepsilon x)u{\overline{v}}\text {d}x\right) , \end{aligned}$$

where $\nabla _{\varepsilon }u=(D_{\varepsilon }^{1}u,\ldots ,D_{\varepsilon }^{N}u)$ and $D_{\varepsilon }^{j}u=i^{-1}\partial _{j}u-A_{j}(\varepsilon x)u$ for $j=1,\ldots ,N$. The norm induced by this inner product is given by

$$\begin{aligned} \Vert u\Vert _{\varepsilon }=\left( \,\,\int \limits _{{\mathbb {R}}^{N}}|\nabla _{\varepsilon }u|^{2}+V(\varepsilon x)|u|^{2}\text {d}x\right) ^{\frac{1}{2}}. \end{aligned}$$

For readers’ convenience, we recall some useful lemmas.

Lemma 2.1

(Diamagnetic Inequality) [20] For any $u\in H_{\varepsilon }$, we get $|u|\in H^{1}({\mathbb {R}}^{N},{\mathbb {R}})$ and it holds

$$\begin{aligned} |\nabla |u||\le |\nabla _{\varepsilon }u|. \end{aligned}$$

(2.2)

Lemma 2.2

[20] The space $H_{\varepsilon }$ is continuously embedded into $L^{r}\left( {\mathbb {R}}^{N},{\mathbb {C}}\right)$ for any $r \in \left[ 2,2^{*}\right]$ and compactly embedded into $L^{r}_{loc}\left( {\mathbb {R}}^{N}, {\mathbb {C}}\right)$ for any $r \in \left[ 1,2^{*}\right)$.

From Gao and Yang [21], we denote by

$$\begin{aligned} \begin{aligned} S_{H,L}:&=\inf _{u\in D^{1,2}({\mathbb {R}}^{N})\backslash \{0\}} \frac{\int \limits _{{\mathbb {R}}^{N}}|\nabla u|^{2}\text {d}x}{\left( \,\,\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{|u(x)|^{2_{\mu }^{*}}|u(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y\right) ^{\frac{N-2}{2N-\mu }}}\\&=\inf _{u\in D_{A}^{1,2}({\mathbb {R}}^{N})\backslash \{0\}} \frac{\int \limits _{{\mathbb {R}}^{N}}|\nabla _{A} u|^{2} \text {d}x}{\left( \,\,\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{|u(x)|^{2_{\mu }^{*}}|u(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y\right) ^{\frac{N-2}{2N-\mu }}}:=S_{A}, \end{aligned} \end{aligned}$$

(2.3)

where $D_{A}^{1,2}({\mathbb {R}}^{N})=\{u \in L^{2^{*}} ({\mathbb {R}}^{N},{\mathbb {C}}):\nabla _{A}u \in L^{2}({\mathbb {R}}^{N},{\mathbb {C}})\}$. The equality between $S_{H,L}$ and $S_{A}$ was proved in Mukherjee and Sreenadh [34]. We remark that $S_{A}$ is attained if and only if ${\text {rot}} A=0$, see [34, Theorem 4.1].

Lemma 2.3

[21] The constant $S_{H,L}$ defined in (2.3) is achieved if and only if

$$\begin{aligned} u=C\left( \frac{b}{b^{2}+|x-a|^{2}}\right) ^{\frac{N-2}{2}}, \end{aligned}$$

where $C>0$ is a fixed constant, $a\in {\mathbb {R}}^{N}$ and $b\in (0,\infty )$ are parameters. Furthermore,

$$\begin{aligned} S_{H,L}=\frac{S}{C(N,\mu )^{\frac{N-2}{2N-\mu }}}, \end{aligned}$$

where S is the best Sobolev constant of the immersion $D^{1,2}({\mathbb {R}}^{N}) \hookrightarrow L^{2^{*}}({\mathbb {R}}^{N})$ and $C(N, \mu )$ depends on N and $\mu$.

If we consider the minimizer for S given by $U(x):=\frac{[N(N-2)]^{\frac{N-2}{4}}}{\left( |1+|x|^{2} \right) ^{\frac{N-2}{2}}}$ (see [42, Theorem 1.42]), then

$$\begin{aligned} {\bar{U}}(x)=S^{\frac{(N-\mu )(2-\mu )}{4(N+2-\mu )}} C(N, \mu )^{\frac{2-N}{2(N+2-\mu )}} \frac{[N(N-2)]^{\frac{N-2}{4}}}{\left( |1+|x|^{2} \right) ^{\frac{N-2}{2}}} \end{aligned}$$

is the unique minimizer for $S_{H,L}$ that satisfies

$$\begin{aligned} -\Delta u=\left( \,\,\int \limits _{{\mathbb {R}}^{N}} \frac{|u|^{2_{\mu }^{*}}}{|x-y|^{\mu }} \text {d}y\right) |u|^{2_{\mu }^{*}-2} u \quad \text{ in } {\mathbb {R}}^{N}, \end{aligned}$$

with

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}|\nabla {\bar{U}}|^{2}\text {d}x =\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{|{\bar{U}}(x)|^{2_{\mu }^{*}}|{\bar{U}}(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y= S_{H,L}^{\frac{2N-\mu }{N+2-\mu }}. \end{aligned}$$

Lemma 2.4

[29] Let $N\ge 3$ and $r\in [2,2^{*})$. If $\left\{ u_{n}\right\} _{n\in {\mathbb {N}}}$ is a bounded sequence in $H^{1}({\mathbb {R}}^{N},{\mathbb {R}})$ and if

$$\begin{aligned} \lim _{n\rightarrow \infty }\sup _{y\in {\mathbb {R}}^{N}} \int \limits _{B_{R}(y)}\left| u_{n}\right| ^{r}\mathrm{{d}}x=0, \end{aligned}$$

where $R>0$, then $u_{n}\rightarrow 0$ in $L^{t}({\mathbb {R}}^{N},{\mathbb {R}})$ for all $t\in (2,2^{*})$.

3 The modified problem

We will adapt an argument explored by the penalization method, which introduced by del Pino and Felmer [19] to overcome the lack of compactness. More precisely, let $K>0$ determined later, there exists a unique number $a>0$ satisfying

$$\begin{aligned} f(a)+a^{(2_{\mu }^{*}-2)/2}=V_{0}/K \end{aligned}$$

by $(f_{3})$, where $V_{0}$ is given in $(V_{2})$. Suppose that

$$\begin{aligned} {\widetilde{f}}(t):=\left\{ \begin{array}{l} \displaystyle f(t)+(t^{+})^{(2_{\mu }^{*}-2)/2}, \hspace{4.14mm}t\le a,\\ \displaystyle V_{0}/K, \hspace{26mm}t>a, \end{array} \right. \end{aligned}$$

and introduce the penalized nonlinearity

$$\begin{aligned} g(x,t):=\chi _{\Lambda }(x)(f(t)+(t^{+})^{2_{\mu }^{*}})+(1-\chi _{\Lambda }(x)){\widetilde{f}}(t), \end{aligned}$$

where $\chi _{\Lambda }$ is the characteristic function on $\Lambda$ and $G(x,t):=\int \limits _{0}^{t}f(s)\text {d}s$.

From assumptions $(f_{1})$–$(f_{3})$, it follows that g satisfies the following properties:

$(g_{1})$:

$g(x,t)=0$ for each $t\le 0$ and $\lim \limits _{t\rightarrow 0^{+}}g(x,t)=0$ uniformly in $x\in {\mathbb {R}}^{N}$;

$(g_{2})$:

$g(x,t)\le f(t)+(t^{+})^{2_{\mu }^{*}}$ for all $t\ge 0$ uniformly in $x\in {\mathbb {R}}^{N}$;

$(g_{3})$:

$0<G(t)\le g(x,t)t$, for each $x\in \Lambda , t>0$;

$(g_{4})$:

$0<G(t)\le g(x,t)t\le V_{0}t/K$, for each $x\in \Lambda ^{c}, t>0$;

$(g_{5})$:

$t\mapsto g(x,t)$ and $t\mapsto \frac{G(x,t)}{t}$ are increasing for all $x\in {\mathbb {R}}^{N}$ and $t\in (0,\infty )$. Then, we consider the following modified problem

$$\begin{aligned} \bigg (\frac{1}{i}\nabla -A_{\varepsilon }(x)\bigg )^{2}u+V_{\varepsilon }(x)u =\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|u|^{2})\right) g(\varepsilon x,|u|^{2})u, \hspace{2mm}x\in {\mathbb {R}}^{N}. \end{aligned}$$

(3.1)

Observe that if u is a nontrivial solution of problem (3.1) with

$$\begin{aligned} |u(x)|^{2}\le a,~~~~~\forall x\in \Lambda ^{c}_{\varepsilon }, \end{aligned}$$

where $\Lambda _{\varepsilon }=\{x\in {\mathbb {R}}^{N}:\varepsilon x\in \Lambda \}$, then u is a nontrivial solution of problem (2.1).

The energy function associated to problem (3.1) is

$$\begin{aligned} J_{\varepsilon }(u)=\frac{1}{2}\int \limits _{{\mathbb {R}}^{N}}|\nabla _{\varepsilon }u|^{2}+V_{\varepsilon }(x)|u|^{2}\text {d}x -\frac{1}{2\cdot 2_{\mu }^{*}} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|u|^{2})\right) G(\varepsilon x,|u|^{2})\text {d}x. \end{aligned}$$

In the sequel, set $p_{0}=\frac{2N}{2N-\mu }$, then for any $u\in H_{\varepsilon }$, we have

$$\begin{aligned} |F(u)|_{p_{0}}\le C\bigg (|u|^{2}_{2p_{0}}+|u|^{q}_{qp_{0}}\bigg ) \end{aligned}$$

(3.2)

and

$$\begin{aligned} |G(u)|_{p_{0}}\le C\bigg (|u|^{2}_{2p_{0}}+|u|^{q}_{qp_{0}}+|u|_{2^{*}}^{2\cdot 2^{*}_{\mu }}\bigg ). \end{aligned}$$

(3.3)

Therefore, the Hardy–Littlewood–Sobolev inequality implies that

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}} \left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|u|^{2})\right) G(\varepsilon x,|u|^{2})\text {d}x \le C|G(u)|_{p_{0}}^{2} \le C\bigg (|u|^{4}_{2p_{0}}+|u|^{2q}_{qp_{0}} +|u|_{2^{*}}^{2\cdot 2^{*}_{\mu }}\bigg ) \end{aligned}$$

(3.4)

and

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|u|^{2})\right) g(\varepsilon x,|u|^{2})|u|^{2}\text {d}x \le C\bigg (|u|^{4}_{2p_{0}}+|u|^{2q}_{qp_{0}} +|u|_{2^{*}}^{2\cdot 2^{*}_{\mu }}\bigg ). \end{aligned}$$

(3.5)

As the argument in the proof of [5, Theorem 1.3], we have the following lemma.

Lemma 3.1

Suppose that $\left( V_1\right)$ and $\left( f_1\right) -\left( f_3\right)$ are satisfied, then $J_{\varepsilon }^{\prime }$ is weakly sequentially continuous. Namely, if $u_n \rightarrow u$ in $H_{\varepsilon }$, then $J_{\varepsilon }^{\prime }\left( u_n\right) \rightarrow J_{\varepsilon }^{\prime }(u)$ in $\left( H_{\varepsilon }\right) ^*$.

Proof

Observe that

$$\begin{aligned} \left\langle J_{\varepsilon }^{\prime }(u), v\right\rangle =\langle u, v\rangle _{\varepsilon }-\frac{1}{2_\mu ^*} \int \limits _{{\mathbb {R}}^N}\left( \frac{1}{|x|^\mu } * G\left( \varepsilon x,|u|^2\right) \right) g\left( \varepsilon x,|u|^2\right) u {\bar{v}} \text {d} x. \end{aligned}$$

Since f(t) has subcritical growth in the sense of the Hardy–Littlewood–Sobolev inequality and the definition of g(x, t), to prove that $J_{\varepsilon }^{\prime }$ is weakly sequentially continuous, we only need to check that if $u_n \rightarrow u$ in $H_{\varepsilon }$, then

$$\begin{aligned} \iint _{{\mathbb {R}}^{2 N}} \frac{\left| u_n(y)\right| ^{2_\mu ^*}\left| u_n(x)\right| ^{2_\mu ^*-2} u_n(x) \overline{v(x)}}{|x-y|^\mu } \text {d} x \text {d} y \longrightarrow \iint _{{\mathbb {R}}^{2 N}} \frac{|u(y)|^{2_\mu ^*}|u(x)|^{2_\mu ^*-2} u(x) \overline{v(x)}}{|x-y|^\mu } \text {d} x \text {d} y \end{aligned}$$

(3.6)

for any $v \in H_{\varepsilon }$ as $n \rightarrow \infty$. Indeed, by the Hardy–Littlewood–Sobolev inequality, the Riesz potential defines a linear continuous map from $L^{\frac{2 N}{2 N-\mu }}\left( {\mathbb {R}}^N\right)$ to $L^{\frac{2 N}{\mu }}\left( {\mathbb {R}}^N\right)$, so we know

$$\begin{aligned} \int \limits _{{\mathbb {R}}^N} \frac{\left| u_n(y)\right| ^{2_\mu ^*}}{|x-y|^\mu } \text {d} y \rightarrow \int \limits _{{\mathbb {R}}^N} \frac{|u(y)|^{2_\mu ^*}}{|x-y|^\mu } \text {d} y \quad \text{ in } \quad L^{\frac{2 N}{\mu }}\left( {\mathbb {R}}^N\right) . \end{aligned}$$

Since $\left| u_n\right| ^{2_\mu ^*-2} u_n \rightarrow |u|^{2_\mu ^*-2} u$ in $L^{\frac{2 N}{N-\mu +2}}\left( {\mathbb {R}}^N\right)$ as $n \rightarrow \infty$, we have

$$\begin{aligned} \left( \,\,\int \limits _{{\mathbb {R}}^N} \frac{\left| u_n(y)\right| ^{2_\mu ^*}}{|x-y|^\mu } \text {d} y\right) \left| u_n(x)\right| ^{2_\mu ^*-2} u_n(x) \rightarrow \left( \,\,\int \limits _{{\mathbb {R}}^N} \frac{|u(y)|^{2_\mu ^*}}{|x-y|^\mu } \text {d} y\right) |u(x)|^{2_\mu ^*-2} u(x) \quad \text{ in } \quad L^{\frac{2N}{N+2}}\left( {\mathbb {R}}^N\right) \end{aligned}$$

as $n \rightarrow \infty$. Hence (3.6) holds. $\square$

By a similar argument to Proposition 3.2 in [33], we have the following results.

Lemma 3.2

Let $N\ge 3$ and $\mu \in (0,N)$. If $H,K\in L^{\frac{2 N}{N-\mu +2}}({\mathbb {R}}^{N})+L^{\frac{2N}{N-\mu }}({\mathbb {R}}^{N})$, and $u\in H_{\varepsilon }$ satisfies

$$\begin{aligned} \left( \frac{1}{i}\nabla -A_{\varepsilon }(x)\right) ^{2}u+u =\left( |x|^{-\mu }*(Hu)\right) K. \end{aligned}$$

Then, $u \in L^{r}({\mathbb {R}}^{N})$ for some $r \in \left[ 2,\frac{N}{N-\mu }\frac{2N}{N-2}\right)$. Moreover, there exists a positive constant C(r) independent of u such that

$$\begin{aligned} \Vert u\Vert _{r} \le C(r)\Vert u\Vert _{2}. \end{aligned}$$

Now, let us define

$$\begin{aligned} {\mathcal {B}}=\left\{ u \in H_{\varepsilon }\left( {\mathbb {R}}^{N}\right) :\Vert u\Vert _{\varepsilon }\le d\right\} \end{aligned}$$

where $d>0$ is a constant, and we set

$$\begin{aligned} K_{\varepsilon }(u)(x)=\frac{1}{|x|^{\mu }}*G\left( \varepsilon x,|u|^{2}\right) . \end{aligned}$$

The next lemma implies that we can treat the convolution term as a bounded term.

Lemma 3.3

Suppose that $(f_{1})-(f_{3})$ hold. Then, there exists $C_{0}>0$ such that

$$\begin{aligned} \sup _{u \in {\mathcal {B}}}\left| K_{\varepsilon }(u)(x)\right| _{L^{\infty }({\mathbb {R}}^{N})}<C_{0}. \end{aligned}$$

(3.7)

Proof

Observe that

$$\begin{aligned} |G(x,|s|^{2})|\le |F(|s|^{2})|+|s|^{2_{\mu }^{*}}, \quad \forall s \in {\mathbb {R}}. \end{aligned}$$

Thereby,

$$\begin{aligned} \begin{aligned} |K_{\varepsilon }(u)(x)|&\le \left| \,\,\,\int \limits _{{\mathbb {R}}^{N}}\frac{F(|u|^{2})}{|x-y|^{\mu }}\text {d}y\right| +\left| \,\,\,\int \limits _{{\mathbb {R}}^{N}}\frac{|u|^{\frac{2N-\mu }{N-2}}}{|x-y|^{\mu }}\text {d}y\right| \\&\le \left| \,\,\,\int \limits _{{\mathbb {R}}^{N}}\frac{F(|u|^{2})}{|x-y|^{\mu }}\text {d}y\right| +\left| \,\,\,\int \limits _{|x-y|\le 1} \frac{|u|^{\frac{2N-\mu }{N-2}}}{|x-y|^{\mu }}\text {d}y\right| +\left| \,\,\,\int \limits _{|x-y|>1} \frac{|u|^{\frac{2N-\mu }{N-2}}}{|x-y|^{\mu }}\text {d}y\right| \\&:=K_{1}+K_{2}+K_{3}. \end{aligned} \end{aligned}$$

From [4, Lemma 2.5], we know there exist a constant C independent of x and $\varepsilon$ such that

$$\begin{aligned} |K_{1}|\le C. \end{aligned}$$

(3.8)

Let $H(u)=K(u)=|u|^{\frac{N-\mu +2}{N-2}}$, then it is easy to verify that

$$\begin{aligned} H(u), K(u)\in L^{\frac{2N}{N-\mu +2}}\left( {\mathbb {R}}^{N}\right) +L^{\frac{2 N}{N-\mu }}({\mathbb {R}}^{N}). \end{aligned}$$

Using Lemma 3.2, we obtain $u\in L^{r}({\mathbb {R}}^{N})$ for $r \in \left[ 2,\frac{N}{N-\mu }\frac{2N}{N-2}\right)$. Moreover,

$$\begin{aligned} |u|_{r}\le C(r)|u|_{2}\le C \end{aligned}$$

for some constant $C>0$ independent of $\varepsilon$.

It follows from $\mu <N$ that

$$\begin{aligned} \frac{2N^{2}}{3N\mu -\mu ^{2}}<\frac{N}{\mu }. \end{aligned}$$

Choosing $t>\frac{N}{\mu }$ and $\frac{2N^{2}}{3N\mu -\mu ^{2}}<s<\frac{N}{\mu }$, then

$$\begin{aligned} \frac{t}{t-1} \frac{2N-\mu }{N-2}<\frac{N}{N-\mu } \frac{2 N}{N-2}, \quad \frac{s}{s-1} \frac{2N-\mu }{N-2}<\frac{N}{N-\mu }\frac{2N}{N-2}. \end{aligned}$$

By the Hölder inequality, we have

$$\begin{aligned} K_{2}\le \left( \,\,\int \limits _{|x-y|\le 1} \frac{1}{|x-y|^{\mu s}}\text {d} y\right) ^{\frac{1}{s}} \left( \,\,\int \limits _{|x-y|\le 1}\left| u\right| ^{\frac{s}{s-1}\frac{2N-\mu }{N-2}}\text {d} y\right) ^{\frac{s-1}{s}} \le C, \end{aligned}$$

and

$$\begin{aligned} K_{3}\le \left( \,\,\int \limits _{|x-y|\ge 1}\frac{1}{|x-y|^{\mu t}} \text {d}y\right) ^{\frac{1}{t}} \left( \,\,\int \limits _{|x-y|\ge 1}|u|^{\frac{t}{t-1}\frac{2N-\mu }{N-2}}\text {d} y\right) ^{\frac{t-1}{t}}\le C, \end{aligned}$$

where the constant C is independent of x and $\varepsilon$.

The above equalities with (3.8) imply (3.7). Then, we can find $C_{0}>0$ such that

$$\begin{aligned} \sup _{u \in {\mathcal {B}}}\left| K_{\varepsilon }(u)(x)\right| _{L^{\infty }({\mathbb {R}}^{N})}<C_{0}. \end{aligned}$$

$\square$

From Lemma 3.3, we know there exists $k_{0}$ such that

$$\begin{aligned} \frac{\sup _{u \in {\mathcal {B}}}|K_{\varepsilon }(u)(x)|_{L^{\infty }({\mathbb {R}}^{N})}}{k_{0}}<\frac{1}{2}, \end{aligned}$$

hence we can choose $K\ge k_{0}$.

We denote by ${\mathcal {N}}_{\varepsilon }$ the Nehari manifold of $J_{\varepsilon }$, that is,

$$\begin{aligned} {\mathcal {N}}_{\varepsilon }=\{u\in H_{\varepsilon }\setminus \{0\}:\langle J'_{\varepsilon }(u),u\rangle =0\}, \end{aligned}$$

and define the number $c_{\varepsilon }$ by

$$\begin{aligned} c_{\varepsilon }=\inf _{u\in {\mathcal {N}}_{\varepsilon }} J_{\varepsilon }(u). \end{aligned}$$

Let $H_{\varepsilon }^{+}$ be the open subset of $H_{\varepsilon }$ given by

$$\begin{aligned} H_{\varepsilon }^{+}=\{u\in H_{\varepsilon }:|{\text {supp}}(u)\cap \Lambda _{\varepsilon }|>0\}, \end{aligned}$$

and $S_{\varepsilon }^{+}=S_{\varepsilon }\cap H_{\varepsilon }^{+}$, where $S_{\varepsilon }$ is the unit sphere of $H_{\varepsilon }$. Observe that $S_{\varepsilon }^{+}$ is a non-complete $C^{1,1}$-manifold of codimension 1, modeled on $H_{\varepsilon }$ and contained in $H_{\varepsilon }^{+}$. Hence, $H_{\varepsilon }=T_{u}S_{\varepsilon }^{+}\oplus {\mathbb {R}}u$ for each $u\in S_{\varepsilon }^{+}$, where $T_{u}S_{\varepsilon }^{+}=\{v\in H_{\varepsilon }:\langle u,v\rangle _{\varepsilon }=0\}$.

The functional $J_{\varepsilon }$ satisfies the mountain pass geometry [42].

Lemma 3.4

For any fixed $\varepsilon >0$, the functional $J_{\varepsilon }$ satisfies the following properties:

(a) there exist $\alpha ,\rho >0$ such that $J_{\varepsilon }(u)\ge \alpha$ with $\Vert u\Vert _{\varepsilon }=\rho$;

(b) there exists $e\in H_{\varepsilon }$ such that $\Vert e\Vert _{\varepsilon }\ge \rho$ and $J_{\varepsilon }(e)<0.$

Proof

• (a) For any $u\in H_{\varepsilon }\setminus \{0\}$, it follows from (3.4) and Lemma 2.2 that

  $$\begin{aligned} \begin{aligned} J_{\varepsilon }(u)&=\frac{1}{2}\Vert u\Vert _{\varepsilon }^{2}-\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1 }{|x|^{\mu }} *G(\varepsilon x, |u|^{2})\right) G(\varepsilon x, |u|^{2})\text {d}x\\&\ge \frac{1}{2}\Vert u\Vert _{\varepsilon }^{2} -C\left( \Vert u\Vert _{\varepsilon }^{4} -\Vert u\Vert _{\varepsilon }^{2q} -\Vert u\Vert _{\varepsilon }^{2\cdot 2_{\mu }^{*}}\right) . \end{aligned} \end{aligned}$$

  Hence, we can find $\alpha ,\rho >0$ such that $J_{\varepsilon }\ge \alpha$ with $\Vert u\Vert _{\varepsilon }=\rho$.

• (b) Fix a positive function $u_{0} \in H_{\varepsilon }^{+}$ with ${\text {supp}}\left( u_{0}\right) \subset \Lambda _{\varepsilon }$, and observe

  $$\begin{aligned} G(\varepsilon x,|u_{0}|^{2})=F(|u_{0}|^{2}). \end{aligned}$$

Set

$$\begin{aligned} \alpha (t):=P\left( \frac{tu_{0}}{\Vert u_{0}\Vert _{\varepsilon }}\right)>0\hspace{2mm}\text {for}\hspace{2mm}t>0, \end{aligned}$$

where

$$\begin{aligned} P(u)=\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1 }{|x|^{\mu }} *G(\varepsilon x, |u|^{2})\right) G(\varepsilon x, |u|^{2})\text {d}x. \end{aligned}$$

By assumption $(f_{3})$, we have

$$\begin{aligned} F(t)\le f(t)t,\hspace{2mm}\text { for } t>0. \end{aligned}$$

Therefore, we deduce that

$$\begin{aligned} \alpha ^{\prime }(t)\ge \frac{2\cdot 2_{\mu }^{*}}{t}\alpha (t). \end{aligned}$$

(3.9)

Integrating (3.9) on $\left[ 1, t\left\| u_{0}\right\| _{\varepsilon }\right]$ with $t>\frac{1}{\left\| u_{0}\right\| _{\varepsilon }}$, we have

$$\begin{aligned} P\left( t u_{0}\right) \ge P\left( \frac{u_{0}}{\left\| u_{0}\right\| _{\varepsilon }}\right) \left\| u_{0}\right\| _{\varepsilon }^{2\cdot 2_{\mu }^{*}} t^{2\cdot 2_{\mu }^{*}}. \end{aligned}$$

Hence, we have

$$\begin{aligned} J_{\varepsilon }\left( t u_{0}\right) =\frac{t^{2}}{2}\Vert u_{0}\Vert _{\varepsilon }^{2} -P\left( t u_{0}\right) \le C_{1} t^{2}-C_{2} t^{2\cdot 2_{\mu }^{*}} \hspace{2mm}\text { for } t>\frac{1}{\left\| u_{0}\right\| _{\varepsilon }}. \end{aligned}$$

Taking $e=t u_{0}$ with t sufficiently large, we can see (b) holds. $\square$

Since f is only continuous, we can apply the next two conclusions to overcome the non-differentiability of ${\mathcal {N}}_{\varepsilon }$ and the incompleteness of $S_{\varepsilon }^{+}$.

Lemma 3.5

Suppose that the function V satisfies $(V_{1})$–$(V_{2})$ and f satisfies $(f_{1})$–$(f_{3})$, then the following properties hold:

$(A_{1})$:

For each $u\in H_{\varepsilon }^{+}$, let $\varphi _{u}:{\mathbb {R}}^{+}\rightarrow {\mathbb {R}}$ be given by $\varphi _{u}(t)=J_{\varepsilon }(tu)$. Then, there exists a unique $t_{u}>0$ such that $\varphi '_{u}(t)>0$ in $(0, t_{u})$ and $\varphi '_{u}(t)<0$ in $(t_{u},\infty )$.

$(A_{2})$:

There is a $\sigma >0$ independent on u such that $t_{u}>\sigma$ for all $u\in S_{\varepsilon }^{+}$. Furthermore, for each compact set ${\mathcal {W}}\subset S_{\varepsilon }^{+}$, there is $C_{{\mathcal {W}}}>0$ such that $t_{u}\le C_{{\mathcal {W}}}$ for all $u\in {\mathcal {W}}$.

$(A_{3})$:

The map ${\widehat{m}}_{\varepsilon }:H_{\varepsilon }^{+}\rightarrow {\mathcal {N}}_{\varepsilon }$ given by ${\widehat{m}}_{\varepsilon }(u)=t_{u}u$ is continuous and $m_{\varepsilon }={\widehat{m}}_{\varepsilon }|_{S_{\varepsilon }^{+}}$ is a homeomorphism between $S_{\varepsilon }^{+}$ and ${\mathcal {N}}_{\varepsilon }$. Moreover, $m_{\varepsilon }^{-1}=\frac{u}{\Vert u\Vert _{\varepsilon }}$.

$(A_{4})$:

If there is a sequence $\{u_{n}\}\subset S_{\varepsilon }^{+}$ such that ${\text {dist}}(u_{n}, \partial S_{\varepsilon }^{+})\rightarrow 0$, then $\Vert m_{\varepsilon }(u_{n})\Vert _{\varepsilon }\rightarrow \infty$ and $J_{\varepsilon }(m_{\varepsilon }(u_{n}))\rightarrow \infty$.

Proof

$(A_{1})$:

We notice from the proof of Lemma 3.4 that $\varphi _{u}(0)=0$, $\varphi _{u}(t)>0$ for $t>0$ small and $\varphi _{u}(t)<0$ for $t>0$ large. Therefore, $\max _{t\ge 0}\varphi _{u}(tu)$ is achieved at a global maximum point $t=t_{u}$ verifying $\varphi '_{u}(t_{u})=0$ and $t_{u}u\in {\mathcal {N}}_{\varepsilon }$. From $(f_{3})$, the definition of $\varphi$ and $|{\text {supp}}(u)\cap \Lambda _{\varepsilon }|>0$, we may obtain the uniqueness of $t_{u}$. Therefore, $\max _{t\ge 0}\varphi _{u}(tu)$ is achieved at a unique $t=t_{u}$ such that $\varphi '_{u}(t_{u})=0$ and $t_{u}u\in {\mathcal {N}}_{\varepsilon }$.

$(A_{2})$:

Suppose $u\in S_{\varepsilon }^{+}$, then from (3.5) we have

$$\begin{aligned} \begin{aligned} t_{u}^{2}=\frac{1}{2^{*}_{\mu }}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon x, |u|^{2})\right) g(\varepsilon x, |t_{u}u|^{2})|t_{u}u|^{2}\text {d}x&\le C\left( t_{u}^{4}+t_{u}^{2q} +t_{u}^{2\cdot 2^{*}_{\mu }}\right) . \end{aligned} \end{aligned}$$

which implies that $t_{u}>\sigma$ for some $\sigma >0$.

If ${\mathcal {W}}\subset S_{\varepsilon }^{+}$ is compact, and suppose by contradiction that there is $\{u_{n}\}\subset {\mathcal {W}}$ such that $t_{n}:=t_{u_{n}}\rightarrow \infty$. Since ${\mathcal {W}}$ is compact, there is $u\in {\mathcal {W}}$ such that $u_{n}\rightarrow u$ in $H_{\varepsilon }$. Then, $u\in {\mathcal {W}}\subset S_{\varepsilon }^{+}$. Moreover, using the proof of Lemma 3.4 (b), we have $J_{\varepsilon }(t_{n}u_{n})\rightarrow -\infty$.

On the other hand, using the proof of lemma 3.4 (a), there exists $\rho >0$ such that $\inf _{\Vert u\Vert _{\varepsilon }=\rho }J_{\varepsilon }(u)>0$. Then combining this with $(A_{1})$, we have $c_{\varepsilon }=\inf _{{\mathcal {N}}_{\varepsilon }}J_{\varepsilon }(u)\ge \inf _{\Vert u\Vert _{\varepsilon }=\rho }J_{\varepsilon }(u)>0$, which yields a contradiction. Hence $(A_{2})$ is true.

$(A_{3})$:

Firstly, we observe that $m_{\varepsilon }$, ${\widehat{m}}_{\varepsilon }$ and $m_{\varepsilon }^{-1}$ are well defined. In fact, by $(A_{1})$, for each $u\in H_{\varepsilon }^{+}$, there exists a unique $t_{u}>0$ such that $t_{u}u\in {\mathcal {N}}_{\varepsilon }$, hence there is a unique ${\widehat{m}}_{\varepsilon }(u)=t_{u}u\in {\mathcal {N}}_{\varepsilon }$. On the other hand, if $u\in {\mathcal {N}}_{\varepsilon }$, then $u\in H_{\varepsilon }^{+}$. Therefore, $m_{\varepsilon }^{-1}=\frac{u}{\Vert u\Vert _{\varepsilon }}\in S_{\varepsilon }^{+}$, is well defined and it is a continuous function. Since

$$\begin{aligned} m_{\varepsilon }^{-1}(m_{\varepsilon }(u)) =m_{\varepsilon }^{-1}(t_{u}u) =\frac{t_{u}u}{\Vert t_{u}u\Vert _{\varepsilon }}=u, \hspace{2mm}\forall u\in S_{\varepsilon }^{+}, \end{aligned}$$

we conclude that $m_{\varepsilon }$ is a bijection. To prove that ${\widehat{m}}_{\varepsilon }:H_{\varepsilon }^{+}\rightarrow {\mathcal {N}}_{\varepsilon }$ is continuous, let $\{u_{n}\}\subset H_{\varepsilon }^{+}$ and $u\in H_{\varepsilon }^{+}$ satisfy $u_{n}\rightarrow u$ in $H_{\varepsilon }^{+}$. By $(A_{2})$, there is a $t_{0}>0$ such that $t_{n}:=t_{u_{n}}\rightarrow t_{0}$. From $t_{n}u_{n}\in {\mathcal {N}}_{\varepsilon }$, we obtain

$$\begin{aligned} \begin{aligned} t_{n}^{2}\Vert u_{n}\Vert _{\varepsilon }^{2}=\frac{1}{2^{*}_{\mu }}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon x, |t_{n}u_{n}|^{2})\right) g(\varepsilon x, |t_{n}u_{n}|^{2})|t_{u}u_{n}|^{2}\text {d}x. \end{aligned} \end{aligned}$$

By Lemma 3.1 and passing to the limit as $n\rightarrow \infty$, it follows that

$$\begin{aligned} \begin{aligned} t_{0}^{2}\Vert u\Vert _{\varepsilon }^{2}=\frac{1}{2^{*}_{\mu }} \int \limits _{{\mathbb {R}}^{N}} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon x, |t_{0}u|^{2})\right) g(\varepsilon x, |t_{0}u|^{2})|t_{0}u|^{2}\text {d}x\\ \end{aligned} \end{aligned}$$

which means that $t_{0}u\in {\mathcal {N}}_{\varepsilon }$ and $t_{u}=t_{0}$. This proves ${\widehat{m}}_{\varepsilon }(u_{n})\rightarrow {\widehat{m}}_{\varepsilon }(u)$ in $H_{\varepsilon }^{+}$. So, ${\widehat{m}}_{\varepsilon }$, $m_{\varepsilon }$ are continuous functions and $(A_{3})$ is proved.

$(A_{4})$:

Let $\{u_{n}\}\subset S_{\varepsilon }^{+}$ be a subsequence such that ${\text {dist}}(u_{n}, \partial S_{\varepsilon }^{+}) \rightarrow 0$, then for each $v \in \partial S_{\varepsilon }^{+}$and $n\in {\mathbb {N}}$, we have $\left| u_{n}\right| =\left| u_{n}-v\right|$ a.e. in $\Lambda _{\varepsilon }$. Therefore, by $(V_{1}),(V_{2})$ and the Sobolev embedding, for any $t \in \left[ 2,2^{*}\right]$, there exists a constant $C_{t}>0$ such that

$$\begin{aligned} \begin{aligned} \Vert u_{n}\Vert _{L^{t}(\Lambda _{\varepsilon })}&\le \inf _{v\in \partial S_{\varepsilon }^{+}} \Vert u_{n}-v\Vert _{L^{t}(\Lambda _{\varepsilon })} \\&\le C_{t}\left( \inf _{v\in \partial S_{\varepsilon }^{+}} \int \limits _{\Lambda _{\varepsilon }}\left( |\nabla _{\varepsilon } (u_{n}-v)|^{2}+V_{\varepsilon }(x)\left| u_{n}-v\right| ^{2}\right) \text {d} x\right) ^{\frac{1}{2}} \\&\le C_{t} {\text {dist}}\left( u_{n}, \partial S_{\varepsilon }^{+}\right) \end{aligned} \end{aligned}$$

for all $n\in {\mathbb {N}}$. By $(g_{2})$ and $(g_{4})$, for each $t>0$, we have

$$\begin{aligned} \begin{aligned} \int \limits _{{\mathbb {R}}^{N}}&\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|tu_{n}|^{2})\right) G(\varepsilon x,|tu_{n}|^{2})\text {d}x\\&=\int \limits _{\Lambda _{\varepsilon }}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|tu_{n}|^{2})\right) G(\varepsilon x,|tu_{n}|^{2})\text {d}x +\int \limits _{\Lambda _{\varepsilon }^{c}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|tu_{n}|^{2})\right) G(\varepsilon x,|tu_{n}|^{2})\text {d}x\\&\le C\left( t^{4}|u_{n}|_{L^{2p_{0}}(\Lambda _{\varepsilon })}^{4} +t^{2q}|u_{n}|_{L^{qp_{0}}(\Lambda _{\varepsilon })}^{2q} +t^{2\cdot 2^{*}_{\mu }}|u_{n}|_{L^{2^{*}}(\Lambda _{\varepsilon })}^{2\cdot 2^{*}_{\mu }}\right) +\frac{k_{0}t^{2}}{2K} \int \limits _{\Lambda _{\varepsilon }^{c}} V(\varepsilon x)\left| u_{n}\right| ^{2} \text {d} x \\&\le C\left( t^{4}|u_{n}|_{L^{2p_{0}}(\Lambda _{\varepsilon })}^{4} +t^{qp_{0}}|u_{n}|_{L^{2q}(\Lambda _{\varepsilon })}^{2q} +t^{2\cdot 2^{*}_{\mu }}|u_{n}|_{L^{2^{*}}(\Lambda _{\varepsilon })}^{2\cdot 2^{*}_{\mu }}\right) +\frac{k_{0}t^{2}}{2K}\Vert u_{n}\Vert _{\varepsilon }^{2} \\&\le C\left( t^{4}{\text {dist}}(u_{n}, \partial S_{\varepsilon }^{+})^{4}+t^{2q}{\text {dist}}(u_{n},\partial S_{\varepsilon }^{+})^{2q}+t^{2\cdot 2^{*}_{\mu }} {\text {dist}}(u_{n},\partial S_{\varepsilon }^{+})^{2\cdot 2^{*}_{\mu }}\right) +\frac{k_{0}t^{2}}{2K}. \end{aligned} \end{aligned}$$

Therefore,

$$\begin{aligned} \limsup _{n} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|tu_{n}|^{2})\right) G(\varepsilon x,|tu_{n}|^{2})\text {d}x \le \frac{k_{0}t^{2}}{2K}, \quad \forall t>0. \end{aligned}$$

On the other hand, from the definition of $m_{\varepsilon }$ and the last inequality, for all $t>0$, we obtain

$$\begin{aligned} \begin{aligned} \liminf _{n}J_{\varepsilon }(m_{\varepsilon }\left( u_{n}\right) )&\ge \liminf _{n}J_{\varepsilon }(t u_{n}) \\&\ge \liminf _{n} \frac{t^{2}}{2}\left\| u_{n}\right\| _{\varepsilon }^{2} -\frac{k_{0}t^{2}}{2K}\\&=\frac{K-k_{0}}{2K}t^{2}. \end{aligned} \end{aligned}$$

This implies that

$$\begin{aligned} \liminf _{n} \frac{1}{2}\Vert m_{\varepsilon }(u_{n})\Vert _{\varepsilon }^{2} \ge \liminf _{n}J_{\varepsilon }\left( m_{\varepsilon }(u_{n})\right) \ge \frac{K-k_{0}}{2 K} t^{2}, \quad \forall t>0. \end{aligned}$$

From the arbitrariness of $t>0$, we see that

$$\begin{aligned} \Vert m_{\varepsilon }(u_{n})\Vert _{\varepsilon }\rightarrow \infty \quad \text{ and } \quad J_{\varepsilon }\left( m_{\varepsilon }(u_{n})\right) \rightarrow \infty \quad \text{ as } n\rightarrow \infty . \end{aligned}$$

This completes the proof of Lemma 3.5.

$\square$

Now we define the function

$$\begin{aligned} {\widehat{\Psi }}_{\varepsilon }:H_{\varepsilon }^{+}\rightarrow {\mathbb {R}}\end{aligned}$$

by ${\widehat{\Psi }}_{\varepsilon }(u)=J_{\varepsilon }({\widehat{m}}_{\varepsilon }(u))$ and denote by $\Psi _{\varepsilon }(u)=({\widehat{\Psi }}_{\varepsilon }(u))|_{S_{\varepsilon }^{+}}$.

From Lemma 3.2, by a similar argument to that of [39, Corollary 10], we have the following results.

Lemma 3.6

Assume that $(V_{1})$–$(V_{2})$ and $(f_{1})$–$(f_{3})$ are satisfied, then

$(B_{1})$:

${\widehat{\Psi }}_{\varepsilon }\in C^{1}(H_{\varepsilon }^{+},{\mathbb {R}})$ and

$$\begin{aligned} \langle {\widehat{\Psi }}'_{\varepsilon }(u),v\rangle =\frac{\Vert {\widehat{m}}_{\varepsilon }(u)\Vert _{\varepsilon }}{\Vert u\Vert _{\varepsilon }}\langle J'_{\varepsilon }({\widehat{m}}_{\varepsilon }(u)),v \rangle ,\hspace{5mm}\forall u\in H_{\varepsilon }^{+} \text { and }\forall v\in H_{\varepsilon }. \end{aligned}$$

$(B_{2})$:

$\Psi _{\varepsilon }\in C^{1}(S_{\varepsilon }^{+},{\mathbb {R}})$ and

$$\begin{aligned} \langle \Psi '_{\varepsilon }(u),v\rangle =\Vert m_{\varepsilon }(u)\Vert _{\varepsilon }\langle J'_{\varepsilon }({\widehat{m}}_{\varepsilon }(u)),v\rangle , \hspace{5mm} \forall v\in T_{u}S_{\varepsilon }^{+}. \end{aligned}$$

$(B_{3})$:

If $\{u_{n}\}$ is a $(PS)_{c}$ sequence of $\Psi _{\varepsilon }$, then $\{m_{\varepsilon }(u_{n})\}$ is a $(PS)_{c}$ sequence of $J_{\varepsilon }$. If $\{u_{n}\}\subset {\mathcal {N}}_{\varepsilon }$ is a bounded $(PS)_{c}$ sequence of $J_{\varepsilon }$, then $\{m_{\varepsilon }^{-1}(u_{n})\}$ is a $(PS)_{c}$ sequence of $\Psi _{\varepsilon }$.

$(B_{4})$:

u is a critical point of $\Psi _{\varepsilon }$ if and only if $m_{\varepsilon }(u)$ is a critical point of $J_{\varepsilon }$. Moreover, the corresponding critical values coincide and

$$\begin{aligned} \inf _{u\in S_{\varepsilon }^{+}}\Psi _{\varepsilon }(u)= \inf _{u\in {\mathcal {N}}_{\varepsilon }}J_{\varepsilon }(u). \end{aligned}$$

As in [39], we have the following variational characterization of the infimum of $J_{\varepsilon }$ over ${\mathcal {N}}_{\varepsilon }$:

$$\begin{aligned} c_{\varepsilon } =\inf _{u\in {\mathcal {N}}_{\varepsilon }} J_{\varepsilon }(u) =\inf _{u\in H_{\varepsilon }^{+}}\sup \limits _{t>0} J_{\varepsilon }(tu) =\inf _{u\in S_{\varepsilon }^{+}}\sup \limits _{t>0} J_{\varepsilon }(tu). \end{aligned}$$

(3.10)

Lemma 3.7

Suppose that $(f_{1})$–$(f_{3})$ hold. Assume that $\{u_{n}\}\subset {\mathcal {N}}_{\varepsilon }$ is a $(PS)_{c}$ sequence with

$$\begin{aligned} 0<c_{\varepsilon }\le c <\frac{N+2-\mu }{2(2N-\mu )} S_{H,L}^{\frac{2N-\mu }{N+2-\mu }}, \end{aligned}$$

then $\{u_{n}\}$ is bounded in $H_{\varepsilon }$. Moreover, $\{u_{n}\}$ cannot be vanishing, namely there exist $r, \delta >0$ and a sequence $\{y_{n}\}\subset {\mathbb {R}}^{N}$ such that

$$\begin{aligned} \liminf _{n\rightarrow \infty } \int \limits _{B_{r}(y_{n})}|u_{n}|^{2}\mathrm{{d}}x\ge \delta . \end{aligned}$$

Proof

Firstly, we prove the boundedness of $\{u_{n}\}$. Arguing by contradiction, we suppose that $\{u_{n}\}$ is unbounded in $H_{\varepsilon }$. Without loss of generality, we assume that $\Vert u_{n}\Vert _{\varepsilon }\rightarrow \infty$. Let $v_{n}:=\frac{u_{n}}{\Vert u_{n}\Vert _{\varepsilon }}$, up to a subsequence, then there exists $v\in H_{\varepsilon }$ such that

$$\begin{aligned} \begin{aligned}&v_{n}\rightharpoonup v\text { weakly in } H_{\varepsilon }, \\&v_{n}\rightarrow v\text { strongly in }L_{\text {loc}}^{r}({\mathbb {R}}^{N}), 2\le r<2^{*},\\&v_{n}(x)\rightarrow v(x)\text { a.e. in }{\mathbb {R}}^{N}. \end{aligned} \end{aligned}$$

If $v_{n}$ is vanishing, i.e.,

$$\begin{aligned} \lim _{n\rightarrow \infty }\sup _{y\in {\mathbb {R}}^{N}} \int \limits _{B_{r}(y)}|v_{n}|^{2}(x)\text {d}x=0, \end{aligned}$$

then Lemma 2.4 implies that $v_{n}\rightarrow 0$ in $L^{t}({\mathbb {R}}^{N},{\mathbb {R}})$ for all $t\in (2,2^{*})$. By (3.2) and (3.3), we have

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{F(|tv_{n}(x)|^{2}) F(|tv_{n}(y)|^{2})}{|x-y|^{\mu }}\text {d}y\text {d}x\rightarrow 0,\quad \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{F(|tv_{n}(y)|^{2})}{|x-y|^{\mu }}|tv_{n}(x)|^{2^{*}_{\mu }}\text {d}y\text {d}x\rightarrow 0. \end{aligned}$$

(3.11)

Hence for sufficiently large n, we have

$$\begin{aligned} \begin{aligned} c_{\varepsilon }+o_{n}(1)&=I_{\varepsilon }\left( u_{n}\right) \ge \sup \limits _{t\ge 0}I_{\varepsilon }\left( tv_{n}\right) \\&\ge \sup _{t\ge 0}\left( \frac{t^{2}}{2} -\frac{t^{2\cdot 2_{\mu }^{*}}}{2\cdot 2_{\mu }^{*}}S_{H, L}^{-2_{\mu }^{*}}\right) +o_{n}(1)\\&=\frac{N+2-\mu }{2(2N-\mu )} S_{H,L}^{\frac{2N-\mu }{N+2-\mu }}+o_{n}(1), \end{aligned} \end{aligned}$$

which is a contradiction. Therefore, $\left\{ v_{n}\right\}$ is nonvanishing, namely there exists $y_{n} \in {\mathbb {R}}^{N}$ and $\delta >0$ such that

$$\begin{aligned} \int \limits _{B_{r}\left( y_{n}\right) }|v_{n}|^{2}(x)\text {d}x>\delta . \end{aligned}$$

(3.12)

Denote ${\tilde{v}}_{n}(\cdot )=v_{n}\left( \cdot +y_{n}\right)$, then we can suppose that

$$\begin{aligned} \begin{aligned}&{\tilde{v}}_{n}\rightharpoonup {\tilde{v}} \text{ weakly } \text{ in } H_{\varepsilon }, \\&{\tilde{v}}_{n} \rightarrow {\tilde{v}} \text{ strongly } \text{ in } L_{\text {loc}}^{r}\left( {\mathbb {R}}^{N}\right) , 2\le r<2^{*},\\&{\tilde{v}}_{n}(x) \rightarrow {\tilde{v}}(x) \text{ a.e. } \text{ in } {\mathbb {R}}^{N}. \end{aligned} \end{aligned}$$

By (3.12), we have ${\tilde{v}}\ne 0$. Hence, there exists a measure set E such that ${\tilde{v}}(x) \ne 0$ for $x\in E$. Let $|{\hat{u}}_{n}|:=|{\tilde{v}}_{n}|\Vert u_{n}\Vert _{\varepsilon }$. Then, $|{\hat{u}}_{n}(x)|\rightarrow +\infty$ for $x\in E$. By $(f_{3})$, we have

$$\begin{aligned} \int \limits _{E}\int \limits _{E}\frac{1}{|x-y|^{\mu }} \frac{F(|{\hat{u}}_{n}(y)|^{2})}{|{\hat{u}}(y)|^{2}}|{\tilde{v}}_{n}(y)|^{2} \frac{F(|{\hat{u}}_{n}(x)|^{2})}{|{\hat{u}}(x)|^{2}}|\tilde{v_{n}}(x)|^{2}\text {d}y\text {d}x \rightarrow +\infty . \end{aligned}$$

Hence, we know

$$\begin{aligned} \lim _{n\rightarrow \infty }\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{1}{|x-y|^{\mu }} \frac{F(|{\hat{u}}_{n}(y)|^{2})}{|{\hat{u}}_{n}(y)|^{2}}|{\tilde{v}}_{n}(y)|^{2}\frac{F(|{\hat{u}}_{n}(x)|^{2})}{|{\hat{u}}_{n}(x)|^{2}}|{\tilde{v}}_{n}(x)|^{2}\text {d}y\text {d}x=+\infty , \end{aligned}$$

namely

$$\begin{aligned} \lim _{n\rightarrow \infty }\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{1}{|x-y|^{\mu }}\frac{F(|u_{n}(y)|^{2})}{\Vert u_{n}\Vert _{\varepsilon }^{2}}\frac{F(|u_{n}(x)|^{2})}{\Vert u_{n}\Vert _{\varepsilon }^{2}}\text {d}y\text {d}x=+\infty . \end{aligned}$$

Then, we have

$$\begin{aligned} \frac{c_{\varepsilon }}{\left\| u_{n}\right\| _{\varepsilon }^{4}}+o_{n}(1)=\frac{I_{\varepsilon }\left( u_{n}\right) }{\left\| u_{n}\right\| _{\varepsilon }^{4}} \rightarrow -\infty , \end{aligned}$$

which is a contradiction. Therefore, $\{u_{n}\}$ is bounded in $H_{\varepsilon }$.

Next we prove the second conclusion. By contradiction, if $\left\{ u_n\right\}$ is vanishing, then similar to (3.12), we have

$$\begin{aligned} c_{\varepsilon }+o_n(1)=J_{\varepsilon }\left( u_n\right) =\frac{1}{2}\left\| u_n\right\| _{\varepsilon }^2-\frac{1}{2\cdot 2_\mu ^*} \int \limits _{{\mathbb {R}}^N} \int \limits _{{\mathbb {R}}^N} \frac{\left| u_n(y)\right| ^{2_\mu ^*}\left| u_n(x)\right| ^{2_\mu ^*}}{|x-y|^\mu } \text {d} y \text {d} x+o_n(1) \end{aligned}$$

(3.13)

and

$$\begin{aligned} 0=\left\| u_n\right\| _{\varepsilon }^2-\int \limits _{{\mathbb {R}}^N} \int \limits _{{\mathbb {R}}^N} \frac{\left| u_n(y)\right| ^{2_\mu ^*}\left| u_n(x)\right| ^{2_\mu ^*}}{|x-y|^\mu } \text {d} y \text {d} x+o_n(1). \end{aligned}$$

(3.14)

If $\left\| u_n\right\| _{\varepsilon } \rightarrow 0$, then it follows from (3.13) and (3.14) that $c_{\varepsilon }=0$, which is impossible. Then, $\left\| u_n\right\| _{\varepsilon } \nrightarrow 0$ and by virtue of (3.14) we get

$$\begin{aligned} \left\| u_n\right\| _{\varepsilon }^2 \le S_{H, L}^{2_\mu ^*}\left\| u_n\right\| _{\varepsilon }^{2 \cdot 2_\mu ^*}+o_n(1). \end{aligned}$$

Hence,

$$\begin{aligned} \liminf _{n \rightarrow \infty }\left\| u_n\right\| _{\varepsilon }^2 \ge S_{H, L}^{\frac{2 N-\mu }{N+2-\mu }}. \end{aligned}$$

From (3.13) and (3.14), we deduce that

$$\begin{aligned} c_{\varepsilon }+o_n(1)=I_{\varepsilon }\left( u_n\right) \ge \frac{N+2-\mu }{2(2 N-\mu )} S_{H, L}^{\frac{2 N-\mu }{N+2-\mu }}, \end{aligned}$$

which yields a contradiction. Therefore, $\left\{ u_n\right\}$ is nonvanishing. $\square$

Lemma 3.8

Assume $(V_{1})$–$(V_{2})$ and $(f_{1})$–$(f_{3})$ hold, the functional $J_{\varepsilon }$ satisfies the $(PS)_{c}$ condiction for all $c \in$ $\left[ c_{\varepsilon }, \frac{N+2-\mu }{2(2N-\mu )} S_{H,L}^{\frac{2N-\mu }{N+2-\mu }}\right)$.

Proof

Let $\left\{ u_{n}\right\}$ be a $(PS)_{c}$ sequence for $J_{\varepsilon }$. By Lemma 3.7, $\left\{ u_{n}\right\}$ is bounded in $H_{\varepsilon }$. Therefore, up to a subsequence, $u_{n}\rightharpoonup u$ in $H_{\varepsilon }$ and $u_{n} \rightarrow u$ in $L_{loc}^{r}({\mathbb {R}}^{N})$ for any $r \in [2,2^{*})$.

Let $R>0$ be such that $\Lambda _{\varepsilon }\subset B_{R}(0)$. We show that for each $\xi >0$ and R large enough, it holds that

$$\begin{aligned} \limsup _{n\rightarrow \infty } \int \limits _{B_{R}^{c}}|\nabla _{\varepsilon }u_{n}|^{2} +V_{\varepsilon }|u_{n}|^{2}\text {d}x<\xi . \end{aligned}$$

(3.15)

Let us consider a cut-off function $\eta _{R} \in C^{\infty }({\mathbb {R}}^{N},{\mathbb {R}})$ defined as

$$\begin{aligned} \eta _{R}(x)= {\left\{ \begin{array}{ll} 0 & \text{ if } x\in B_{R}, \\ 1 & \text{ if } x\notin B_{2R}, \end{array}\right. } \end{aligned}$$

and $|\nabla \eta _{R}|_{\infty }\le C/R$, where $C>0$ is a constant independent of R. Since $\{u_{n}\eta _{R}\}$ is bounded in $H_{\varepsilon }$, it follows that

$$\begin{aligned} \langle J_{\varepsilon }^{\prime }(u_{n}), u_{n}\eta _{R}\rangle =o_{n}(1), \end{aligned}$$

therefore,

$$\begin{aligned} \begin{aligned} \Re \left( \,\,\int \limits _{{\mathbb {R}}^{N}}|\nabla _{\varepsilon } u_{n}\overline{\nabla _{\varepsilon }(u_{n}\eta _{R})}\text {d}x\right)&+\int \limits _{{\mathbb {R}}^{N}}V_{\varepsilon }|u_{n}|^{2}\eta _{R}\text {d}x\\&=\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g(\varepsilon x,|u_{n}|^{2})|u_{n}|^{2}\eta _{R}\text {d}x+o_{n}(1). \end{aligned} \end{aligned}$$

Since $\overline{\nabla _{\varepsilon }(u_{n}\eta _{R})} =i{\overline{u}}_{n}\nabla \eta _{R}+\eta _{R}\overline{\nabla _{\varepsilon }u_{n}}$, using $(g_{4})$, we obtain

$$\begin{aligned} \begin{aligned} \int \limits _{{\mathbb {R}}^{N}}&\left( \left| \nabla _{\varepsilon } u_{n}\right| ^{2}+V_{\varepsilon }\left| u_{n}\right| ^{2}\right) \eta _{R}\text {d}x\\ =&\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x,|u_{n}|^{2})\right) g(\varepsilon x,|u_{n}|^{2})|u_{n}|^{2}\eta _{R}\text {d}x -\Re \left( \,\,\int \limits _{{\mathbb {R}}^{N}}i{\bar{u}}_{n} \nabla _{\varepsilon }u_{n}\nabla \eta _{R} \text {d} x\right) +o_{n}(1)\\ \le&\frac{k_{0}}{2\cdot 2_{\mu }^{*}K}\int \limits _{{\mathbb {R}}^{N}}V_{\varepsilon }(x)|u(x)|^{2}\text {d}x\text {d}y-\Re \left( \,\,\int \limits _{{\mathbb {R}}^{N}}i{\bar{u}}_{n} \nabla _{\varepsilon }u_{n}\nabla \eta _{R}\text {d}x\right) +o_{n}(1).\\ \end{aligned} \end{aligned}$$

By the Hölder inequality and the boundedness of $\left\{ u_{n}\right\}$ in $H_{\varepsilon }$, we deduce that

$$\begin{aligned} \begin{aligned} \bigg (1-\frac{k_{0}}{2\cdot 2_{\mu }^{*}K}\bigg )\left( \int \limits _{{\mathbb {R}}^{N}}(|\nabla _{\varepsilon }u_{n}|^{2} +V_{\varepsilon }|u_{n}|^{2})\text {d}x\right)&\le \frac{C}{R}|{\bar{u}}_{n}|_{2}|\nabla _{\varepsilon } u_{n}|_{2}+o_{n}(1)\\&\le \frac{C_{1}}{R}+o_{n}(1), \end{aligned} \end{aligned}$$

and so (3.15) holds.

Next we prove that $u_{n} \rightarrow u$ in $H_{\varepsilon }$ as $n \rightarrow \infty$. Setting $\omega _{n}=\left\| u_{n}-u\right\| _{\varepsilon }^{2}$, we have

$$\begin{aligned} \omega _{n}=\langle J'_{\varepsilon }(u_{n}), u_{n}\rangle -\langle J'_{\varepsilon }(u_{n}),u\rangle +\frac{1}{2_{\mu }^{*}} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g(\varepsilon x, |u_{n}|^{2})(|u_{n}|^{2}-|u|^{2})\text {d}x+o_{n}(1). \end{aligned}$$

Observe that $\left\langle J_{\varepsilon }^{\prime }(u_{n}), u_{n}\right\rangle =\left\langle J_{\varepsilon }^{\prime }\left( u_{n}\right) , u\right\rangle =o_{n}(1)$, so we only need to prove that

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g(\varepsilon x, |u_{n}|^{2})(|u_{n}|^{2}-|u|^{2})\text {d}x=o_{n}(1)\text { as } n\rightarrow \infty . \end{aligned}$$

By the [21, lemma 2.4], we have

$$\begin{aligned} \lim _{n\rightarrow \infty }\int \limits _{B_{R}} \left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g(\varepsilon x, |u_{n}|^{2})(|u_{n}|^{2}-|u|^{2})\text {d}x=0. \end{aligned}$$

Now, by $(g_{4})$ and (3.15) we obtain

$$\begin{aligned} \int \limits _{B_{R}^{c}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g(\varepsilon x, |u_{n}|^{2})(|u_{n}|^{2}-|u|^{2})\text {d}x \le \frac{k_{0}}{K}\int \limits _{B_{R}^{c}}|\nabla _{\varepsilon }u_{n}|^{2}+V_{\varepsilon }|u_{n}|^{2}\text {d}x<\frac{k_{0}}{K}\xi . \end{aligned}$$

Therefore, we have

$$\begin{aligned} \lim _{n\rightarrow \infty } \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g(\varepsilon x, |u_{n}|^{2})(|u_{n}|^{2}-|u|^{2})\text {d}x=0. \end{aligned}$$

$\square$

Since f is only assumed to be continuous, the following Lemma is required for the multiplicity result in the next section.

Lemma 3.9

The functional $\Psi _{\varepsilon }$ satisfies the $(PS)_{c}$ condiction in $S_{\varepsilon }^{+}$ for all $c \in$ $\left[ c_{\varepsilon }, \frac{N+2-\mu }{2(2N-\mu )} S_{H,L}^{\frac{2N-\mu }{N+2-\mu }}\right)$.

Proof

Let $\{u_{n}\} \subset S_{\varepsilon }^{+}$ be a $(PS)_{c}$ sequence for $\Psi _{\varepsilon }$. Thus, $\Psi _{\varepsilon }\left( u_{n}\right) \rightarrow c$ and $\left\| \Psi _{\varepsilon }^{\prime }\right\| _{*} \rightarrow 0$, where $\Vert \cdot \Vert _{*}$ is the norm in the dual space $(T_{u_{n}} S_{\varepsilon }^{+})^{*}$. It follows from Lemma 3.6$(B_{3})$ that $\{m_{\varepsilon }(u_{n})\}$ is a $(PS)_{c}$ sequence for $J_{\varepsilon }$ in $H_{\varepsilon }$. From Lemma 3.8, we see that there exists a $u \in S_{\varepsilon }^{+}$ such that $m_{\varepsilon }(u_{n}) \rightarrow m_{\varepsilon }(u)$ in $H_{\varepsilon }$. From Lemma 3.5$(A_{3})$, we conclude that

$$\begin{aligned} u_{n}\rightarrow u\text { in } S_{\varepsilon }^{+}, \end{aligned}$$

and the proof is complete. $\square$

4 The autonomous problem

For our scope, we start by considering the limit problem associated to (2.1), namely the problem

$$\begin{aligned} -\Delta u+V_{0}u =\left( \frac{|u(y)|^{2_{\mu }^{*}}+F(|u(y)|^{2})}{|x-y|^{\mu }}\right) \left( |u|^{2_{\mu }^{*}-2}u+\frac{1}{2_{\mu }^{*}}f(|u|^{2})u\right) \hspace{1.14mm}\text { in }\hspace{1.14mm}{\mathbb {R}}^{N}, \end{aligned}$$

(4.1)

which has the following associated functional

$$\begin{aligned} I_{0}(u)=\frac{1}{2}\int \limits _{{\mathbb {R}}^{N}}(|\nabla u|^{2} +V_{0}|u|^{2})\text {d}x -\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{(|u(x)|^{2_{\mu }^{*}}+F(|u(x)|^{2}))(|u(y)|^{2_{\mu }^{*}}+F(|u(y)|^{2}))}{|x-y|^{\mu }}\text {d}x\text {d}y. \end{aligned}$$

The functional $I_{0}$ is well defined on the Hilbert space $H_{0}=H^{1}({\mathbb {R}}^{N},{\mathbb {R}})$ with the inner product

$$\begin{aligned} \langle u,v\rangle _{V_{0}} =\int \limits _{{\mathbb {R}}^{N}}(\nabla u\nabla v+V_{0}uv)\text {d}x, \end{aligned}$$

and the norm

$$\begin{aligned} \Vert u\Vert _{V_{0}}=\left( \,\,\int \limits _{{\mathbb {R}}^{N}}(|\nabla u|^{2}+V_{0}|u|^{2})\text {d}x\right) ^{\frac{1}{2}}. \end{aligned}$$

We denote the Nehari manifold associated to $I_{0}$ by

$$\begin{aligned} {\mathcal {N}}_{0}=\{u\in H_{0}\setminus \{0\}:\langle I'_{0}(u),u\rangle =0\}. \end{aligned}$$

and the least energy on ${\mathcal {N}}_{0}$ is defined by

$$\begin{aligned} c_{V_{0}}=\inf _{u\in {\mathcal {N}}_{0}} I_{0}(u). \end{aligned}$$

Let $H_{0}^{+}$ be the open subset of $H_{0}$ given by

$$\begin{aligned} H_{0}^{+}=\{u\in H_{0}:|{\text {supp}}(u)\cap \Lambda _{\varepsilon }|>0\}, \end{aligned}$$

and $S_{0}^{+}=S_{0}\cap H_{0}^{+}$, where $S_{0}$ is the unit sphere of $H_{0}$. Note that $S_{0}^{+}$ is a non-complete $C^{1,1}$-manifold of codimension 1, modeled on $H_{0}$ and contained in $H_{0}^{+}$. Therefore, $H_{0}=T_{u}S_{0}^{+}\oplus {\mathbb {R}}u$ for each $u\in S_{0}^{+}$, where $T_{u}S_{0}^{+}=\{v\in H_{0}:\langle u,v\rangle _{V_{0}}=0\}$.

Next we have the following Lemmas and the proofs that follow from a similar argument used in the proofs of Lemmas 3.5 and 3.6.

Lemma 4.1

Let $V_{0}$ be given in $(V_{1})$ and suppose that f satisfies $(f_{1})$–$(f_{3})$. Then, the following properties hold:

$(a_{1})$:

For each $u\in H_{0}\setminus \{0\}$, let $\phi _{u}:{\mathbb {R}}^{+}\rightarrow {\mathbb {R}}$ be given by $\phi _{u}(t)=I_{0}(tu)$. Then, there exists a unique $t_{u}>0$ such that $\phi '_{u}(t)>0$ in $(0, t_{u})$ and $\phi '_{u}(t)<0$ in $(t_{u}, \infty )$.

$(a_{2})$:

There is a $\sigma >0$ independent on u such that $t_{u}>\sigma$ for all $u\in S_{0}^{+}$. Moreover, for each compact set ${\mathcal {W}}\subset S_{0}^{+}$, there is $C_{{\mathcal {W}}}>0$ such that $t_{u}\le C_{{\mathcal {W}}}$ for all $u\in {\mathcal {W}}$.

$(a_{3})$:

The map ${\widehat{m}}:H_{0}\setminus \{0\}\rightarrow {\mathcal {N}}_{0}$ given by ${\widehat{m}}(u)=t_{u}u$ is continuous and $m={\widehat{m}}|_{S_{0}^{+}}$ is a homeomorphism between $S_{0}^{+}$ and ${\mathcal {N}}_{0}$. Moreover, $m^{-1}(u)=\frac{u}{\Vert u\Vert _{0}}$.

$(a_{4})$:

If there is a sequence $\{u_{n}\}\subset S_{\varepsilon }^{+}$ such that ${\text {dist}}(u_{n}, \partial S_{\varepsilon }^{+})\rightarrow 0$, then we have $\Vert m_{\varepsilon }(u_{n})\Vert _{\varepsilon }\rightarrow \infty$ and $I_{0}(m_{\varepsilon }(u_{n}))\rightarrow \infty$.

We shall consider the functional defined by

$$\begin{aligned} {\widehat{\Psi }}_{0}(u)=I_{0}({\widehat{m}}(u)) \hspace{1cm}\text {and}\hspace{1cm} \Psi _{0}=({\widehat{\Psi }}_{0})|_{S_{0}^{+}}. \end{aligned}$$

Lemma 4.2

Let $V_{0}$ be given in $(V_{1})$ and suppose that f satisfies $(f_{1})$-$(f_{3})$, then

$(b_{1})$:

${\widehat{\Psi }}_{0}\in C^{1}(H_{0}^{+},{\mathbb {R}})$ and

$$\begin{aligned} \langle {\widehat{\Psi }}'_{0}(u),v\rangle =\frac{\Vert {\widehat{m}}(u)\Vert _{0}}{\Vert u\Vert _{0}}\langle I'_{0}({\widehat{m}}(u)),v\rangle ,\hspace{2mm} \forall u\in H_{0}^{+} \text { and }\hspace{2mm} \forall v\in H_{0}. \end{aligned}$$

$(b_{2})$:

$\Psi _{0}\in C^{1}(S_{0}^{+},{\mathbb {R}})$ and

$$\begin{aligned} \langle \Psi '_{0}(u),v\rangle =\Vert m(u)\Vert _{0}\langle I'_{0}(m(u)),v\rangle ,\forall v\in TS_{0}^{+}. \end{aligned}$$

$(b_{3})$:

If $\{u_{n}\}$ is a $(PS)_{c}$ sequence of $\Psi _{0}$, then $\{m(u_{n})\}$ is a $(PS)_{c}$ sequence of $I_{0}$. If $\{u_{n}\}\subset {\mathcal {N}}_{0}$ is a bounded $(PS)_{c}$ sequence of $I_{0}$, then $\{m^{-1}(u_{n})\}$ is a $(PS)_{c}$ sequence of $\Psi _{0}$.

$(b_{4})$:

u is a critical point of $\Psi _{0}$ if and only if m(u) is a critical point of $I_{0}$. Moreover, the corresponding critical values coincide and

$$\begin{aligned} \inf _{u\in S_{0}^{+}}\Psi _{\varepsilon }(u)= \inf _{u\in {\mathcal {N}}_{0}}I_{0}(u). \end{aligned}$$

Similar to the previous argument, we have the following variational characterization of the infimum of $I_{0}$ over ${\mathcal {N}}_{0}$:

$$\begin{aligned} c_{V_{0}} =\inf _{u\in {\mathcal {N}}_{0}} I_{0}(u) =\inf _{u\in H_{0}^{+}}\sup \limits _{t>0} I_{0}(tu) =\inf _{u\in S_{0}^{+}}\sup \limits _{t>0} I_{0}(tu). \end{aligned}$$

The next result is useful in later arguments.

Lemma 4.3

Let $\{u_{n}\}\in H_{0}$ be a $(PS)_{c}$ sequence with $c\in \left[ c_{V_{0}},\frac{N+2-\mu }{2(2N-\mu )} S_{H,L}^{\frac{2N-\mu }{N+2-\mu }}\right)$ for $I_{0}$ such that $u_{n}\rightharpoonup 0$. Then, only one of the following conclusions holds.

(i):

$u_{n}\rightarrow 0$ in $H_{0}$ as $n\rightarrow \infty$;

(ii):

there exists $R, \beta >0$ and $\{y_{n}\}\subset {\mathbb {R}}^{N}$ such that

$$\begin{aligned} \int \limits _{B_{R}(y_{n})}|u_{n}|^{2}\ge \beta . \end{aligned}$$

Proof

Suppose that (ii) does not hold. Then, for any $R>0$, we have

$$\begin{aligned} \lim \limits _{n}\sup \limits _{y\in {\mathbb {R}}^{N}}\int \limits _{B_{R}(y_{n})}|u_{n}|^{2}=0. \end{aligned}$$

Similarly to Lemma 3.7, $\{u_{n}\}$ is bounded in $H_{0}$. Then, by Lemma 2.4, we know

$$\begin{aligned} u_{n}\rightarrow 0 \hspace{2mm}\text {in}\hspace{2mm}L^{r}({\mathbb {R}}^{N}), r\in (2,2^{*}). \end{aligned}$$

Thus, by (f2) we have

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{(|u_{n}(x)|^{2_{\mu }^{*}}+F(|u_{n}(x)|^{2}))(|u_{n}(y)|^{2_{\mu }^{*}}+F(|u_{n}(y)|^{2}))}{|x-y|^{\mu }}\text {d}x\text {d}y =o_{n}(1). \end{aligned}$$

From $\langle I_{0}^{\prime }(u_{n}),u_{n}\rangle \rightarrow 0$, we get

$$\begin{aligned} \left\| u_{n}\right\| _{0}^{2}=o_{n}(1). \end{aligned}$$

Therefore the conclusion follows. $\square$

From Lemma 4.3, we can see that, if u is the weak limit of a $(PS)_{c_{V_{0}}}$ sequence $\{u_{n}\}$ for the functional $I_{0}$, then we can assume $u_{n}\ne 0$. Otherwise we would have $u_{n}\rightharpoonup 0$ and once it does not occur that $u_{n} \rightarrow 0$, we conclude from Lemma 4.3 that there exist $\{y_{n}\}\subset {\mathbb {R}}^{N}$ and $R, \beta >0$ such that

$$\begin{aligned} \liminf _{n\rightarrow +\infty } \int \limits _{B_{R}\left( y_{n}\right) }|u_{n}|^{2}\text {d}x\ge \beta >0. \end{aligned}$$

Then set $v_{n}(x)=u_{n}(x+y_{n})$, making a change of variable, we can prove that $\{v_{n}\}$ is also a $(P S)_{c_{V_{0}}}$ sequence for the functional $I_{0}$, it is bounded in $H_{0}$ and there exists $v \in H_{0}$ such that $v_{n} \rightarrow v$ in $H_{0}$ with $v \ne 0$.

Next we devote to estimating the least energy $c_{V_{0}}$.

Lemma 4.4

There exists $u_{\varepsilon }$ such that

$$\begin{aligned} \sup _{t\ge 0}I_{0}(tu_{\varepsilon })<\frac{N+2-\mu }{2(2 N-\mu )}(S_{H,L})^{\frac{2N-\mu }{N+2-\mu }}, \end{aligned}$$

provided that one of the following conditions holds:

(i):

$\frac{N+2-\mu }{N-2}<p<2_{\mu }^{*}, N=3,4$ and $\lambda >0$;

(ii):

$\frac{2N-\mu }{N}<p\le \frac{N+2-\mu }{N-2}, N=3,4$ and $\lambda$ sufficiently large;

(iii):

$\frac{2N+2-\mu }{N-2}<p<2_{\mu }^{*}, N\ge 5$ and $\lambda >0$;

(iv):

$\frac{2 N-\mu }{N}<p\le \frac{2N-2-\mu }{N-2}, N\ge 5$ and $\lambda$ sufficiently large.

Proof

We consider $\psi \in C_{0}^{\infty }\left( {\mathbb {R}}^{N}\right)$ such that

$$\begin{aligned} \psi (x)=\left\{ \begin{array}{l} 1, \text{ if } x \in B_{\delta }, \\ 0, \text{ if } x \in {\mathbb {R}}^{N} \backslash B_{2 \delta }, \end{array} \quad 0 \le |\psi (x)| \le 1, \quad |D \psi (x)| \le C, \right. \end{aligned}$$

where $C>0$ is a constant. For $\varepsilon >0$, we define

$$\begin{aligned} \begin{aligned} U_{\varepsilon }(x)&:=\varepsilon ^{\frac{2-N}{2}}U\left( \frac{x}{\varepsilon }\right) ,\\ u_{\varepsilon }(x)&:=\psi (x)U_{\varepsilon }(x).\\ \end{aligned} \end{aligned}$$

From [21], we have the estimates

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\left| \nabla u_{\varepsilon }\right| ^{2} \text {d}x=C(N,\mu )^{\frac{N-2}{2N-\mu } \cdot \frac{N}{2}} S_{H,L}^{\frac{N}{2}}+O\left( \varepsilon ^{N-2}\right) \end{aligned}$$

(4.2)

and

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{\left| u_{\varepsilon }(x)\right| ^{2_{\mu }^{*}}\left| u_{\varepsilon }(y)\right| ^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y \ge C(N,\mu )^{\frac{N}{2}}S_{H,L}^{\frac{2 N-\mu }{2}}-O\left( \varepsilon ^{N-\frac{\mu }{2}}\right) . \end{aligned}$$

(4.3)

By direct computation, we have

$$\begin{aligned} \begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}&\frac{|u_{\varepsilon }(x)|^{p}|u_{\varepsilon }(y)|^{p}}{|x-y|^{\mu }}\text {d}x\text {d}y =\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{|\psi (x) U_{\varepsilon }(x)|^{p}|\psi (y) U_{\varepsilon }(y)|^{p}}{|x-y|^{\mu }}\text {d}x\text {d}y\\&=\int \limits _{B_{2\delta }}\int \limits _{B_{2\delta }}\frac{|\psi (x) U_{\varepsilon }(x)|^{p}|\psi (y) U_{\varepsilon }(y)|^{p}}{|x-y|^{\mu }}\text {d}x\text {d}y \ge \int \limits _{B_{\delta }} \int \limits _{B_{\delta }} \frac{|U_{\varepsilon }(x)|^{p}|U_{\varepsilon }(y)|^{p}}{|x-y|^{\mu }}\text {d}x\text {d}y\\&=[N(N-2)]^{(N-2)p}\varepsilon ^{(2-N)p}\int \limits _{B_{\delta }} \int \limits _{B_{\delta }} \frac{1}{\left( 1+\left| \frac{x}{\varepsilon }\right| ^{2}\right) \frac{(N-2)p}{2}|x-y|^{\mu }\left( 1+\left| \frac{y}{\varepsilon }\right| ^{2}\right) ^{\frac{(N-2)p}{2}}}\text {d}x\text {d}y\\&=[N(N-2)]^{\frac{(N-2)p}{2}}\varepsilon ^{2 N-\mu -(N-2)p}\int \limits _{B_{\delta }}\int \limits _{B_{\delta }} \frac{1}{\left( 1+|x|^{2}\right) ^{\frac{(N-2) p}{2}}|x-y|^{\mu }\left( 1+|y|^{2}\right) ^{\frac{(N-2) p}{2}}}\text {d}x\text {d}y\\&=C_{3} \varepsilon ^{2N-\mu -(N-2)p}. \end{aligned} \end{aligned}$$

(4.4)

Case 1. $\frac{N+2-\mu }{N-2}<p<2_{\mu }^{*}$ and $N=3,4$ or $\frac{2N-2-\mu }{N-2}<p<2_{\mu }^{*}$ and $N\ge 5$.

From Lemma 4.1, there exists unique $t_{\varepsilon }>0$ such that $t_{\varepsilon }u_{\varepsilon }\in {\mathcal {N}}_{0}$. Now we estimate $I_{0}(t_{\varepsilon }u_{\varepsilon })$. Notice that

$$\begin{aligned} \begin{aligned} I_{0}(t_{\varepsilon }u_{\varepsilon }) =&\frac{t_{\varepsilon }^{2}}{2}\int \limits _{{\mathbb {R}}^{N}}(|\nabla u_{\varepsilon }|^{2}+V_{0}|u_{\varepsilon }|^{2})\text {d}x\\&-\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{(|t_{\varepsilon }u_{\varepsilon }(x)|^{2_{\mu }^{*}}+F(|t_{\varepsilon }u_{\varepsilon }(x)|^{2}))(|t_{\varepsilon }u_{\varepsilon }(y)|^{2_{\mu }^{*}}+F(|t_{\varepsilon }u_{\varepsilon }(y)|^{2}))}{|x-y|^{\mu }}\text {d}x\text {d}y\\ \le&\frac{t_{\varepsilon }^{2}}{2}\int \limits _{{\mathbb {R}}^{N}} (|\nabla u_{\varepsilon }|^{2}+V_{0}|u_{\varepsilon }|^{2})\text {d}x -\frac{t_{\varepsilon }^{2\cdot 2_{\mu }^{*}}}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{|u_{\varepsilon }(x)|^{2_{\mu }^{*}}|u_{\varepsilon }(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y\\ &-\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{F(|t_{\varepsilon }u_{\varepsilon }(x)|^{2})F(|t_{\varepsilon }u_{\varepsilon }(y)|^{2})}{|x-y|^{\mu }}\text {d}x\text {d}y\\ \le&\left( \frac{t_{\varepsilon }^{2}}{2}\int \limits _{{\mathbb {R}}^{N}}|\nabla u_{\varepsilon }|^{2}\text {d}x -\frac{t_{\varepsilon }^{2\cdot 2_{\mu }^{*}}}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{|u_{\varepsilon }(x)|^{2_{\mu }^{*}}|u_{\varepsilon }(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y\right) \\&+\left( \frac{t_{\varepsilon }^{2}}{2}\int \limits _{{\mathbb {R}}^{N}}V_{0}|u_{\varepsilon }|^{2}\text {d}x -\frac{\lambda ^{2}}{2\cdot 2_{\mu }^{*}}t_{\varepsilon }^{2p}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{|u_{\varepsilon }(x)|^{p}|u_{\varepsilon }(y)|^{p}}{|x-y|^{\mu }}\text {d}x\text {d}y\right) \\:=&I_{1}+I_{2}. \end{aligned} \end{aligned}$$

For $I_{1}$, we may assume that

$$\begin{aligned} \begin{aligned}&E:=\int \limits _{{\mathbb {R}}^{N}}|\nabla u_{\varepsilon }|^{2}\text {d}x,\\&F=\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \frac{|u_{\varepsilon }(x)|^{2_{\mu }^{*}}|u_{\varepsilon }(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y, \end{aligned} \end{aligned}$$

and consider the function $\theta :[0,\infty )\rightarrow {\mathbb {R}}$ defined by

$$\begin{aligned} \theta (t)=\frac{1}{2}Et^{2}- \frac{1}{2\cdot 2_{\mu }^{*}}Ft^{2\cdot 2_{\mu }^{*}}. \end{aligned}$$

We have that $t_{0}=\left( \frac{E}{F}\right) ^{\frac{1}{2\cdot 2_{\mu }^{*}-2}}$ is a maximum point of $\theta$ and

$$\begin{aligned} \theta (t_{0})=\frac{2_{\mu }^{*}-1}{2\cdot 2_{\mu }^{*}} E^{\frac{2_{\mu }^{*}}{2_{\mu }^{*}-1}} \frac{1}{F^{1-2_{\mu }^{*}}}. \end{aligned}$$

Hence, we have

$$\begin{aligned} & \begin{aligned} I_{1}&\le \frac{N+2-\mu }{2(2N-\mu )} \left( \frac{\int \limits _{{\mathbb {R}}^{N}}|\nabla u_{\varepsilon }|^{2}\text {d}x}{\left( \,\,\int \limits _{{\mathbb {R}}^{N}} \int \limits _{{\mathbb {R}}^{N}} \frac{|u_{\varepsilon }(x)|^{2_{\mu }^{*}}|u_{\varepsilon }(y)|^{2_{\mu }^{*}}}{|x-y|^{\mu }}\text {d}x\text {d}y\right) ^{\frac{N-2}{2 N-\mu }}}\right) ^{\frac{2N-\mu }{N+2-\mu }}\\&\le \frac{N+2-\mu }{2(2N-\mu )} \left( \frac{(C(N,\mu ))^{\frac{N-2}{2N-\mu }\cdot \frac{N}{2}} S_{H,L}^{\frac{N}{2}}+O\left( \varepsilon ^{N-2}\right) }{\left( C(N,\mu )^{\frac{N}{2}}S_{H, L}^{\frac{2 N-\mu }{2}}-O\left( \varepsilon ^{\frac{2 N-\mu }{2}}\right) \right) ^{\frac{N-2}{2 N-\mu }}}\right) ^{\frac{2N-\mu }{N+2-\mu }}\\ \end{aligned}\\ & \begin{aligned}&=\frac{N+2-\mu }{2(2N-\mu )} \left( S_{H, L}\right) ^{\frac{2N-\mu }{N+2-\mu }} \left( \frac{1+O\left( \varepsilon ^{N-2}\right) }{\left( 1-O\left( \varepsilon ^{\frac{2N-\mu }{2}}\right) \right) ^{\frac{2 N-\mu }{N+2-\mu }}}\right) ^{\frac{2N-\mu }{N+2-\mu }}\\&\le \frac{N+2-\mu }{2(2N-\mu )} \left( S_{H,L}\right) ^{\frac{2N-\mu }{N+2-\mu }}\left( 1+C(N, \mu )\frac{O\left( \varepsilon ^{N-2}\right) +O\left( \varepsilon ^{\frac{2N-\mu }{2}}\right) }{\left( 1-O\left( \varepsilon ^{\frac{2 N-\mu }{2}}\right) \right) ^{\frac{N-2}{2N-\mu }}}\right) . \end{aligned} \end{aligned}$$

Observing that for $\varepsilon >0$ sufficiently small, it holds

$$\begin{aligned} \left( 1-O\left( \varepsilon ^{\frac{N-2}{2 N-\mu }}\right) \right) ^{\frac{N-2}{2 N-\mu }}\ge \frac{1}{2}. \end{aligned}$$

Therefore, we conclude that for any $\varepsilon >0$ sufficiently small, we have

$$\begin{aligned} \begin{aligned} I_{1}\left( t_{\varepsilon } u_{\varepsilon }\right) \le \frac{N+2-\mu }{2(2N-\mu )} \left( S_{H,L}\right) ^{\frac{2N-\mu }{N+2-\mu }}+O\left( \varepsilon ^{\min \left\{ N-2,\frac{2N-\mu }{2}\right\} }\right) . \end{aligned} \end{aligned}$$

(4.5)

We claim that there exists $C_{0}>0$ such that for all $\varepsilon >0$,

$$\begin{aligned} t_{\varepsilon }^{2p}\ge C_{0}. \end{aligned}$$

In fact, assume that there exists a sequence $\{\varepsilon _{n}\}$ and $\varepsilon _{n} \rightarrow 0$ as $n \rightarrow \infty$, such that $t_{\varepsilon _{n}} \rightarrow 0$ as $n \rightarrow \infty$. Thus,

$$\begin{aligned} 0<c_{V_{0}}\le \sup _{t\ge 0}I_{0}(tu_{\varepsilon _{n}}) =I_{0}(t_{\varepsilon _{n}}u_{\varepsilon _{n}}). \end{aligned}$$

Since $u_{\varepsilon _{n}}\in H_{0}$ is bounded and $t_{\varepsilon _{n}} \rightarrow 0$ as $n \rightarrow \infty$, we have $t_{\varepsilon _{n}} u_{\varepsilon _{n}} \rightarrow 0$ in $H_{0}$.

The continuity of $I_{0}$ implies that $I_{0}(t_{\varepsilon _{n}} u_{\varepsilon _{n}})\rightarrow I_{0}(0)=0$. Therefore,

$$\begin{aligned} 0<c_{V_{0}} \le \lim _{n \rightarrow \infty } I_{0}(t_{\varepsilon _{n}} u_{\varepsilon _{n}})=0, \end{aligned}$$

which is a contradiction. The claim is holds.

From (4.5) and the above claim, we have

$$\begin{aligned} I_{0}(t_{\varepsilon } u_{\varepsilon })<\frac{N+2-\mu }{2(N-\mu )}(S_{H,L})^{\frac{2 N-\mu }{N+2-\mu }}+O\left( \varepsilon ^{\eta }\right) +C_{2} \int \limits _{{\mathbb {R}}^{N}}|u_{\varepsilon }(x)|^{2}\text {d}x -C_{3}\varepsilon ^{2N-\mu -(N-2)p}. \end{aligned}$$

(4.6)

We want to obtain that

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0} \varepsilon ^{-\eta }(C_{2}\int \limits _{{\mathbb {R}}^{N}}|u_{\varepsilon }(x)|^{2}\text {d}x-C_{3}\varepsilon ^{2N-\mu -(N-2)p})=-\infty . \end{aligned}$$

(4.7)

In order to do that, it suffices to show that

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0} \varepsilon ^{-\eta }(C_{2}\int \limits _{B_{\delta }}|u_{\varepsilon }(x)|^{2}\text {d}x-C_{3}\varepsilon ^{2N-\mu -(N-2)p})=-\infty \end{aligned}$$

(4.8)

and

$$\begin{aligned} C_{2}\int \limits _{B_{2\delta }\backslash B_{\delta }} |u_{\varepsilon }(x)|^{2}\text {d}x-C_{3}\varepsilon ^{2N-\mu -(N-2) p}=O\left( \varepsilon ^{\eta }\right) . \end{aligned}$$

(4.9)

Suppose that (4.8) and (4.9) hold, let us continue to complete our proof. Since

$$\begin{aligned} \begin{aligned} O\left( \varepsilon ^{\eta }\right) +C_{2}\int \limits _{{\mathbb {R}}^{N}}|u_{\varepsilon }(x)|^{2}\text {d}x&-C_{3}\varepsilon ^{2N-\mu -(N-2)p}\\&=\varepsilon ^{\eta }\left[ \frac{O\left( \varepsilon ^{\eta }\right) }{\varepsilon ^{\eta }} +\varepsilon ^{-\eta }\left( C_{2} \int \limits _{{\mathbb {R}}^{N}}|u_{\varepsilon }(x)|^{2}\text {d}x -C_{3}\varepsilon ^{2N-\mu -(N-2)p}\right) \right] , \end{aligned} \end{aligned}$$

from (4.7), we have

$$\begin{aligned} O\left( \varepsilon ^{\eta }\right) +C_{2} \int \limits _{{\mathbb {R}}^{N}}|u_{\varepsilon }(x)|^{2}\text {d}x -C_{3}\varepsilon ^{2N-\mu -(N-2)p}<0 \end{aligned}$$

(4.10)

for $\varepsilon >0$ sufficiently small. Thus, (4.6) and (4.10) guarantee

$$\begin{aligned} \sup _{t\ge 0}I_{0}(t u_{\varepsilon })<\frac{N+2-\mu }{2(2 N-\mu )}\left( S_{H,L}\right) ^{\frac{2N-\mu }{N+2-\mu }} \end{aligned}$$

for $\varepsilon >0$ sufficiently small and fixed. Once (4.8) and (4.9) are verified, the proof of Case 1 is complete.

Now we prove (4.8). By direct computation, we know

$$\begin{aligned} \int \limits _{B_{\delta }}|u_{\varepsilon }(x)|^{2}\text {d}x=N \omega _{N}[N(N-2)]^{\frac{N-2}{2}} \varepsilon ^{2} \int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{N-1}}{\left( 1+r^{2}\right) ^{N-2}}\text {d}r, \end{aligned}$$

where $\omega _{N}$ denotes the volume of the unit ball in ${\mathbb {R}}^{N}$.

Let

$$\begin{aligned} \begin{aligned} I_{\varepsilon }:&=\varepsilon ^{-\eta }\left( C_{2} \int \limits _{B_{\delta }}|u_{\varepsilon }(x)|^{2}\text {d}x -C_{3} \varepsilon ^{2N-\mu -(N-2)p}\right) \\&=\varepsilon ^{-\eta }\left( C_{4}\varepsilon ^{2} \int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{N-1}}{\left( 1+r^{2}\right) ^{N-2}}\text {d}r -C_{3}\varepsilon ^{2N-\mu -(N-2)p}\right) . \end{aligned} \end{aligned}$$

The Case N=3. In this case, we have $5-\mu<p<2_{\mu }^{*}$, thus $5-\mu -p<0$. And we know $\min \left\{ N-2, \frac{2N-\mu }{2}\right\} =N-2=1$ by $0<\mu <N$. It is easy to show that

$$\begin{aligned} \varepsilon ^{2}\int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{2}}{1+r^{2}}\text {d}r=\varepsilon \left( \delta -\varepsilon \arctan \left( \frac{\delta }{\varepsilon }\right) \right) . \end{aligned}$$

Therefore,

$$\begin{aligned} I_{\varepsilon }=C_{4}\left( \delta -\varepsilon \arctan \left( \frac{\delta }{\varepsilon }\right) \right) -C_{3} \varepsilon ^{5-\mu -p}. \end{aligned}$$

Our claim follows.

The Case N=4. In this case, $\frac{6-\mu }{2}<p<2_{\mu }^{*}$ implies $6-\mu -2p<0$ and $\min \left\{ N-2,\frac{2N-\mu }{2}\right\} =N-2=2$ if $0<\mu <4$. We also have

$$\begin{aligned} \varepsilon ^{2} \int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{3}}{\left( 1+r^{2}\right) ^{2}} \mathrm {~d} r=\frac{\varepsilon ^{2}}{2}\left[ \ln \left( 1+\frac{\delta ^{2}}{\varepsilon ^{2}}\right) +\frac{\varepsilon ^{2}}{\varepsilon ^{2}+\delta ^{2}}-1\right] . \end{aligned}$$

So,

$$\begin{aligned} I_{\varepsilon }=\frac{C_{4}}{2}\left( \ln \left( 1+\frac{\delta ^{2}}{\varepsilon ^{2}}\right) +\frac{\varepsilon ^{2}}{\varepsilon ^{2}+\delta ^{2}}-1\right) -C_{3} \varepsilon ^{6-\mu -2p}. \end{aligned}$$

Our claim follows.

The Case $N\ge 5$ . We have

$$\begin{aligned} I_{\varepsilon } =\varepsilon ^{2-\min \left\{ N-2,\frac{2 N-\mu }{2}\right\} } \left( C_{4}\int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{N-1}}{\left( 1+r^{2}\right) ^{N-2}} \mathrm {~d} r-C_{3} \varepsilon ^{2 N-\mu -(N-2)p-2}\right) . \end{aligned}$$

It is easy to show that if $N \ge 5$, then the integral

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0} \int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{N-1}}{\left( 1+r^{2}\right) ^{N-2}} \mathrm {~d} r \end{aligned}$$

converges.

There are two cases left to be considered:

• $0<\mu <4$ and $N \ge 5$;

• $\mu \ge 4$ and $N \ge 5$.

Assume that $0<\mu <4$ and $N\ge 5$, we have

$$\begin{aligned} 2-\eta =2-\min \left\{ N-2,\frac{2 N-\mu }{2}\right\} =-N+4<0. \end{aligned}$$

Also $\frac{2N-\mu -2}{N-2}<p<\frac{2N-\mu }{N-2}$ implies $2N-\mu -(N-2)p-2<0.$ Therefore, $I_{\varepsilon } \rightarrow -\infty$ as $\varepsilon \rightarrow 0$. Now we consider the case $\mu \ge 4$ and $N\ge 5$. We have $N-2\ge \frac{2N-\mu }{2}$ and therefore

$$\begin{aligned} 2-\eta =2-\min \left\{ N-2,\frac{2 N-\mu }{2}\right\} =2-N+\frac{\mu }{2}<0. \end{aligned}$$

Since

$$\begin{aligned} I_{\varepsilon }=\varepsilon ^{2-N+\frac{\mu }{2}}\left[ C_{4} \int \limits _{0}^{\frac{\delta }{\varepsilon }} \frac{r^{N-1}}{\left( 1+r^{2}\right) ^{N-2}}\text {d}r-C_{3} \varepsilon ^{2 N-\mu -(N-2)p-2}\right] , \end{aligned}$$

we conclude that $I_{\varepsilon } \rightarrow -\infty$ as $\varepsilon \rightarrow 0$. We are done.

Now we prove (4.9).

For $\delta >0$ sufficiently large, we have $U_{\varepsilon }^{2}(x) \le \varepsilon ^{1+\eta }$ if $|x| \ge \delta$. Since

$$\begin{aligned} \begin{aligned} \frac{1}{\varepsilon ^{\eta }}\left[ C_{2}\int \limits _{B_{2\delta } \backslash B_{\delta }}|u_{\varepsilon }(x)|^{2}\text {d}x -C_{3}\varepsilon ^{2N-\mu -(N-2)p}\right]&<\frac{C_{2}}{\varepsilon ^{\eta }} \int \limits _{B_{2 \delta } \backslash B_{\delta }} \psi ^{2}(x) U_{\varepsilon }^{2}(x) \textrm{d} x \le C_{2} \varepsilon \Vert \psi \Vert _{2} \le C_{1} \varepsilon \Vert \psi \Vert _{\varepsilon }, \end{aligned} \end{aligned}$$

our proof is complete.

Case 2. For $\lambda$ sufficiently large, $\frac{2 N-\mu }{N}<p\le \frac{N+2-\mu }{N-2}$ and $N=3,4$ or $\frac{2 N-\mu }{N}<p\le \frac{2N-2-\mu }{N-2}$ and $N\ge 5$.

From lemma 4.1, we know that $\max _{t\ge 0} I_{0}(tu_{\varepsilon })$ is attained at some $t_{\lambda }>0$. Since $I'_{0}(t_{\lambda }u_{\varepsilon })=0$, we have

$$\begin{aligned} \begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\left( |\nabla u_{\varepsilon }|^{2} +V_{0}|u_{\varepsilon }|^{2}\right) \text {d}x&=\frac{1}{2_{\mu }^{*}t_{\lambda }^{2}}\int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{(|u_{\varepsilon }(x)|^{2_{\mu }^{*}}+F(|u_{\varepsilon }(x)|^{2}))(|u_{\varepsilon }(y)|^{2_{\mu }^{*}}+F(|u_{\varepsilon }(y)|^{2}))}{|x-y|^{\mu }}\text {d}x\text {d}y\\&\ge \frac{1}{2_{\mu }^{*}} \lambda ^{2}t_{\lambda }^{2p-2} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}}\frac{|u_{\varepsilon }(x)|^{p}|u_{\varepsilon }(y)|^{p}}{|x-y|^{\mu }}\text {d}x\text {d}y. \end{aligned} \end{aligned}$$

Thus, $t_{\lambda }\rightarrow 0$ as $\lambda \rightarrow +\infty$ and

$$\begin{aligned} \begin{aligned} \max _{t\ge 0}I_{0}\left( tu_{\varepsilon }\right)&=\frac{t_{\lambda }^{2}}{2} \int \limits _{{\mathbb {R}}^{N}}\left( |\nabla u_{\varepsilon }|^{2} +V_{0}|u_{\varepsilon }|^{2}\right) \text {d}x -\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |t_{\lambda }u_{\varepsilon }|^{2})\right) G(\varepsilon x, |t_{\lambda }u_{\varepsilon }|^{2})\text {d}x\\&<\frac{t_{\lambda }^{2}}{2} \int \limits _{{\mathbb {R}}^{N}}\left( |\nabla u_{\varepsilon }|^{2} +V_{0}|u_{\varepsilon }|^{2}\right) \text {d}x. \end{aligned} \end{aligned}$$

Since $t_{\lambda } \rightarrow 0$ as $\lambda \rightarrow +\infty$ and $\frac{N+2-\mu }{2(N-\mu )}\left( S_{H,L}\right) ^{\frac{2 N-\mu }{N+2-\mu }}>0$, we conclude that

$$\begin{aligned} \frac{t_{\lambda }^{2}}{2}\int \limits _{{\mathbb {R}}^{N}}\left( |\nabla u_{\varepsilon }|^{2} +V_{0}|u_{\varepsilon }|^{2}\right) \text {d}x <\frac{N+2-\mu }{2(2 N-\mu )}\left( S_{H,L}\right) ^{\frac{2N-\mu }{N+2-\mu }} \end{aligned}$$

for $\lambda >0$ sufficiently large.

Therefore,

$$\begin{aligned} \sup _{t\ge 0}I_{0}\left( t u_{\varepsilon }\right) <\frac{N+2-\mu }{2(2 N-\mu )}\left( S_{H,L}\right) ^{\frac{2N-\mu }{N+2-\mu }} \end{aligned}$$

for $\lambda >0$ sufficiently large. $\square$

Theorem 4.1

Assume that $(f_{1})$–$(f_{3})$ hold. Then, autonomous problem (4.1) has a positive ground state solution u with $I_{0}(u)=c_{V_{0}}$.

Proof

By Lemma 3.4 with $V(x)=V_{0}$ and the mountain pass theorem without (PS) condition, there exists a $(PS)_{c_{V_{0}}}$ sequence $\{u_{n}\}\subset H_{0}$ of $I_{0}$ with

$$\begin{aligned} c_{V_{0}}<\frac{N-\mu +2}{2(2 N-\mu )}\left( S_{H,L}\right) ^{\frac{2N-\mu }{N+2-\mu }}. \end{aligned}$$

If $u_{0}\in {\mathcal {N}}_{0}$ satisfies $I_{0}\left( u_{0}\right) =c_{V_{0}}$, then $m^{-1}(u_{0})\in S_{0}$ is a minimizer of $\Psi _{0}$, so that $u_{0}$ is a critical point of $I_{0}$ by Lemma 4.2. Now, we show that there exists a minimizer $u \in {\mathcal {N}}_{0}$ of $\left. I_{0}\right| _{{\mathcal {N}}_{0}}.$ Since $\inf _{S_{0}} \Psi _{0}=\inf _{{\mathcal {N}}_{0}} I_{0}=c_{V_{0}}$ and $S_{0}$ is a $C^{1}$ manifold, by Ekeland’s variational principle, there exists a sequence $\omega _{n} \subset S_{0}$ with $\Psi _{0}\left( \omega _{n}\right) \rightarrow c_{V_{0}}$ and $\Psi _{0}^{\prime }\left( \omega _{n}\right) \rightarrow 0$ as $n \rightarrow \infty$. Put $u_{n}=m\left( \omega _{n}\right) \in {\mathcal {N}}_{0}$ for $n \in {\mathbb {N}}$. Then by Lemma 4.2$(b_{3})$, we have $I_{0}\left( u_{n}\right) \rightarrow c_{V_{0}}$ and $I_{0}^{\prime }\left( u_{n}\right) \rightarrow 0$ as $n \rightarrow \infty$. Similar to the proof of Lemma 3.7, it is easy to check that $\{u_{n}\}$ is bounded in $H_{0}$. Thus, we have $u_{n} \rightharpoonup u$ in $H_{0}$, $u_{n} \rightarrow u$ in $L_{loc}^{r}({\mathbb {R}}^{N}), 1 \le r<2^{*}$ and $u_{n} \rightarrow u$ a.e. in ${\mathbb {R}}^{N}$. Similar to the proof of Lemma 3.1, we have $I_{0}^{\prime }(u)=0$. From Lemma 4.3, we know that $u \ne 0$. Moreover,

$$\begin{aligned} \begin{aligned} c_{V_{0}}&\le I_{0}(u) =I_{0}(u)-\frac{1}{2}\left\langle I_{0}^{\prime }(u), u\right\rangle \\&=\frac{1}{2\cdot 2_{\mu }^{*}} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1 }{|x|^{\mu }}*G(\varepsilon x,|u|^{2})\right) \left( g(\varepsilon x,|u|^{2})-G(\varepsilon x,|u|^{2})\right) \text {d}x\\&\le \frac{1}{2\cdot 2_{\mu }^{*}} \int \limits _{{\mathbb {R}}^{N}}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) \left( g(\varepsilon x,|u_{n}|^{2})-G(\varepsilon x,|u_{n}|^{2})\right) \text {d}x\\&=\liminf _{n \rightarrow \infty } \left( I_{0}(u_{n})-\frac{1}{2}\langle I_{0}^{\prime }(u_{n}), u_{n}\rangle \right) \\&=c_{V_{0}}, \end{aligned} \end{aligned}$$

where we used Fatou’s lemma. Therefore, $I_{0}(u)=c_{V_{0}}$, which means that u is a ground state solution for (4.1). From the assumption of f, by [11, Propositions 6 and 7], we know that $u(x)>0$ for $x\in {\mathbb {R}}^{N}$. The proof is complete. $\square$

The next result is a compactness result on autonomous problem which we will use later.

Lemma 4.5

Let $(u_{n})\subset {\mathcal {N}}_{0}$ be a sequence such that $I_{0}\left( u_{n}\right) \rightarrow c_{V_{0}}$. Then, $\left\{ u_{n}\right\}$ has a convergent subsequence in $H_{0}$.

Proof

Since $\left\{ u_{n}\right\} \subset {\mathcal {N}}_{0}$, it follows from Lemma 4.1$(a_{3})$, Lemma 4.2$(b_{4})$ and the definition of $c_{V_{0}}$ that

$$\begin{aligned} v_{n}=m^{-1}\left( u_{n}\right) =\frac{u_{n}}{\left\| u_{n}\right\| _{V_{0}}} \in S_{0}^{+}, \quad \forall n \in {\mathbb {N}} \end{aligned}$$

and

$$\begin{aligned} \Psi _{0}\left( v_{n}\right) =I_{0}\left( u_{n}\right) \rightarrow c_{V_{0}}=\inf _{S_{0}^{+}}\Psi _{0}(u). \end{aligned}$$

Although $S_{0}^{+}$ is not a complete $C^{1}$ manifold, we still can apply Ekeland’s variational principle to the functional ${\mathcal {E}}_{0}: H \rightarrow {\mathbb {R}}\cup \{\infty \}$ defined by

$$\begin{aligned} {\mathcal {E}}_{0}(u):={\widehat{\Psi }}_{0}(u) \quad \text{ if } u \in S_{0}^{+} \end{aligned}$$

and

$$\begin{aligned} {\mathcal {E}}_{0}(u):=\infty \quad \text{ if } u \in \partial S_{0}^{+}, \end{aligned}$$

where $H=\overline{S_{0}^{+}}$ is the complete metric space equipped with the metric

$$\begin{aligned} d(u, v):=\Vert u-v\Vert _{0}. \end{aligned}$$

In fact, by Lemma 4.1$(a_{4})$, ${\mathcal {E}}_{0} \in C(H,{\mathbb {R}}\cup \{\infty \})$, and from Lemma 4.2$(b_{4})$, ${\mathcal {E}}_{0}$ is bounded from below. Therefore, there exists a sequence $\left\{ {\tilde{v}}_{n}\right\} \subset S_{0}^{+}$ such that $\left\{ {\tilde{v}}_{n}\right\}$ is a $(P S)_{c_{V_{0}}}$ sequence for $\Psi _{0}$ on $S_{0}^{+}$ and

$$\begin{aligned} \left\| {\tilde{v}}_{n}-v_{n}\right\| _{0}=o_{n}(1). \end{aligned}$$

Arguing as in Lemma 3.8, we obtain the conclusion of this lemma. $\square$

5 Solutions for the penalized problem

This subsection is devoted to proving a multiplicity result for the modified problem (3.1) by applying the Ljusternik–Schnirelmann category theory.

Let $\delta >0$ be such that $M_{\delta }\subset \Lambda$, $\omega \in H^{1}({\mathbb {R}}^{N},{\mathbb {R}})$ be a positive ground state solution of the limit problem (4.1), and $\eta \in C^{\infty }({\mathbb {R}}^{+},[0,1])$ such that $\eta =1$ if $0\le t\le \frac{\delta }{2}$ and $\eta =0$ if $t\ge \delta$. We define

$$\begin{aligned} \Psi _{\varepsilon ,y}(x):=\eta (|\varepsilon x-y|) \omega \left( \frac{\varepsilon x-y}{\varepsilon }\right) \exp \left( i \tau _{y}\left( \frac{\varepsilon x-y}{\varepsilon }\right) \right) , \end{aligned}$$

for each $y\in M$, where $\tau _{y}:=\sum \limits _{1}^{N}A_{i}(y)x_{i}$. Let $t_{\varepsilon }>0$ be the unique positive number such that

$$\begin{aligned} \max _{t\ge 0}J_{\varepsilon }(t\Psi _{\varepsilon , y})=J_{\varepsilon }(t_{\varepsilon }\Psi _{\varepsilon ,y}). \end{aligned}$$

By noticing that $t_{\varepsilon }\Psi _{\varepsilon ,y}\in {\mathcal {N}}_{\varepsilon }$, we consider the function $\Phi _{\varepsilon }:=M \rightarrow {\mathcal {N}}_{\varepsilon }$ defined by setting

$$\begin{aligned} \Phi _{\varepsilon }:=t_{\varepsilon }\Psi _{\varepsilon ,y}. \end{aligned}$$

By construction, $\Phi _{\varepsilon }(y)$ has compact support for any $y\in M$.

Lemma 5.1

The limit

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0^{+}} J_{\varepsilon }(\Phi _{\varepsilon }(y))=c_{V_{0}}. \end{aligned}$$

holds uniformly in $y\in M$.

Proof

Arguing by contradiction, we deduce that there exist $\delta _{0}>0,\{y_{n}\}\subset M$ and $\varepsilon _{n} \rightarrow 0^{+}$ satisfying

$$\begin{aligned} \left| J_{\varepsilon _{n}}\left( \Phi _{\varepsilon _{n}}(y)\right) -c_{V_{0}}\right| \ge \delta _{0}. \end{aligned}$$

(5.1)

For simplicity, we write $\Phi _{n}, \Psi _{n}$ and $t_{n}$ for $\Phi _{\varepsilon _{n}}, \Psi _{\varepsilon _{n}, y_{n}}$ and $t_{\varepsilon _{n}}$, respectively.

Arguing as in [13, Lemma 3.2], we see that

$$\begin{aligned} \begin{aligned} \Vert \Psi _{n}\Vert _{\varepsilon _{n}}^{2}\rightarrow \Vert u\Vert _{V_{0}}\text { as } n\rightarrow \infty . \end{aligned} \end{aligned}$$

(5.2)

On the other hand, since $\left\langle J_{\varepsilon _{n}}^{\prime }\left( t_{n} \Psi _{n}\right) , t_{n} \Psi _{n}\right\rangle =0$, by the change of variables $z=(\varepsilon _{n} x-y_{n})/\varepsilon _{n}$, we obtain

$$\begin{aligned} \begin{aligned} \Vert \Psi _{n}\Vert _{\varepsilon _{n}}^{2}&=\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}x,|t_{n} \Psi _{n}|^{2})\right) g(\varepsilon _{n}x,|t_{n} \Psi _{n}|^{2})|\Psi _{n}|^{2}\text {d}x\\&=\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}z+y_{n},|t_{n} \Psi _{n}|^{2})\right) g(\varepsilon _{n}z+y_{n},t_{n}^{2} \eta ^{2}(|\varepsilon _{n} z|)\omega ^{2}(z)\eta ^{2}(|\varepsilon _{n} z|)\omega ^{2}(z)\text {d}z. \end{aligned} \end{aligned}$$

(5.3)

If $z\in B_{\delta /\varepsilon _{n}}(0)$, then $\varepsilon _{n}z+y_{n}\in B_{\delta }(y_{n})\subset M_{\delta }\subset \Lambda$. Hence, $g(x,s)=f(s)$ for any $x\in \Lambda$ and $\Psi (s)=0$ for $s\ge \delta$, the last equality leads to

$$\begin{aligned} \begin{aligned} \Vert \Psi _{n}\Vert _{\varepsilon _{n}}^{2}&\ge \frac{F(|t_{n}\alpha |^{2} ) f(|t_{n} \alpha |^{2})}{2_{\mu }^{*}\cdot |\alpha |^{2}} \int \limits _{B_{\delta /2}(0)}\int \limits _{B_{\delta /2}(0)} \frac{|\omega (y)|^{2}|\omega (z)|^{2}}{|y-z|^{\mu }}\text {d}x\text {d}y, \end{aligned} \end{aligned}$$

where $\alpha =\min \{\omega (z):|z|\le \delta /2\}$.

If $t_{n}\rightarrow \infty$, by $(f_{3})$ we deduce that $\left\| \Psi _{n}\right\| _{\varepsilon _{n}}^{2} \rightarrow \infty$ which contradicts (5.2). Therefore, up to a subsequence, we may assume that $t_{n} \rightarrow t_{0}\ge 0$.

Since g has critical growth and $t_{n} \Psi _{n}\in {\mathcal {N}}_{\varepsilon }$, it follows that $t_{0}>0$. Thereby, taking the limit in (5.3), we obtain

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}(|\nabla \omega |^{2}+V_{0}\omega ^{2})\text {d}x =\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}x,|t_{0} \omega |^{2})\right) g(\varepsilon _{n}x,|t_{0} \omega |^{2})| \omega |^{2}\text {d}x, \end{aligned}$$

which implies that $t_{0} \omega \in {\mathcal {N}}_{V_{0}}$. Since $\omega \in {\mathcal {N}}_{V_{0}}$, we obtain $t_{0}=1$. Using the Lebesgue dominated convergence theorem, we get

$$\begin{aligned} \int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}x,|t_{n} \Psi _{n}|^{2})\right) G(\varepsilon _{n}y,|t_{n} \Psi _{n}|^{2})\text {d}x =\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}x,|\omega |^{2})\right) G(\varepsilon _{n}y,|\omega |^{2})\text {d}x. \end{aligned}$$

Hence,

$$\begin{aligned} \lim _{n\rightarrow \infty } J_{\varepsilon _{n}}(\Phi _{n}(y_{n}))=I_{0}(\omega )=c_{V_{0}}, \end{aligned}$$

which is a contradiction with (5.1) and the proof is complete. $\square$

Now we define the barycenter map.

Let $\rho >0$ be such that $M_{\delta } \subset B_{\rho }$ and $\Upsilon : {\mathbb {R}}^{N} \rightarrow$ ${\mathbb {R}}^{N}$ be defined by

$$\begin{aligned} \Upsilon (x):={\left\{ \begin{array}{ll} x& \text{ if } |x|<\rho ,\\ \rho x /|x| & \text{ if } |x| \ge \rho .\end{array}\right. } \end{aligned}$$

The barycenter map $\beta _{\varepsilon }: {\mathcal {N}}_{\varepsilon }\rightarrow {\mathbb {R}}^{N}$ is defined by

$$\begin{aligned} \beta _{\varepsilon }(u):=\frac{1}{\Vert u\Vert _{2}^{2}} \int \limits _{{\mathbb {R}}^{N}} \Upsilon (\varepsilon x)|u|^{2}\text {d}x. \end{aligned}$$

Arguing as Lemma 4.3 in [8], it is easy to see that the function $\beta _{\varepsilon }$ verifies the following limit:

Lemma 5.2

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0^{+}} \beta _{\varepsilon }\left( \Phi _{\varepsilon }(y)\right) =y \end{aligned}$$

holds uniformly in $y\in M$.

We are next to establish the following useful compactness result.

Lemma 5.3

Let $\varepsilon _{n} \rightarrow 0^{+}$and $\left( u_{n}\right) \subset {\mathcal {N}}_{\varepsilon _{n}}$ be such that $J_{\varepsilon _{n}}\left( u_{n}\right) \rightarrow c_{V_{0}}$. Then, there exists a sequence $\{{\tilde{y}}_{n}\}\subset {\mathbb {R}}^{N}$ such that $v_{n}:=\left| u_{n}\right| \left( \cdot +{\tilde{y}}_{n}\right)$ has a convergent subsequence in $H^{1}({\mathbb {R}}^{N}, {\mathbb {R}})$. Moreover, up to a subsequence, $y_{n}:=\varepsilon _{n}{\tilde{y}}_{n} \rightarrow y\in M$ as $n\rightarrow \infty$.

Proof

Since $\langle J_{\varepsilon _{n}}^{\prime }(u_{n}),u_{n}\rangle =0$ and $J_{\varepsilon _{n}}\left( u_{n}\right) \rightarrow c_{V_{0}}$, arguing as in the proof of Lemma 3.4, we can prove that there exists $C>0$ such that $\left\| u_{n}\right\| _{\varepsilon _{n}} \le C$ for all $n \in {\mathbb {N}}$. We claim that there exist a sequence $\left( {\tilde{y}}_{n}\right) \subset {\mathbb {R}}^{N}$ and constants $R,\beta >0$ such that

$$\begin{aligned} \liminf _{n \rightarrow \infty } \int \limits _{B_{R}\left( {\tilde{y}}_{n}\right) }\left| u_{n}\right| ^{2} \geqslant \beta . \end{aligned}$$

(5.4)

Thereby, for some subsequence,

$$\begin{aligned} v_{n}:=\left| u_{n}\right| \left( \cdot +{\tilde{y}}_{n}\right) \rightarrow v \ne 0 \hspace{2mm}\text {weakly in}\hspace{2mm} H^{1}\left( {\mathbb {R}}^{N},{\mathbb {R}}\right) . \end{aligned}$$

Let $t_{n}>0$ be such that ${\tilde{v}}_{n}:=t_{n}v_{n}\in {\mathcal {N}}_{V_{0}}$. By the diamagnetic inequality (2.1), we have

$$\begin{aligned} c_{V_{0}} \le J_{0}\left( {\tilde{v}}_{n}\right) \le \max _{t \ge 0} J_{\varepsilon _{n}}\left( t v_{n}\right) =J_{\varepsilon _{n}}\left( u_{n}\right) =c_{V_{0}}+o_{n}(1). \end{aligned}$$

Hence, $J_{0}\left( {\tilde{v}}_{n}\right) \rightarrow c_{V_{0}}$ as $n \rightarrow +\infty$.

Since the sequences $\left\{ v_{n}\right\}$ and $\left\{ {\tilde{v}}_{n}\right\}$ are bounded in $H^{1}\left( {\mathbb {R}}^{N}, {\mathbb {R}}\right)$ and $v_{n}\nrightarrow 0$ in $H^{1}\left( {\mathbb {R}}^{N}, {\mathbb {R}}\right)$, we deduce $\left( t_{n}\right)$ is also bounded and up to a subsequence, we may assume that $t_{n} \rightarrow t_{0}\ge 0$.

If $t_{0}=0$, in view of the boundedness of $v_{n}$ in $H_{0}$, we have ${\tilde{v}}_{n}:=t_{n}v_{n}\rightarrow 0$ in $H_{0}$. Hence, $I_{0}({\tilde{v}}_{n})\rightarrow 0$, which contradicts $c_{V_{0}}>0$. Thus, up to a subsequence, we may assume that ${\tilde{v}}_{n} \rightharpoonup {\tilde{v}}:=t_{0} v \ne 0$ in $H_{0}$, and, by Lemma 4.5, we can deduce that ${\tilde{v}}_{n} \rightarrow {\tilde{v}}$ in $H_{0}$, which gives $v_{n}\rightarrow v$ in $H_{0}$.

Now we show the final part, namely that $\left\{ y_{n}\right\}$ has a subsequence such hat $y_{n} \rightarrow y \in M$. Assume by contradiction that $\left\{ y_{n}\right\}$ is not bounded and so, up to a subsequence, $\left| y_{n}\right| \rightarrow +\infty$ as $n \rightarrow +\infty$. Choose $R>0$ such that $\Lambda \subset B_{R}(0)$. Then for n large enough, we have $\left| y_{n}\right| >2 R$, and for any $x \in B_{R / \varepsilon _{n}}(0)$,

$$\begin{aligned} \left| \varepsilon _{n} x+y_{n}\right| \ge \left| y_{n}\right| -\varepsilon _{n}|x|>R. \end{aligned}$$

Since $u_{n} \in {\mathcal {N}}_{\varepsilon _{n}}$, using $(V_{1})$ and the diamagnetic inequality (2.2), we obtain

$$\begin{aligned} \begin{aligned} \int \limits _{{\mathbb {R}}^{N}}&\left( |\nabla v_{n}|^{2}+V_{0}|v_{n}|^{2}\right) \text {d}x\\ \le&\int \limits _{{\mathbb {R}}^{N}}\left( |\nabla _{\varepsilon } v_{n}|^{2}+V(\varepsilon _{n} x+y_{n})|v_{n}|^{2}\right) \text {d}x\\ =&\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1 }{|x|^{\mu }}*G(\varepsilon _{n} x+y_{n}, |v_{n}|^{2})\right) g(\varepsilon _{n} x+y_{n},\left| v_{n}\right| ^{2}) \left| v_{n}\right| ^{2}\text {d}x\\ \le&\frac{1}{2_{\mu }^{*}}\int \limits _{B_{R/\varepsilon _{n}}(0)} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon _{n} x+y_{n}, |v_{n}|^{2})\right) {\tilde{f}}(|v_{n}|^{2})|v_{n}|^{2}\text {d}x\\&+\frac{1}{2_{\mu }^{*}}\int \limits _{B_{R/\varepsilon _{n}}^{c}(0)} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon _{n} x+y_{n}, |v_{n}|^{2})\right) f(|v_{n}|^{2})|v_{n}|^{2}\text {d}x. \end{aligned} \end{aligned}$$

(5.5)

Since $v_{n}\rightarrow v$ in $H_{0}$, $v_{n}\in {\mathcal {B}}$ and ${\tilde{f}}(t)\le V_{0}/K$, we obtain

$$\begin{aligned} \min \left\{ 1, \frac{V_{0}}{2\cdot 2_{\mu }^{*}}\right\} \int \limits _{{\mathbb {R}}^{N}}\left( |\nabla v_{n}|^{2} +|v_{n}|^{2}\right) \text {d}x=o_{n}(1), \end{aligned}$$

that is to say $v_{n}\rightarrow 0$ in $H_{0}$, which contradicts $v\ne 0$.

Therefore, we may assume that $y_{n} \rightarrow y_{0} \in {\mathbb {R}}^{N}$. Assume by contradiction that $y_{0}\notin {\bar{\Lambda }}$. Then, there exists $r>0$ such that for every n large enough, we have that $|y_{n}-y_{0}|<r$ and $B_{2r}(y_{0})\subset {\bar{\Lambda }}^{c}$. Then, if $x\in B_{r /\varepsilon _{n}}(0)$, we have that $|\varepsilon _{n}x+y_{n}-y_{0}|<2r$ so that $\varepsilon _{n}x+y_{n}\in {\bar{\Lambda }}^{c}$, and so, arguing as before, we achieve a contradiction. Thus, $y_{0} \in {\bar{\Lambda }}$.

To prove that $V\left( y_{0}\right) =V_{0}$, we suppose by contradiction that $V\left( y_{0}\right) >V_{0}$. Using Fatou’s lemma, the change of variable $z=x+{\tilde{y}}_{n}$ and $\max _{t \ge 0} J_{\varepsilon _{n}}\left( t u_{n}\right) =J_{\varepsilon _{n}}\left( u_{n}\right)$, we obtain

$$\begin{aligned} \begin{aligned} \displaystyle c_{V_{0}}&=I_{0}({\tilde{v}}) <\frac{1}{2} \int \limits _{{\mathbb {R}}^{N}}\left( |\nabla {\tilde{v}}|^{2} +V(y_{0})|{\tilde{v}}|^{2}\right) \text {d}x -\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1 }{|x|^{\mu }}*G(\varepsilon x, |{\tilde{v}}|^{2})\right) G(\varepsilon x, |{\tilde{v}}|^{2})\text {d}x\\&\le \liminf _{n}\bigg (\frac{1}{2} \int \limits _{{\mathbb {R}}^{N}}(|\nabla {\tilde{v}}_{n}|^{2} +V(\varepsilon _{n}x+y_{n})|{\tilde{v}}_{n}|^{2})\text {d}x\\&\hspace{1.6cm}-\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}x,|{\tilde{v}}_{n}|^{2})\right) G(\varepsilon _{n}x,|{\tilde{v}}_{n}|^{2})\text {d}x\bigg )\\&=\lim \inf _{n}\bigg (\frac{t_{n}^{2}}{2}\int \limits _{{\mathbb {R}}^{N}} (|\nabla |u_{n}||^{2}+V(\varepsilon _{n} z)|u_{n}|^{2})\text {d}x\\&\hspace{1.6cm}-\frac{1}{2\cdot 2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}} \left( \frac{1}{|x|^{\mu }}*G(\varepsilon _{n}x,|t_{n}u_{n}|^{2})\right) G(\varepsilon _{n}x,|t_{n}u_{n}|^{2})\text {d}x\bigg ) \\&\le \lim \inf _{n}J_{\varepsilon _{n}}(t_{n}u_{n}) \le \liminf _{n} J_{\varepsilon _{n}}(u_{n})=c_{V_{0}}, \end{aligned} \end{aligned}$$

which is impossible and the proof is complete. $\square$

Let $h:{\mathbb {R}}^{+}\rightarrow {\mathbb {R}}^{+}$ be any positive function satisfying $h(\varepsilon )\rightarrow 0$ as $\varepsilon \rightarrow 0^{+}$. Consider the following subset of the Nehari manifold

$$\begin{aligned} \tilde{{\mathcal {N}}}_{\varepsilon }=\{u\in {\mathcal {N}}_{\varepsilon }: J_{\varepsilon }(u)\le c_{V_{0}}+h(\varepsilon )\}. \end{aligned}$$

Fixed $y\in M$, we conclude from Lemma 5.1 that $|J_{\varepsilon }\left( \Phi _{\varepsilon }(y)\right) -c_{V_{0}}|\rightarrow 0$ as $\varepsilon \rightarrow 0^{+}$. Thus, $\Phi _{\varepsilon }(y)\in \widetilde{{\mathcal {N}}}_{\varepsilon }$ and $\tilde{{\mathcal {N}}}_{\varepsilon }\ne \emptyset$ for $\varepsilon >0$ small enough.

Arguing as Lemma 4.5 in [8], the next statement is to involve in the relationship between $\widetilde{{\mathcal {N}}}_{\varepsilon }$ and the barycenter map.

Lemma 5.4

For any $\delta >0$, we have

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0^{+}} \sup _{u \in \tilde{{\mathcal {N}}}_{\varepsilon }} {\text {dist}}\left( \beta _{\varepsilon }(u), M_{\delta }\right) =0. \end{aligned}$$

Next we prove our multiplicity result by presenting a relation between the topology of M and the number of solutions of the modified problem (3.1).

Theorem 1.2

For any $\delta >0$ such that $M_{\delta } \subset \Lambda$, there exists ${\tilde{\varepsilon }}_{\delta }>0$ such that, for any $\varepsilon \in (0, {\tilde{\varepsilon }}_{\delta })$, problem (3.1) has at least ${\text {cat}}_{M_{\delta }} M$ nontrivial solutions.

Proof

For any given $\delta >0$ such that $M_\delta \subset \Lambda$, by Lemmas 5.1, 5.2, and 5.3, we employ an argument as in [15] to deduce the existence of ${\tilde{\varepsilon }}_\delta >0$ such that, for any $\varepsilon \in \left( 0, \varepsilon _\delta \right)$, the following diagram

$$\begin{aligned} M \xrightarrow {\Phi _{\varepsilon }} \tilde{{\mathcal {N}}}_{\varepsilon } \xrightarrow {\beta _{\varepsilon }} M_\delta \end{aligned}$$

is well defined and $\beta _{\varepsilon } \circ \Phi _{\varepsilon }$ is homotopically equivalent to the embedding $\iota : M \rightarrow M_\delta$. This fact implies that ${\text {cat}}_{\widetilde{{\mathcal {N}}}_{\varepsilon }}(\widetilde{{\mathcal {N}}}_{\varepsilon }) \ge {\text {cat}}_{M_\delta }(M)$ due to [15, Lemma 2.2]. It follows from Lemma 3.9 and standard Ljusternik–Schnirelmann theory that $J_{\varepsilon }$ possesses at least cat $_{\tilde{{\mathcal {N}}}_{\varepsilon }}(\tilde{{\mathcal {N}}}_{\varepsilon })$ critical points on ${\mathcal {N}}_{\varepsilon }$. Consequently, equation (3.1) possess at least ${\text {cat}}_{M_\delta }(M)$ critical points. $\square$

6 Proof of Theorem (1.2)

In this section, we prove our main result. The idea is to show that the solutions $u_{\varepsilon }$ obtained in Theorem 5.5 satisfy

$$\begin{aligned} \left| u_{\varepsilon }(x)\right| \le a \hspace{3mm}\text { for } x \in \Lambda _{\varepsilon }^{c} \end{aligned}$$

for $\varepsilon$ small enough. Arguing as in [4, Lemma 4.1], we have the following important result.

Lemma 6.1

Let $\varepsilon _{n} \rightarrow 0^{+}$and $u_{n} \in \widetilde{{\mathcal {N}}}_{\varepsilon _{n}}$ be a solution of (3.1). Then, $J_{\varepsilon _{n}}\left( u_{n}\right) \rightarrow c_{V_{0}}$ and $|u_{n}|\in L^{\infty }\left( {\mathbb {R}}^{N}\right)$. Moreover, for any given $\gamma >0$, there exists $R>0$ and $n_{0}\in {\mathbb {N}}$ such that

$$\begin{aligned} \left\| u_{n}\right\| _{L^{\infty }\left( B_{R}\left( {\tilde{y}}_{n}\right) ^{c}\right) }<\gamma \quad \text { for all } n \ge n_{0}, \end{aligned}$$

(6.1)

where ${\tilde{y}}_{n}$ is given by Lemma 5.3.

Now, we are ready to give a proof of Theorem 1.2.

Proof of Theorem 1.2

Let $\delta >0$ be such that $M_{\delta } \subset \Lambda$. We first claim that there exists ${\tilde{\varepsilon }}_{\delta }>0$ such that for any $0<\varepsilon <{\tilde{\varepsilon }}_{\delta }$ and any solution $u \in \widetilde{{\mathcal {N}}}_{\varepsilon }$ of the problem (3.1), it holds

$$\begin{aligned} \Vert u\Vert _{L^{\infty }\left( {\mathbb {R}}^{N} \backslash \Lambda _{\varepsilon }\right) }<a. \end{aligned}$$

(6.2)

To prove the above claim, we argue by contradiction. Assume that there are two sequences $\varepsilon _{n} \rightarrow 0^{+}$and $u_{n} \in \widetilde{{\mathcal {N}}}_{\varepsilon _{n}}$ verifying $J_{\varepsilon _{n}}^{\prime }\left( u_{n}\right) =0$ and

$$\begin{aligned} \left\| u_{n}\right\| _{L^{\infty }\left( {\mathbb {R}}^{N} \backslash \Lambda _{\varepsilon _{n}}\right) }\ge a. \end{aligned}$$

(6.3)

As in Lemma 5.1, we have that $J_{\varepsilon _{n}}(u_{n}) \rightarrow c_{V_{0}}$. Therefore, by Lemma 5.3, there exists obtain a sequence $\{{\tilde{y}}_{n}\}\subset {\mathbb {R}}^{N}$ such that $\varepsilon _{n} {\tilde{y}}_{n} \rightarrow y_{0} \in M$. If we take $r>0$ such that $B_{r}\left( y_{0}\right) \subset B_{2 r}\left( y_{0}\right) \subset \Lambda$ we have that

$$\begin{aligned} B_{r / \varepsilon _{n}}\left( y_{0} / \varepsilon _{n}\right) =\frac{1}{\varepsilon _{n}} B_{r}\left( y_{0}\right) \subset \Lambda _{\varepsilon _{n}}. \end{aligned}$$

Moreover, for any $z \in B_{r / \varepsilon _{n}}\left( {\tilde{y}}_{n}\right)$, it holds

$$\begin{aligned} \left| z-\frac{y_{0}}{\varepsilon _{n}}\right| \leqslant \left| z-{\tilde{y}}_{n}\right| +\left| {\tilde{y}}_{n}-\frac{y_{0}}{\varepsilon _{n}}\right|<\frac{1}{\varepsilon _{n}}\left( r+o_{n}(1)\right) <\frac{2 r}{\varepsilon _{n}} \end{aligned}$$

for n large. For these values of n, we have that $B_{r / \varepsilon _{n}}\left( {\tilde{y}}_{n}\right) \subset \Lambda _{\varepsilon _{n}}$, that is, ${\mathbb {R}}^{N} \backslash \Lambda _{\varepsilon _{n}}\subset {\mathbb {R}}^{N} \backslash B_{r / \varepsilon _{n}}\left( {\tilde{y}}_{n}\right)$. On the other hand, it follows from Lemma 6.1 with $\gamma =a$ that, for any $n \ge n_{0}$ such that $r / \varepsilon _{n}>R$, it holds

$$\begin{aligned} \left\| u_{n}\right\| _{L^{\infty }\left( {\mathbb {R}}^{N} \backslash \Lambda _{\varepsilon _{n}}\right) } \leqslant \left\| u_{n}\right\| _{L^{\infty }\left( {\mathbb {R}}^{N} \backslash B_{r / \varepsilon _{n}}\left( {\tilde{y}}_{n}\right) \right) } \leqslant \left\| u_{n}\right\| _{L^{\infty }\left( {\mathbb {R}}^{N} \backslash B_{R}\left( {\tilde{y}}_{n}\right) \right) }<a, \end{aligned}$$

which contradicts with (6.3). So the claim is correct.

Let ${\hat{\varepsilon }}_{\delta }>0$ be given by Theorem 5.5 and set $\varepsilon _{\delta }:=\min \left\{ {\hat{\varepsilon }}_{\delta }, {\tilde{\varepsilon }}_{\delta }\right\}$. We will show the theorem holds for this choice of $\varepsilon _{\delta }$. Let $0<\varepsilon <\varepsilon _{\delta }$ be fixed. By Theorem 5.5, there is ${\text {cat}}_{M_{\delta }}(M)$ nontrivial solutions of the problem (3.1). If $u \in H_{\varepsilon }$ is one of these solutions, we have that $u \in \widetilde{{\mathcal {N}}}_{\varepsilon }$. Then, by (6.2) $g(\varepsilon x,|u|^{2})= f(|u|^{2})$, u is a solution of the problem (3.1). An easy calculation shows that ${\hat{u}}(x):=u(x / \varepsilon )$ is a solution of the original problem (2.1). Then, problem (2.1) has at least ${\text {cat}}_{M_{\delta }}(M)$ nontrivial solutions.

Take $\varepsilon _{n}\rightarrow 0^{+}$ and $\{u_{n}\}$ a sequence of solutions to (3.1). In order to study the behavior of the maximum points of $\left| u_{n}\right|$, we first notice that, by (g4), there exists $\gamma >0$ such that

$$\begin{aligned} g\left( \varepsilon x, s^{2}\right) s^{2} \leqslant \frac{V_{0}}{K} s^{2} \quad \text{ for } \text{ all } x \in {\mathbb {R}}^{N},|s| \leqslant \gamma . \end{aligned}$$

(6.4)

By Lemma 6.1, there are $R>0$ and $\{{\tilde{y}}_{n}\}\subset {\mathbb {R}}^{N}$ such that

$$\begin{aligned} \left\| u_{n}\right\| _{L^{\infty }\left( B_{R}\left( {\tilde{y}}_{n}\right) ^{c}\right) }<\gamma . \end{aligned}$$

(6.5)

Up to a subsequence, we can assume that

$$\begin{aligned} \Vert u_{n}\Vert _{L^{\infty }\left( B_{R}\left( {\tilde{y}}_{n}\right) \right) } \geqslant \gamma . \end{aligned}$$

(6.6)

Indeed, if this is not the case, we have $\left\| u_{n}\right\| _{L^{\infty }\left( {\mathbb {R}}^{N}\right) }<\gamma$, and therefore, it follows from $J_{\varepsilon _{n}}^{\prime }\left( u_{n}\right) =0$, (6.4) and the diamagnetic inequality (2.2) that

$$\begin{aligned} \begin{aligned} \int \limits _{{\mathbb {R}}^{N}}(|\nabla |u_{n}||^{2}+V_{0}|u_{n}|^{2})\text {d}x&\le \Vert u_{n}\Vert _{\varepsilon _{n}}^{2}\\&=\frac{1}{2_{\mu }^{*}}\int \limits _{{\mathbb {R}}^{N}}\left( \frac{1}{|x|^{\mu }}*G(\varepsilon x, |u_{n}|^{2})\right) g_{\varepsilon _{n}}(x,|u_{n}|^{2})|u_{n}|^{2}\text {d}x\\&\le \frac{V_{0}}{2\cdot 2_{\mu }^{*} }\int \limits _{{\mathbb {R}}^{N}}|u_{n}|^{2}\text {d}x. \end{aligned} \end{aligned}$$

The above expression implies that $\Vert |u_{n}|\Vert _{H^{1}({\mathbb {R}}^{N},{\mathbb {R}})}=0$, which is a contradiction. Thus, (6.6) holds. By (6.5) and (6.6), we can infer the maximum point $p_{n}\in {\mathbb {R}}^{N}$ of $|u_{n}|$ belongs to $B_{R}({\tilde{y}}_{n})$. Hence, $p_{n}={\tilde{y}}_{n}+q_{n}$ for some $q_{n}\in B_{R}(0)$. Recalling that the associated solution of (2.1) is of the form ${\hat{u}}_{n}(x)=u_{n}(x /\varepsilon _{n})$, we conclude that the maximum point $\eta _{n}$ of $|{\hat{u}}_{n}|$ is $\eta _{\varepsilon _{n}}:=\varepsilon _{n} {\tilde{y}}_{n}+\varepsilon _{n} q_{n}$. Since $\left( q_{n}\right) \subset B_{R}(0)$ is bounded and $\varepsilon _{n} {\tilde{y}}_{n} \rightarrow y_{0} \in M$, we obtain

$$\begin{aligned} \lim _{n \rightarrow \infty } V\left( \eta _{\varepsilon _{n}}\right) =V\left( y_{0}\right) =V_{0}. \end{aligned}$$

The proof is complete. $\square$