Critical Fractional (p, q)-Kirchhoff Type Problem with a Generalized Choquard Nonlinearity and Magnetic Field

Abstract

In this article, using variational methods, we obtain that the existence of a nontrivial solution for a fractional (p, q)-Kirchhoff type problem with a generalized Choquard nonlinearity, a critical Hardy–Sobolev term and magnetic field.

1 Introduction

In this paper, we consider the following fractional (p, q)-Kirchhoff type problem

$$\begin{aligned} \left\{ \begin{array}{ll} M([u]^p_{p, A})(-\Delta )^s_{p, A}u+(-\Delta )_{q, A}^{s}u = ({\mathcal {I}}_\mu *F(|u|^2))f(|u|^2)u+\frac{|u|^{{p^*_\alpha }-2}u}{|x|^\alpha } +\lambda k(x)|u|^{r-2}u\ \ \ {} &{} \textrm{in}\ \Omega , \\ u=0 \ \ \quad &{} \textrm{in}\ {\mathbb {R}}^N\backslash \Omega , \end{array} \right. \nonumber \\ \end{aligned}$$

(1.1)

where $\Omega$ is a bounded smooth domain of ${\mathbb {R}}^N$ containing 0 with Lipschitz boundary, $1<q<p$, $0\le \alpha<ps<N$ with $s\in (0, 1)$, $A\in C({\mathbb {R}}^N, {\mathbb {R}}^N)$ is a magnetic potential, $M:{\mathbb {R}}^+_0\rightarrow {\mathbb {R}}^+$ is a Kirchhoff function, f is a continuous function, $F(u)=\int _0^uf(t)dt$, here ${\mathcal {I}}_\mu (x)=|x|^{-\mu }$ is the Riesz potential of order $\mu \in (0, \min \{N, 2ps\})$. $p^*_\alpha =\frac{p(N-\alpha )}{N-ps}$ is critical Sobolev–Hardy exponent, when $\alpha =0$, $p^*=\frac{pN}{N-ps}$ is critical Sobolev exponent, $1<r<p^*_\alpha \le p^*$, $k(x)\in L^\frac{p^*}{p^*-r}(\Omega , {\mathbb {C}})$ and the fractional p-Laplacian magnetic operator $(-\Delta )^s_{p, A}$ is the differential of the convex functional

$$\begin{aligned} \ u\mapsto \frac{1}{p}[u]^p_{p, A}:=\frac{1}{p}\iint _{{\mathbb {R}}^{2N}}\frac{|u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|^p}{|x-y|^{N+ps}}\, \textrm{d}x\, \textrm{d}y \end{aligned}$$

defined on the Banach space (with respect to the norm $[u]_{p, A}$ defined above)

$$\begin{aligned} W^{s, p}_{0, A}(\Omega , {\mathbb {C}}):=\left\{ u\in L^p({\mathbb {R}}^N, {\mathbb {C}})\,\ u\equiv 0 \quad \text {a. e. in } {\mathbb {R}}^N\backslash \Omega \quad \text {and } \ [u]^p_{p, A}<+\infty \right\} . \end{aligned}$$

Let us first mention some results for $A=0$. If $p=q=2$ in (1.1), the operator $(-\Delta )^s_{p, A}$ becomes the fractional Laplacian operator $(-\Delta )^s$ without magnetic, which arises in the study of several physical phenomena like phase transitions, crystal dislocations, quasi-geostrophic flows, flame propagations and so on. It can be seen as the infinitesimal generators of Lévy stable diffusion processes [4]. Recently, there are many works dedicated to study Kirchhoff problem with singular and critical terms but without a Hardy potential and a generalized Choquard term, namely with $\alpha =0$ and $\mu =0$.

Xiang and Wang [19] considered the existence, multiplicity and asymptotic behavior of nonnegative solutions for a fractional Schrödinger–Poisson–Kirchhoff type system

$$\begin{aligned} \left\{ \begin{array}{ll} \big (a+b{||u||}^2\big )[(-\Delta )^{s}u+V(x)u]+\phi k(x)|u|^{p-2}u =\lambda h(x)|u|^{q-2}u+|u|^{2^*_s-2}u\quad \ \ \ {} &{} \textrm{in}\ {\mathbb {R}}^{3}, \\ (-\Delta )^t\phi =k(x)|u|^{p-2}u\ \ \quad &{} \textrm{in}\ {\mathbb {R}}^{3}. \end{array} \right. \end{aligned}$$

When $2p\le q\le 2^*_s$, $2^*_s=\frac{2N}{N-2s}$ and $\lambda >0$ is large enough, existence of nonnegative solutions is obtained by the mountain pass theorem. Then, via the Ekeland variational principle, existence of nonnegative solutions is investigated when $1<q<2$ and $\lambda >0$ is small enough.

If $p=q\ne 2$, Chen [6] established the existence of positive solutions by finding the minimizer of the corresponding energy functional for the following problem

$$\begin{aligned} \qquad \left\{ \begin{array}{ll} M([u]^p_{s, p})(-\Delta )^s_p u = \lambda ({\mathcal {I}}_\mu *F(u))f(u)+\frac{|u|^{{p^*_\alpha }-2}u}{|x|^\alpha }, \quad u>0\ \ \ {} &{} \textrm{in}\ \Omega , \\ u=0 \ \ \quad &{} \textrm{in}\ {\mathbb {R}}^N\backslash \Omega , \end{array} \right. \end{aligned}$$

where $M:{\mathbb {R}}^+_0\rightarrow {\mathbb {R}}^+$ is a Kirchhoff function, $f\in C^1({\mathbb {R}}, {\mathbb {R}})$ fulfills the Ambrosetti-Rabinowitz type condition, $F(u)=\int _0^uf(t)dt$ and $0\le \alpha<ps<N$ with $s\in (0, 1)$.

If $p\ne q$, we can see that, for the classical setting, problem (1.1) reduces to a fractional (p, q)-Laplacian elliptic problem

$$\begin{aligned} (-\Delta )^s_{p}u+(-\Delta )^s_{q}u=g(x,u) \quad \text {in }\Omega , \qquad u=0 \quad \textrm{in}\ {\mathbb {R}}^N\backslash \Omega . \end{aligned}$$

The above fractional (p, q)-Laplacian elliptic problem has been discussed widely in recent years, see [2, 5] for more details. Particularly, using concentration-compactness principle and the Kajikiya’s new version of symmetric mountain pass lemma, Ambrosio and Isernia [3] obtained the existence of infinitely many solutions to the fractional (p, q)-Laplacian problem involving critical Sobolev–Hardy exponent. Moreover, Lin and Zheng [14] considered the following fractional (p, q)-Kirchhoff type problem involving critical Sobolev–Hardy exponent

$$\begin{aligned} \left\{ \begin{array}{ll} \big (a+b[u]_{s,p}^{(\theta -1)p}\big )(-\Delta )_{p}^{s}u+(-\Delta )_{q}^{s}u =\frac{|u|^{p_{\alpha }^{*}-2}u}{|x|^{\alpha }}+\lambda f(x)\frac{|u|^{r-2}u}{|x|^{c}}\quad \ \ \ {} &{} \textrm{in}\ \Omega , \\ u=0\ \quad &{} \textrm{in}\ {\mathbb {R}}^N\backslash \Omega , \end{array} \right. \end{aligned}$$

where $a,b>0$, $c<sr+N(1-\frac{r}{p})$, $\theta \in (1,\frac{p_{\alpha }^{*}}{p})$. The authors proved that there are at least two nontrivial solutions for small $\lambda >0$ by the mountain pass theorem and Ekeland’s variational principle.

However, a lot of attention has been focused on the study of problems with magnetic field in the last decades; both for the pure mathematical research and applications, we refer to [1, 8,9,10, 15, 17, 18, 20] and references therein. If A is a smooth function and $p=q=2$, there are many results on Kirchhoff type problems with magnetic field and involving nonlinear convolution terms, such as Choquard equations. Ambrosio [1] studied the existence and concentration of nontrivial solutions for a fractional Choquard equation with f is continuous and subcritical growth. Xiang et al. [18] obtained the existence and multiplicity of solutions for the following critical fractional Choquard-Kirchhoff type equation

$$\begin{aligned} \ M(\Vert u\Vert ^2_{s, A})[(-\Delta )^s_A u+u] = \lambda \int _\mathbb {{R}^N}\frac{F(|u|^2)}{|x-y|^\alpha }\textrm{d}y f(|u|^2)u+|u|^{{2^*_s}-2}u\ \ \ \textrm{in}\ {\mathbb {R}}^N. \end{aligned}$$

Fiscella and Pucci [11] proposed the nonlinear Schrödinger equations and related systems with magnetic fields and Hardy–Sobolev critical exponents. Yang and An [20] considered the existence of infinitely many solutions of a degenerate magnetic fractional problem. By variational approach, Yang et al. [21] studied the existence of the solutions for the following fractional Schrödinger-Kirchhoff equation involving critical Sobolev–Hardy nonlinearities

$$\begin{aligned} \ M(\Vert u\Vert ^2_{s, A})[(-\Delta )^s_A u+V(x)u]=\frac{|u|^{{2^*_s(\alpha )}-2}u}{|x|^\alpha } +\lambda f(x, |u|)u+g(x, |u|)u\ \ \ \textrm{in}\ {\mathbb {R}}^N, \ \end{aligned}$$

where $2^*_s(\alpha )=\frac{2(N-\alpha )}{N-2s}$ with $\alpha \in [0, 2\,s)$ is the fractional Hardy–Sobolev critical exponent. The main novelty is the presence of the magnetic field and critical term as well as the possible degenerate nature of the Kirchhoff function M.

The fractional p-Kirchhoff type problems with magnetic fields have been studied extensively. Liang and Zhang [12] obtained the existence of infinitely many solutions for the p-fractional Kirchhoff equations with magnetic fields and critical nonlinearity by using the concentration-compactness principle and the Kajikiya’s new version of the symmetric mountain pass lemma. By the variational methods, Song and Shi [17] studied the existence and multiplicity solutions for the p-fractional Schrödinger–Kirchhoff type equations with magnetic field and critical nonlinearity.

The aim of this work is to consider the existence of solutions for the fractional (p, q)-Kirchhoff type problem with a generalized Choquard nonlinearity, a critical Hardy–Sobolev term and magnetic field.

Now we give the following assumptions on the Kirchhoff function M:

($M_1$) $M:{\mathbb {R}}^+_0\rightarrow {\mathbb {R}}^+$ is a continuous function, and there exists $m_0>0$ such that $\inf _{t\ge 0} M(t) =m_0$.

($M_2$) There exists $\theta \in [1, \frac{N-\alpha }{N-ps})$ such that $M(t)t\le \theta {\mathscr {M}}(t)$, $\forall t\ge 0,$ where ${\mathscr {M}}(t)=\int _0^t M(\tau )d\tau$.

A typical example is $M(t)=m_0+bt^{\theta -1}$, where $b\ge 0$, $t\ge 0$.

Moreover, we assume that $f\in C^1({\mathbb {R}}^+, {\mathbb {R}})$, which satisfies

($F_1$) $\lim \limits _{t\rightarrow 0}\frac{|f(t)|}{t^{\frac{p-2}{2}}}=0$,

($F_2$) $\lim \limits _{t\rightarrow \infty }\frac{|f(t)|}{t^{\frac{h-2}{2}}}=0$ for some $\frac{(2N-\mu )p}{2N}<h<\frac{(2N-\mu )p}{2(N-ps)}$,

($F_3$) there exists $\kappa \in (p\theta , r)$ such that for all $t>0$, $0<\kappa {F(t)}\le 4{f(t)}t,$ where ${F}(t)=\int _0^t f(r)dr$.

Furthermore, we assume that

($K_1$) $k(x)\in L^\frac{p^*}{p^*-r}(\Omega , {\mathbb {C}})$ with $1<r<p^*$ and there are two positive constants $\omega _1$ and $\omega _2$ such that $0<\omega _1\le k(x)\le \omega _2<+\infty$, for all $x\in \Omega$.

The main result can be stated as follows:

Theorem 1.1

Assume that $1<q<p$, $0\le \alpha<ps<N$, $0<\mu <\min \{N, 2ps\}$, $(M_1){-}(M_2)$, $(K_1)$ with $p\theta<r<p^*_\alpha$ and $(F_1)-(F_3)$ hold. Then, there exists a constant $\lambda _*>0$ such that problem (1.1) has a nontrivial solution u for all $\lambda >\lambda _*$.

The main feature and difficulty is the presence of (p, q)-Laplacian magnetic operator and Kirchhoff function M. The appearance of the magnetic field brings extra difficulties to the problem. Second, It is difficult to get the Palais–Smale [(PS) for short] condition due to critical Sobolev–Hardy nonlinearity. For this purpose, we use a p-fractional version of concentration-compactness principle with magnetic field to show that the energy functional satisfies local $(PS)_c$ condition for c less than some critical level when the parameter $\lambda$ is large enough.

This paper is organized as follows: In Sect. 2, we give some preliminaries. The proof of Theorem 1.1 will be given in Sect. 3.

2 Preliminaries

In this section, we briefly recall the relevant definitions and notations. The fractional Sobolev space $W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$ is defined by

$$\begin{aligned} W^{s, p}_{0, A}(\Omega , {\mathbb {C}}):=\left\{ u\in L^p({\mathbb {R}}^N, {\mathbb {C}})\,\ u\equiv 0 \quad \text {a. e. in } {\mathbb {R}}^N\backslash \Omega \quad \text {and } \ [u]^p_{p, A}<+\infty \right\} , \end{aligned}$$

where $[u]_{p, A}$ denotes the magnetic Gagliardo semi-norm defined by

$$\begin{aligned}{}[u]_{p, A}=\left( \int \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|^p}{|x-y|^{N+ps}}\textrm{d}x\textrm{d}y\right) ^{\frac{1}{p}}. \end{aligned}$$

According to [8], for $p>1$, $W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$ is a separable reflexive Banach space with the norm $\Vert \cdot \Vert =[\cdot ]_{p,A}$ and the completion with respect to the norm $\Vert \cdot \Vert =[\cdot ]_{p,A}$ of $C_c^\infty (\Omega , {\mathbb {C}})$. The topological dual of $W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$ will be denoted by $W^{-s, p'}_{0, A}(\Omega , {\mathbb {C}})$ with the corresponding duality pairing $\langle \cdot , \cdot \rangle :W^{-s, p'}_{0, A}(\Omega , {\mathbb {C}})\times W^{s, p}_{0, A}(\Omega , {\mathbb {C}}) \rightarrow {\mathbb {R}}$. Due to reflexivity, the weak and weak $*$ convergence in $W^{-s, p'}_{0, A}(\Omega , {\mathbb {C}})$ coincides.

For $1<q<p$, let us set ${\mathcal {W}} =W^{s, p}_{0, A}(\Omega , {\mathbb {C}})\cap W^{s, q}_{0, A}(\Omega , {\mathbb {C}})$ endowed with the norm $[u]_{{\mathcal {W}}}:=[u]_{p, A}+[u]_{q, A}.$ Moreover, set ${{\mathcal {W}}}'=W^{-s, p'}_{0, A}(\Omega , {\mathbb {C}})\cap W^{-s, q'}_{0, A}(\Omega , {\mathbb {C}}).$

According to the diamagnetic inequality $||u(x)|-|u(y)||\le |u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|$, for $a. e. x, y\in {\mathbb {R}}^N$, in [9], we have the following inequality.

Lemma 2.1

For every $u\in W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$, we get $|u|\in W^{s, p}_0(\Omega )$. More precisely, $[|u|]_s\le [u]_{p, A},$ where $[|u|]_s=\left( \int \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^p}{|x-y|^{N+ps}}\textrm{d}x\textrm{d}y\right) ^{\frac{1}{p}}$.

Lemma 2.2

Assume that $0\le \alpha \le ps<N$. Then, there exists a positive constant C such that

$$\begin{aligned} \Big (\int _{{\mathbb {R}}^{N}}\frac{|u|^{p^*_\alpha }}{|x|^\alpha } \, \textrm{d}x \Big )^{\frac{1}{p^*_\alpha }} \le C[u]_{p, A} \quad \text {for every } u\in W^{s, p}_{0, A}(\Omega , {\mathbb {C}}). \end{aligned}$$

Proof

Combining the results of Lemma 2.1 and [7, Lemma 2.1], we can get the result. $\square$

In particular, $W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$ embeds continuously into $L^h {(\Omega , {\textrm{d}x}/{|x|^\alpha })}$ for all $\alpha \in [0, ps]$ and $h\in [1, p^*_\alpha ]$. Moreover, if $h\in [1, p^*_\alpha )$, the embedding is compact. Thanks to the previous lemma, we can define for any $\alpha \in [0, ps]$ the positive numbers, when $\alpha =0$, $S_\alpha$ becomes the best Sobolev constant S.

$$\begin{aligned} S_\alpha =\inf \left\{ \int \int _{{\mathbb {R}}^{2N}}\frac{|u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|^p}{|x-y|^{N+ps}}\textrm{d}x\textrm{d}y: u\in W^{s, p}_{0, A}(\Omega , {\mathbb {C}})\quad \text {with } \int _{\Omega }\frac{|u|^{p^*_\alpha }}{|x|^\alpha } \, \textrm{d}x=1 \right\} . \end{aligned}$$

Next, we recall the following Hardy–Littlewood–Sobolev inequality.

Lemma 2.3

[13] Assume that $1<\nu$, $t<\infty$, $0<\mu <N$ and $\frac{1}{\nu }+\frac{1}{t}+\frac{\mu }{N}=2.$ Then, there exists $C(N, \mu , \nu , t)>0$ such that

$$\begin{aligned} \iint _{{\mathbb {R}}^{2N}}\frac{|g(x)||b(y)|}{|x-y|^\mu }\, \textrm{d}x\, \textrm{d}y\le C(N, \mu , \nu , t)\Vert g\Vert _\nu \Vert b\Vert _t \end{aligned}$$

for all $g\in L^\nu ({\mathbb {R}}^N, {\mathbb {C}})$ and $b\in L^t({\mathbb {R}}^N, {\mathbb {C}})$.

In particular, $F(t)=|t|^{h}$ for some $h>0$, by the Hardy–Littlewood–Sobolev inequality, the integral

$$\begin{aligned} \iint _{{\mathbb {R}}^{2N}}\frac{ F\left( |u(x)|^2\right) F\left( |u(y)|^2\right) }{|x-y|^\mu }\, \textrm{d}x\, \textrm{d}y \end{aligned}$$

is well defined if $F(|u|^2)\in L^t({\mathbb {R}}^N)$ for some $t>1$ satisfying $\frac{2}{t}+\frac{\mu }{N}=2$, that is, $t=\frac{2N}{2N-\mu }.$ Hence, thanks to the fact that the fractional Sobolev embedding theorem, if $u\in W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$, we must require that $th \in [p, p^*_\alpha ]$. Thus, for the subcritical case, we must assume

$$\begin{aligned} {\tilde{p}}_{\mu , s}:=\frac{(2N-\mu )p}{2N}<h<\frac{(2N-\mu )p}{2(N-ps)}:=p_{\mu , s}^*. \end{aligned}$$

Hence, ${\tilde{p}}_{\mu , s}$ is said to be the lower critical exponent and $p_{\mu , s}^*$ is called the upper critical exponent in the sense of the Hardy–Littlewood–Sobolev inequality.

The energy functional ${\mathcal {J}}_\lambda$ formally associated with problem (1.1) is

$$\begin{aligned} {\mathcal {J}}_\lambda (u) =\Phi (u)- \Psi (u)-H_\alpha (u)-\lambda K(u), \end{aligned}$$

with

$$\begin{aligned} \Phi (u)=\frac{1}{p}{\mathscr {M}}([u]^p_{p, A})+\frac{1}{q}[u]_{q,A}^{q}, \quad \Psi (u)= \frac{1}{4} \int _{\Omega }\int _{\Omega }\frac{ F(|u(x)|^2)F(|u(y)|^2)}{|x-y|^\mu }\, \textrm{d}x\, \textrm{d}y, \quad \end{aligned}$$

$$\begin{aligned} H_\alpha (u)=\frac{1}{p^*_\alpha }\int _{\Omega }\frac{|u|^{p^*_\alpha }}{|x|^\alpha } \, \textrm{d}x,\quad K(u)=\frac{1}{r}\int _{\Omega }k(x)|u|^r\textrm{d}x.\qquad \qquad \quad \qquad \qquad \quad \end{aligned}$$

Let $\Re e$ and the bar denote the real part of a complex number and the complex conjugation, respectively. We have the following results.

Lemma 2.4

[6, Lemma 2.3] Let $(M_1)$ hold. Then, $\Phi$ is of class $C^1$ and

$$\begin{aligned} \langle \Phi '(u), \varphi \rangle&=M([u]^p_{p, A})\Re e\iint _{{\mathbb {R}}^{2N}}\frac{1}{|x-y|^{N+ps}}|u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|^{p-2}\\&\quad \times \left( u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)\right) \overline{\left( {\varphi (x)-e^{i(x-y)A(\frac{x+y}{p})}\varphi (y)}\right) }\textrm{d}x\textrm{d}y\\&\quad +\Re e\iint _{{\mathbb {R}}^{2N}}\frac{1}{|x-y|^{N+qs}}|u(x)-e^{i(x-y)A(\frac{x+y}{q})}u(y)|^{q-2}\\&\quad \times \left( u(x)-e^{i(x-y)A(\frac{x+y}{q})}u(y)\right) \overline{\left( {\varphi (x)-e^{i(x-y)A(\frac{x+y}{q})}\varphi (y)}\right) }\textrm{d}x\textrm{d}y, \end{aligned}$$

for all $u, \varphi \in {\mathcal {W}}$. Moreover, $\Phi$ is weakly lower semi-continuous in ${\mathcal {W}}$.

Lemma 2.5

[7, Lemma 2.3] Let $0\le \alpha \le ps<N$. Then, $H_\alpha$ is of class $C^1$ with

$$\begin{aligned} \langle H'_\alpha (u), \varphi \rangle =\Re e\int _{\Omega }\frac{|u|^{{p^*_\alpha }-2}u{\overline{\varphi }}}{|x|^\alpha }\, \textrm{d}x \quad \text {for every } u, \varphi \in W^{s, p}_{0, A}(\Omega , {\mathbb {C}}). \end{aligned}$$

Moreover, the operator $H'_\alpha : W_{0, A}^{s, p}(\Omega , {\mathbb {C}}) \rightarrow W^{-s, p'}_{0, A}(\Omega , {\mathbb {C}})$ is sequentially weak-to-weak continuous.

Lemma 2.6

[6, Lemma 2.5] Assume $(F_1)$ and $(F_2)$ hold, we have

$$\begin{aligned} \left| \int _{\Omega }\int _{\Omega } \frac{F(|u(y)|^2)}{|x-y|^\mu }f(|u(x)|^2)|u(x)|^2\, \textrm{d}x\, \textrm{d}y\right| \le C([u]^{2p}_{p, A}+[u]^{2h}_{p, A}) \end{aligned}$$

and

$$\begin{aligned} \left| \int _{\Omega }\int _{\Omega } \frac{F(|u(x)|^2)F(|u(y)|^2)}{|x-y|^\mu }\, \textrm{d}x\, \textrm{d}y\right| \le C([u]^{2p}_{p, A}+[u]^{2h}_{p, A}). \end{aligned}$$

Lemma 2.7

[6, Lemma 2.6] Let $(F_1)-(F_2)$ hold and $0<\mu <\min \{N, 2ps\}$. Then, $\Psi$ and $\Psi '$ are weakly strongly continuous on $W^{s, p}_{0, A}(\Omega , {\mathbb {C}})$.

From Lemmas 2.4–2.7, and conditions $(F_1)$–$(F_3)$, we have that ${\mathcal {J}}_\lambda (u)$ is of class $C^1$. We say that $u\in {\mathcal {W}}$ is a weak solution of problem (1.1), if

$$\begin{aligned} M([u]^p_{p, A})\langle u, \varphi \rangle _{p, A}+\langle u,\varphi \rangle _{q, A}&= \Re e\int _{\Omega }\int _{\Omega } \frac{F(|u(y)|^2)}{|x-y|^\mu }f(|u(x)|^2)u(x)\overline{\varphi (x)}\, \textrm{d}x\, \textrm{d}y\\&\quad +\Re e\int _{\Omega }\frac{|u|^{{p^*_\alpha }-2}u\overline{\varphi (x)}}{|x|^\alpha }\, \textrm{d}x\\&\quad +\lambda \Re e\int _{\Omega } k(x)|u|^{r-2}u\overline{\varphi (x)}\textrm{d}x, \end{aligned}$$

where ${\langle u, \varphi \rangle }_{t, A}$ with $t\in \{p,q\}$ is defined by

$$\begin{aligned} {\langle u, \varphi \rangle }_{t, A}&=\Re e\iint _{{\mathbb {R}}^{2N}}\frac{1}{|x-y|^{N+ts}}|u(x)\\&\quad -e^{i(x-y)A(\frac{x+y}{t})}u(y)|^{t-2}\left( u(x)-e^{i(x-y)A(\frac{x+y}{t})}u(y)\right) \\&\quad \times \overline{\left( {\varphi (x)-e^{i(x-y)A(\frac{x+y}{t})}\varphi (y)}\right) }\textrm{d}x\textrm{d}y, \end{aligned}$$

for all $\varphi \in {\mathcal {W}}$. Clearly, the critical points of ${\mathcal {J}}_\lambda (u)$ are exactly the weak solutions of problem (1.1).

3 Proof of Theorem 1.1

We start by showing that functional ${\mathcal {J}}_\lambda$ has the geometric structure of the mountain pass theorem.

Lemma 3.1

Assume that $(M_1)-(M_2)$ and $(F_1)-(F_3)$ hold. Then,

(i) there exist $\vartheta , \rho >0$ such that ${\mathcal {J}}_\lambda (u)\ge \vartheta$ for all $u\in {\mathcal {W}}$ with $[u]_{{\mathcal {W}}}=\rho$.

(ii) There exist $e\in {\mathcal {W}}$ and $\rho >0$ such that $[e]_{{\mathcal {W}}}>\rho$ and ${\mathcal {J}}_\lambda (e)<0$.

Proof

(i) From Lemma 2.6, $(M_1)-(M_2)$, and by Hölder inequality and the fractional Hardy–Sobolev embedding, we get

$$\begin{aligned} {\mathcal {J}}_\lambda (u)&=\frac{1}{p}{\mathscr {M}}([u]^p_{p, A})+\frac{1}{q}[u]_{q,A}^{q}\\&\quad - \frac{1}{4}\int _{\Omega }\int _{\Omega }\frac{ F({|u(x)|}^2)F({|u(y)|}^2)}{|x-y|^\mu }\, \textrm{d}x\,\textrm{d}y \\&\quad -\frac{1}{p^*_\alpha }\int _{\Omega }\frac{|u|^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x-\frac{\lambda }{r}\int _{\Omega }k(x)|u|^r\textrm{d}x\\&\ge \frac{1}{p\theta }M([u]^p_{p, A})[u]^p_{p, A}-C([u]^{2p}_{p, A}\\&\quad +[u]^{2h}_{p, A})-C[u]^{p^*_\alpha }_{p, A}-C||k(x)||_{\frac{p^*}{p^*-r}}[u]^r_{p, A}\\&\ge \frac{m_0}{p\theta }[u]^p_{p, A}-C([u]^{2p}_{p, A}+[u]^{2h}_{p, A})\\&\quad -C[u]^{p^*_\alpha }_{p, A}-C||k(x)||_{\frac{p^*}{p^*-r}}[u]^r_{p, A}. \end{aligned}$$

Since $\frac{(2N-\mu )p}{2N}<h<\frac{(2N-\mu )p}{2(N-ps)}$ and $p\theta<r<p^*_\alpha$, we have $p<2h$, $p<r$ and $p<p^*_\alpha$, and then the claim follows if we choose $\rho$ small enough.

(ii) Assume $u_0\in {\mathcal {W}}$, $(M_2)$ and $(F_3)$ implies that

$$\begin{aligned} {\mathcal {J}}_\lambda (tu_0)&\le \frac{1}{p}{\mathscr {M}}([tu_0]^p_{p, A})+\frac{1}{q}[tu_0]_{q,A}^{q}\\&\quad - \frac{1}{4}\int _{\Omega }\int _{\Omega }\frac{ F({|tu_0{(x)}|}^2)F({|tu_0{(y)}|}^2)}{|x-y|^\mu }\, \textrm{d}x\,\textrm{d}y \\&\quad -\frac{t^{p^*_\alpha }}{p^*_\alpha }\int _{\Omega }\frac{|u_0|^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x\\&\le \frac{1}{p}{\mathscr {M}}(1)t^{p\theta }[u_0]^{p\theta }_{p, A}+\frac{1}{q}t^q[u_0]_{q,A}^{q}\\&\quad -\frac{t^{p^*_\alpha }}{p^*_\alpha }\int _{\Omega }\frac{|u_0|^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x\rightarrow -\infty , \quad \text {as } t\rightarrow +\infty , \end{aligned}$$

since $q<p<\theta p<p^{*}_\alpha$. Thus, there exist $e\in {\mathcal {W}}$ and $\rho >0$ such that $[e]_{{\mathcal {W}}}>\rho$ and ${\mathcal {J}}_\lambda (e)<0$. $\square$

Lemma 3.2

If conditions $(M_1)-(M_2)$ and $(F_1)-(F_3)$ hold. Let $\{u_n\}$ be a $(PS)_{c_\lambda }$ sequence of functional ${\mathcal {J}}_\lambda$ with $c_\lambda <\left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) (m_0 S_\alpha )^{\frac{p^*_\alpha }{p^*_\alpha -p}}$. Then exists a subsequence of ${\{u_n\}}$ strongly converges in ${\mathcal {W}}$.

Proof

Since ${\mathcal {J}}_\lambda (u_n)\rightarrow c_\lambda$ and ${\mathcal {J}}'_\lambda (u_n)\rightarrow 0$ in ${{\mathcal {W}}}'$, by $(M_1)$, $(M_2)$ and $(F_3)$, then there exists $C > 0$ such that

$$\begin{aligned}&C+o(1)[u]_{p, A}\ge {\mathcal {J}}_\lambda (u_n)-\frac{1}{\kappa }\langle {\mathcal {J}}'_\lambda (u_n), u_n\rangle \\&\quad =\frac{1}{p}{\mathscr {M}}([u_n]^p_{p, A})+\frac{1}{q}[u_n]_{q,A}^{q}- \frac{1}{\kappa }M([u_n]^p_{p, A})[u_n]^p_{p, A}\\&\qquad -\frac{1}{\kappa }[u_n]_{q,A}^{q}+\lambda \left( \frac{1}{\kappa }-\frac{1}{r}\right) \int _{\Omega }k(x)|u_n|^r\textrm{d}x \\&\qquad + \int _{\Omega }\left( {\mathcal {I}}_\mu *F({|u_n|}^2)\right) \left( \frac{1}{\kappa }f(|u_n|^2)|u_n|^2-\frac{1}{4}F(|u_n|^2)\right) \textrm{d}x\\&\qquad +\left( \frac{1}{\kappa }-\frac{1}{p^*_\alpha }\right) \int _{\Omega }\frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x \\&\quad \ge \left( \frac{1}{p\theta }-\frac{1}{\kappa }\right) M([u_n]^p_{p, A})[u_n]^p_{p, A}\\&\quad \ge \left( \frac{1}{p\theta }-\frac{1}{\kappa }\right) m_0[u_n]^p_{p, A}. \end{aligned}$$

This implies that $\{u_n\}$ is bounded in ${\mathcal {W}}$ with $\kappa>p\theta >q$. By the concentration-compactness principle [8], there exist $u\in {\mathcal {W}}$, two Borel regular measures $\mu$ and $\nu$, at most countable set $\{{x_j}\}_ J\subseteq {{\overline{\Omega }}}$, and non-negative numbers $\{{\mu _j}\}_{j\in J}$, $\{{\nu _j}\}_{j\in J}\subset [0, \infty )$ such that, up to subsequence, $u_n\rightharpoonup u$ in ${\mathcal {W}}$, $u_n\rightarrow u\ \text{ a. } \text{ e. } \text{ in } \Omega$ and

$$\begin{aligned}&u_n\rightarrow u\ \ \text{ in } \ L^r(\Omega , \textrm{d}x/|x|^\alpha ) \ \text{ for } p\le r<p^*_\alpha ,\ 0\le \alpha <ps, \end{aligned}$$

(3.1)

as $n\rightarrow \infty$. Moreover

$$\begin{aligned}&\mu _n\rightharpoonup ^*\mu , \quad \frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\rightharpoonup ^*\nu ,\\&\mu =\int _{{\mathbb {R}}^{N}}\frac{|u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|^p}{|x-y|^{N+ps}}\textrm{d}y+\sum _{j\in J} \mu _j\delta _{x_j}+{\tilde{\mu }}, \quad \mu _j :=\mu (\{x_j\}), \end{aligned}$$

$$\begin{aligned} \nu =\frac{|u|^{p^*_\alpha }}{|x|^\alpha }+\sum _{j\in J} \nu _j \delta _{x_j}, \quad \nu _j :=\nu (\{x_j\}), \end{aligned}$$

(3.2)

$$\begin{aligned} \mu _j\ge S_\alpha \nu _j^{\frac{p}{p^*_\alpha }}. \end{aligned}$$

Fix $i_0\in J$, we are ready to prove that either $v_{i_0}=0$ or

$$\begin{aligned} v_{i_0}\ge (m_0 S_\alpha )^{\frac{p^*_\alpha }{p^*_\alpha -p}}. \end{aligned}$$

(3.3)

In fact, let $\varphi _\epsilon \in C^\infty _0{(B_{2\epsilon }(x_{i_0}))}$ satisfy $0\le \varphi _\epsilon \le 1$, $\varphi _\epsilon |_{B_{\epsilon }(x_{i_0})}=1$ and $||\bigtriangledown \varphi _\epsilon ||_\infty \le \frac{C}{\epsilon }$. Clearly $\{ \varphi _\varepsilon u_n\}$ is bounded in ${\mathcal {W}}$ and $\langle {\mathcal {J}}'_\lambda (u_n), \varphi _\varepsilon u_n\rangle \rightarrow 0$ as $n\rightarrow \infty$. Thus

$$\begin{aligned}&M([u_n]^p_{p, A})\langle u_n, \varphi _\varepsilon u_n\rangle _{p, A}+\langle u_n,\varphi _\varepsilon u_n\rangle _{q, A} \nonumber \\&\quad = \Re e\int _{\Omega }\int _{\Omega } \frac{F(|u_n{(y)}|^2)}{|x-y|^\mu } f(|u_n{(x)}|^2)u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\, \textrm{d}x\, \textrm{d}y\nonumber \\&\qquad +\Re e\int _{\Omega }\frac{|u_n{(x)}|^{p^*_\alpha -2}}{|x|^\alpha }u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\, \textrm{d}x\nonumber \\&\qquad +\lambda \Re e \int _{\Omega }k(x)|u_n(x)|^{r-2}u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\textrm{d}x. \end{aligned}$$

(3.4)

On the one hand, $\langle u_n, \varphi _\varepsilon u_n\rangle _{t, A}$ with $t\in \{p,q\}$ is defined by

$$\begin{aligned}&\langle u_n, \varphi _\varepsilon u_n\rangle _{t, A} \nonumber \\&\quad =\Re e\iint _{{\mathbb {R}}^{2N}}\frac{1}{|x-y|^{N+ts}}|u_n{(x)}-e^{i(x-y)A(\frac{x+y}{t})}u_n{(y)}|^{t-2}\left( u_n{(x)}-e^{i(x-y)A(\frac{x+y}{t})}u_n{(y)}\right) \nonumber \\&\qquad \times \overline{\left( \varphi _\epsilon (x)u_n{(x)}-e^{i(x-y)A(\frac{x+y}{t})}\varphi _\epsilon (y)u_n{(y)}\right) }\textrm{d}x\textrm{d}y\nonumber \\&\quad =\iint _{{\mathbb {R}}^{2N}}\frac{|u_n{(x)}-e^{i(x-y)A(\frac{x+y}{t})}u_n{(y)}|^{t}\varphi _\epsilon (x)}{|x-y|^{N+ts}}\textrm{d}x\textrm{d}y\nonumber \\&\qquad +\Re e\iint _{{\mathbb {R}}^{2N}}\frac{|u_n{(x)}-e^{i(x-y)A(\frac{x+y}{t})}u_n{(y)}|^{t-2}\left( u_n{(x)}-e^{i(x-y)A(\frac{x+y}{t})}u_n{(y)}\right) }{|x-y|^{N+ts}}\nonumber \\&\qquad \times \overline{e^{i(x-y)A(\frac{x+y}{t})}u_n{(y)}}\left( \varphi _\epsilon (x)-\varphi _\epsilon (y)\right) \textrm{d}x\textrm{d}y. \end{aligned}$$

(3.5)

First,

$$\begin{aligned}&\iint _{{\mathbb {R}}^{2N}}\frac{|u_n{(x)}-e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}|^{p-2} \left( u_n{(x)}-e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}\right) ^2\varphi _\epsilon (x)}{|x-y|^{N+ps}}\textrm{d}x\textrm{d}y \\&\quad \rightarrow \int _{{\mathbb {R}}^{N}}\varphi _\epsilon (x)\, d\mu , \end{aligned}$$

as $n\rightarrow \infty$. Taking $\epsilon \rightarrow 0$, we obtain at once that

$$\begin{aligned}&\lim _{\epsilon \rightarrow 0}\lim _{n\rightarrow \infty }\iint _{{\mathbb {R}}^{2N}}\frac{|u_n{(x)}-e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}|^{p-2}\left( u_n{(x)}-e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}\right) ^2\varphi _\epsilon (x)}{|x-y|^{N+ps}}\textrm{d}x\textrm{d}y\nonumber \\ {}&\quad =\mu _{i_0}. \end{aligned}$$

(3.6)

From [16, Lemma 2.6], we have

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\limsup _{n\rightarrow \infty }\int _{{\mathbb {R}}^{N}}\int _{{\mathbb {R}}^{N}}\frac{|\varphi _\epsilon (x)-\varphi _\epsilon (y)|^p}{|x-y|^{N+ps}}|u_n(x)|^p\textrm{d}x\textrm{d}y=0. \end{aligned}$$

Then, using the Hölder inequality, we get

$$\begin{aligned}&\Big |\iint _{{\mathbb {R}}^{2N}}\frac{|u_n{(x)}-e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}|^{p-2}\left( u_n{(x)}-e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}\right) }{|x-y|^{N+ps}}\nonumber \\&\quad \times \overline{e^{i(x-y)A(\frac{x+y}{p})}u_n{(y)}}\left( \varphi _\epsilon (x)-\varphi _\epsilon (y)\right) \textrm{d}x\textrm{d}y\Big |\nonumber \\&\le [u_n]^{p-1}_{p, A}\left( \int _{{\mathbb {R}}^{N}}\int _{{\mathbb {R}}^{N}}\frac{{|\varphi _\epsilon (x)-\varphi _\epsilon (y)|}^p}{|x-y|^{N+ps}} {|u_n{(y)}|}^p \textrm{d}x\textrm{d}y\right) ^{\frac{1}{p}}\rightarrow 0, \end{aligned}$$

(3.7)

as $n\rightarrow \infty$ and $\epsilon \rightarrow 0$. For the second term on the left-hand side of (3.4), similarly, we obtain

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0} \lim _{n\rightarrow \infty }{\langle u_n, \varphi _\epsilon u_n\rangle }_{q, A}\rightarrow \mu _{i_0}. \end{aligned}$$

By the continuity of M(t) and (3.5)–(3.7), we have

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\lim _{n\rightarrow \infty }\Big [M([u_n]^{p}_{p, A})\langle u_n, \varphi _\varepsilon u_n\rangle _{p, A}+\langle u_n,\varphi _\varepsilon u_n\rangle _{q, A}\Big ]\ge M(d^p)\mu _{i_0}, \end{aligned}$$

(3.8)

where $d=\lim _{n\rightarrow \infty }[u_n]_{p, A}$. On the other hand,

$$\begin{aligned}&\lim _{n\rightarrow \infty }\int _{\Omega }\int _{\Omega } \frac{F(|u_n{(y)}|^2)}{|x-y|^\mu } f(|u_n{(x)}|^2)u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\, \textrm{d}x\, \textrm{d}y\\&\quad =\int _{\Omega }\int _{\Omega } \frac{F(|u{(y)}|^2)}{|x-y|^\mu } f(|u{(x)}|^2){|u(x)|}^2\varphi _\epsilon (x)\, \textrm{d}x\, \textrm{d}y \end{aligned}$$

and

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\int _{\Omega }\int _{\Omega } \frac{F(|u{(y)}|^2)}{|x-y|^\mu } f(|u{(x)}|^2){|u(x)|}^2\varphi _\epsilon (x)\, \textrm{d}x\, \textrm{d}y=0. \end{aligned}$$

Then, we have shown that

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\lim _{n\rightarrow \infty }\int _{\Omega }\int _{\Omega } \frac{F(|u_n{(y)}|^2)}{|x-y|^\mu } f(|u_n{(x)}|^2)u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\, \textrm{d}x\, \textrm{d}y=0. \end{aligned}$$

(3.9)

Meanwhile,

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\lim _{n\rightarrow \infty }\int _{\Omega }k(x)|u_n(x)|^{r-2}u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\textrm{d}x=0. \end{aligned}$$

(3.10)

Furthermore, turning to (3.2), we deduce that

$$\begin{aligned} \lim _{\epsilon \rightarrow 0}\lim _{n\rightarrow \infty }\int _{\Omega }\frac{|u_n{(x)}|^{p^*_\alpha -2}}{|x|^\alpha }u_n(x)\varphi _\epsilon (x)\overline{u_n(x)}\, \textrm{d}x=\lim _{\epsilon \rightarrow 0}\int _{\Omega }\varphi _\epsilon \, d\nu =\nu _{i_0}. \end{aligned}$$

(3.11)

Therefore, taking the limit for $n\rightarrow \infty$ and $\epsilon \rightarrow 0$ in (3.4), from (3.8) to (3.11), one has $M(d^p)\mu _{i_0}\le \nu _{i_0}.$ This together with $(M_1)$ implies that

$$\begin{aligned} m_0 \mu _{i_0}\le M(d^p)\mu _{i_0}\le \nu _{i_0}. \end{aligned}$$

It follows from $\mu _j\ge S_\alpha \nu _j^{\frac{p}{p^*_\alpha }}$ for all $j\in \Lambda$ that

$$\begin{aligned} v_{i_0}\le {S_\alpha }^{-\frac{p^*_\alpha }{p}} \left( \frac{\nu _{i_0}}{m_0}\right) ^{\frac{p^*_\alpha }{p}}. \end{aligned}$$

(3.12)

Hence $\nu _{i_0}=0$ or $v_{i_0}\ge (m_0 S_\alpha )^{\frac{p^*_\alpha }{p^*_\alpha -p}}$.

Next, we conclude that (3.3) can not occur; hence, $\nu _j=0$ for all $j\in \Lambda$.

By contradiction, we assume that there exists ${i_0}\in \Lambda$ such that (3.3) holds. By ${\mathcal {J}}_\lambda (u_n)\rightarrow c_\lambda$ and ${\mathcal {J}}'_\lambda (u_n)\rightarrow 0$ in ${{\mathcal {W}}}'$, we have

$$\begin{aligned} c_\lambda =\lim _{n\rightarrow \infty }\left( {\mathcal {J}}_\lambda (u_n)-\frac{1}{p\theta }\langle {\mathcal {J}}'_\lambda (u_n), u_n\rangle \right) . \end{aligned}$$

(3.13)

From $(M_2)$ and $(F_3)$, one has

$$\begin{aligned}&{\mathcal {J}}_\lambda (u_n)-\frac{1}{p\theta }\langle {\mathcal {J}}'_\lambda (u_n), u_n\rangle \nonumber \\&\quad \ge \frac{1}{p}{\mathscr {M}}([u_n]^p_{p, A})- \frac{1}{p\theta }M([u_n]^p_{p, A})[u_n]^p_{p, A}\nonumber \\&\qquad + \int _{\Omega }\left( {\mathcal {I}}_\mu *F({|u_n|}^2)\right) \left( \frac{1}{p\theta }f(|u_n|^2)|u_n|^2-\frac{1}{4}F(|u_n|^2)\right) \textrm{d}x\nonumber \\&\qquad +\left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) \int _{\Omega }\frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x+\lambda \left( \frac{1}{p\theta }-\frac{1}{r}\right) \int _{\Omega }k(x)|u_n|^r\textrm{d}x\nonumber \\&\quad \ge \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) \int _{\Omega }\frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x\nonumber \\&\quad \ge \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) \int _{\Omega }\frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\varphi _\epsilon (x)\, \textrm{d}x, \end{aligned}$$

(3.14)

since $\theta \ge 1$, $p\theta <p^*_\alpha$, and $0\le \varphi _\epsilon \le 1$, where $\varphi _\epsilon$ is defined as above. From (3.2), (3.13) and (3.14), we find

$$\begin{aligned}{} & {} c_\lambda =\lim _{n\rightarrow \infty }{\mathcal {J}}_\lambda (u_n)=\lim _{n\rightarrow \infty }\left( {\mathcal {J}}_\lambda (u_n)-\frac{1}{p\theta }\langle {\mathcal {J}}'_\lambda (u_n), u_n\rangle \right) \\{} & {} \quad \ge \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) \int _{\Omega }\varphi _\epsilon (x)\, d\nu . \end{aligned}$$

Letting $\epsilon \rightarrow 0$ and using (3.3), it holds that

$$\begin{aligned} c_\lambda \ge \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) \nu _{i_0}\ge \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) (m_0 S_\alpha )^{\frac{p^*_\alpha }{p^*_\alpha -p}}, \end{aligned}$$

which contradicts the assumption. Hence $\nu _j\equiv 0$ for all $j\in \Lambda$ and then

$$\begin{aligned} \frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\rightarrow \frac{|u|^{p^*_\alpha }}{|x|^\alpha }, \quad \text {as } n\rightarrow \infty . \end{aligned}$$

(3.15)

Finally, we show that $u_n\rightarrow u$ strongly in ${\mathcal {W}}$. In fact, for simplicity, let $\varphi \in {\mathcal {W}}$ be fixed and $B^t_\varphi$ be the linear functional on $W^{s, t}_{0, A}(\Omega , {\mathbb {C}})$ defined by

$$\begin{aligned}&B^t_\varphi (v) \\&\quad =\Re e\iint _{{\mathbb {R}}^{2N}}\frac{|\varphi (x)-e^{i(x-y)A(\frac{x+y}{t})}\varphi (y)|^{t-2}\left( \varphi (x)-e^{i(x-y)A(\frac{x+y}{t})}\varphi (y)\right) \overline{\left( v(x)-e^{i(x-y)A(\frac{x+y}{t})}v(y)\right) }}{|x-y|^{N+ts}}\, \textrm{d}x\, \textrm{d}y, \end{aligned}$$

for all $v\in {\mathcal {W}}$.

By the Hölder inequality, we have $\left| B^p_\varphi (v)\right| \le [\varphi ]_{p, A}^{p-1}[v]_{p, A},$ for all $v\in {\mathcal {W}}$. Hence, (3.1) gives that

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }\Big (M([u_n]^p_{p, A})-M([u]^p_{p, A})\Big )B^p_u(u_n-u)=0, \end{aligned}$$

(3.16)

since $\Big \{M([u_n]^p_{p, A})-M([u]^p_{p, A})\Big \}_n$ is bounded in ${\mathbb {R}}$.

Since ${\mathcal {J}}'_\lambda (u_n)\rightarrow 0$ in ${\mathcal {W}}'$ and $u_n\rightharpoonup u$ in ${\mathcal {W}}$, we have $\langle {\mathcal {J}}'_\lambda (u_n)-{\mathcal {J}}'_\lambda (u), u_n-u\rangle \rightarrow 0$, as $n\rightarrow \infty .$

$$\begin{aligned} o(1)&=\langle {\mathcal {J}}'_\lambda (u_n)-{\mathcal {J}}'_\lambda (u), u_n-u\rangle \nonumber \\&=M([u_n]^p_{p, A})B^p_{u_n}(u_n-u)-M([u]^p_{p, A})B^p_{u}(u_n-u)+B^q_{u_n}(u_n-u)-B^q_{u}(u_n-u)\nonumber \\&\quad -\Re e\int _{\Omega }\Big [({\mathcal {I}}_\mu *F({|u_n|}^2))f({|u_n|}^2)u_n-({\mathcal {I}}_\mu *F(|u|^2))f(|u|^2)u\Big ]\overline{(u_n-u)}\textrm{d}x\nonumber \\&\quad -\Re e\int _{\Omega }\left[ \frac{|u_n|^{p^*_\alpha -2} u_n}{|x|^\alpha }-\frac{|u|^{p^*_\alpha -2}u}{|x|^\alpha }\right] \overline{(u_n-u)}\, \textrm{d}x\nonumber \\&\quad -\lambda \Re e\int _{\Omega } \left[ k(x)|u_n|^{r-2}u_n-k(x)|u|^{r-2}u\right] \overline{(u_n-u)}\textrm{d}x\nonumber \\&=M([u_n]^p_{p, A}) \Big [B^p_{u_n}(u_n-u)- B^p_{u}(u_n-u)\Big ] \nonumber \\&\quad +\Big (M([u_n]^p_{p, A})-M([u]^p_{p, A})\Big ) B^p_{u}(u_n-u)+B^q_{u_n}(u_n-u)-B^q_{u}(u_n-u)\nonumber \\&\quad -\Re e\int _{\Omega }\Big [({\mathcal {I}}_\mu *F({|u_n|}^2)f({|u_n|}^2)u_n-({\mathcal {I}}_\mu *F(|u|^2))f(|u|^2)u\Big ]\overline{(u_n-u)}\textrm{d}x\nonumber \\&\quad -\Re e\int _{\Omega }\left[ \frac{|u_n|^{p^*_\alpha -2}u_n}{|x|^\alpha }-\frac{|u|^{p^*_\alpha -2}u}{|x|^\alpha }\right] \overline{(u_n-u)}\, \textrm{d}x\nonumber \\&\quad -\lambda \Re e\int _{\Omega } \left[ k(x)|u_n|^{r-2}u_n-k(x)|u|^{r-2}u\right] \overline{(u_n-u)}\textrm{d}x. \end{aligned}$$

(3.17)

From Lemma 2.7, we have

$$\begin{aligned}&\int _{\Omega }\Big [({\mathcal {I}}_\mu *F({|u_n|}^2))f({|u_n|}^2)u_n-({\mathcal {I}}_\mu *F(|u|^2))f(|u|^2)u\Big ]\overline{(u_n-u)}\textrm{d}x\rightarrow 0, \nonumber \\&\quad \text{ as }\ n\rightarrow \infty . \end{aligned}$$

(3.18)

Moreover, from (3.15) and the Brezis–Lieb Lemma, we have

$$\begin{aligned} \int _{\Omega }\frac{|u_n-u|^{p^*_\alpha }}{|x|^\alpha }\textrm{d}x=\int _{\Omega }\frac{|u_n|^{p^*_\alpha }}{|x|^\alpha }\textrm{d}x-\int _{\Omega }\frac{|u|^{p^*_\alpha }}{|x|^\alpha }\textrm{d}x+o(1)\rightarrow 0, \quad \text{ as }\ n\rightarrow \infty . \end{aligned}$$

This together with the Hölder inequality implies

$$\begin{aligned} \int _{\Omega }\left[ \frac{|u_n|^{p^*_\alpha -2}u_n}{|x|^\alpha }-\frac{|u|^{p^*_\alpha -2}u}{|x|^\alpha }\right] \overline{(u_n-u)}\, \textrm{d}x\rightarrow 0, \quad \text {as } n\rightarrow \infty . \end{aligned}$$

(3.19)

Since $k\in L^\frac{p^*}{p^*-r}(\Omega , {\mathbb {C}})$, by the Vitali convergence theorem one can deduce that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _{\Omega } k(x)|u_n|^{r}\textrm{d}x=\int _{\Omega } k(x)|u|^{r}\textrm{d}x. \end{aligned}$$

(3.20)

This together with the Brezis–Lieb Lemma yields that

$$\begin{aligned} \int _{\Omega } \left[ k(x)|u_n|^{r-2}u_n-k(x)|u|^{r-2}u\right] \overline{(u_n-u)}\textrm{d}x\rightarrow 0, \quad \text {as } n\rightarrow \infty . \end{aligned}$$

(3.21)

Let us now recall the well-known Simon inequalities. There exist positive numbers $c_p$ and $C_p$, depending only on p, such that

$$\begin{aligned} |\xi -\eta |^{p} \le {\left\{ \begin{array}{ll} c_p (|\xi |^{p-2}\xi -|\eta |^{p-2}\eta )(\xi -\eta ) &{} \text {for } p\ge 2, \\[3pt] C_p\big [(|\xi |^{p-2}\xi -|\eta |^{p-2}\eta ) (\xi -\eta ) \big ]^{\frac{p}{2}}(|\xi |^p+|\eta |^p)^{\frac{2-p}{2}} &{} \text {for }1<p<2, \end{array}\right. }\nonumber \\ \end{aligned}$$

(3.22)

for all $\xi , \eta \in {\mathbb {R}}^N$. Therefore, to the third term on the right hand side of (3.17), we obtain

$$\begin{aligned} B^q_{u_n}(u_n-u)-B^q_{u}(u_n-u)\ge 0. \end{aligned}$$

(3.23)

From (3.16) to (3.23) and $(M_1)$, we obtain

$$\begin{aligned} \lim \limits _{n\rightarrow \infty }M([u_n]^p_{p, A})[B^p_{u_n}(u_n-u)- B^p_{u}(u_n-u)]\le 0. \end{aligned}$$

Since $M([u_n]^p_{p, A})[ B^p_{u_n}(u_n-u)- B^p_{u}(u_n-u)] \le 0$ for all n by convexity and $(M_1)$, we have

$$\begin{aligned} \lim \limits _{n\rightarrow \infty } \Big [B^p_{u_n}(u_n-u)- B^p_{u}(u_n-u)\Big ]\le 0. \end{aligned}$$

(3.24)

According to the Simon inequality, we divide the discussion into two cases.

Case I $p\ge 2$, from (3.22) and (3.24), we have

$$\begin{aligned}&0\le [u_n-u]_{p, A}^p\\&\quad =\iint _{{\mathbb {R}}^{2N}}\frac{|u_n(x)-u(x)-e^{i(x-y)A(\frac{x+y}{p})}u_n(y)+e^{i(x-y)A(\frac{x+y}{p})}u(y)|^p}{|x-y|^{N+ps}}\textrm{d}x\textrm{d}y\\&\quad \le c_p\iint _{{\mathbb {R}}^{2N}}\Big [\frac{|u_n(x) -e^{i(x-y)A(\frac{x+y}{p})}u_n(y)|^{p-2}\left( u_n(x) -e^{i(x-y)A(\frac{x+y}{p})}u_n(y)\right) }{|x-y|^{N+ps}}\\&\qquad -\frac{|u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)|^{p-2}\left( u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)\right) }{|x-y|^{N+ps}}\Big ]\nonumber \\&\qquad \times \Big (u_n(x)-u(x)-e^{i(x-y)A(\frac{x+y}{p})}u_n(y)+e^{i(x-y)A(\frac{x+y}{p})}u(y)\Big )\textrm{d}x\textrm{d}y\\&\quad =c_p\Big [B^p_{u_n}(u_n-u)- B^p_{u}(u_n-u)\Big ]\le 0, \quad \text{ as }\ n\rightarrow \infty . \end{aligned}$$

Case II $1<p<2$, taking $\xi =u_n(x)-e^{i(x-y)A(\frac{x+y}{p})}u_n(y)$ and $\eta =u(x)-e^{i(x-y)A(\frac{x+y}{p})}u(y)$ in (3.22), as $n\rightarrow \infty$, we have

$$\begin{aligned} 0\le [u_n-u]_{p, A}^p&\le C_p\big [B^p_{u_n}(u_n-u)-B^p_{u}(u_n-u) \big ]^{\frac{p}{2}}([u_n]_{p, A}^p+[u]_{p, A}^p)^{\frac{2-p}{2}}\\&\le C_p\big [B^p_{u_n}(u_n-u)-B^p_{u}(u_n-u) \big ]^{\frac{p}{2}}([u_n]_{p, A}^{p(2-p)/2}+[u]_{p, A}^{\frac{p(2-p)}{2}})\\&\le C\big [B^p_{u_n}(u_n-u)-B^p_{u}(u_n-u) \big ]^{\frac{p}{2}}\le 0. \end{aligned}$$

Here, we use the fact that $[u_n]_{p, A}$ and $[u]_{p, A}$ are bounded, and the elementary inequality

$$\begin{aligned} (a+b)^{\frac{2-p}{2}}\le a^{\frac{2-p}{2}}+b^{\frac{2-p}{2}}\ \ \text{ for } \text{ all }\ a, b\ge 0\ \text{ and }\ 1<p<2. \end{aligned}$$

In conclusion, $u_n\rightarrow u$ strongly in ${\mathcal {W}}$, the proof is complete. $\square$

Lemma 3.3

If conditions $(M_1)$–$(M_2)$ and $(F_1)$–$(F_3)$ hold, then there exists $\lambda _*> 0$ such that

$$\begin{aligned} c_\lambda =\inf _{\gamma \in \Gamma }\max _{0\le t\le 1}{\mathcal {J}}_\lambda (\gamma (t))<\left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) (m_0 S_\alpha )^\frac{p^*_\alpha }{p^*_\alpha -p}, \end{aligned}$$

(3.25)

for all $\lambda \ge \lambda _*$, where $\Gamma =\{\gamma \in C([0, 1], {\mathcal {W}}):\gamma (0)=0, \gamma (1)=e\}.$

Proof

We choose $v_0\in {\mathcal {W}}$ with $[v_0]_{p, A}=1$. Then, $\lim _{t\rightarrow 0}{\mathcal {J}}_\lambda (t v_0)=0$ and $\lim _{t\rightarrow \infty }{\mathcal {J}}_\lambda (t v_0)=-\infty$, and then, there exists $t_\lambda >0$ such that $\sup _{t\ge 0}{\mathcal {J}}_\lambda (t v_0)={\mathcal {J}}(t_\lambda v_0)$. Hence, $t_\lambda$ satisfies

$$\begin{aligned} M([t_\lambda v_0]^p_{p, A})[t_\lambda v_0]^p_{p, A}+[t_\lambda v_0]_{q,A}^{q}&= \int _{\Omega }\int _{\Omega } \frac{F(|t_\lambda v_0|^2)}{|x-y|^\mu }f(|t_\lambda v_0|^2)|t_\lambda v_0|^2\, \textrm{d}x\, \textrm{d}y\nonumber \\&\quad +\int _{\Omega }\frac{{|t_\lambda v_0|}^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x\nonumber \\&\quad +\lambda \int _{\Omega } k(x)|t_\lambda v_0|^{r}\textrm{d}x. \end{aligned}$$

(3.26)

By $(M_2)$ and $(F_3)$, we have

$$\begin{aligned}&\theta {\mathscr {M}}([t_\lambda v_0]^p_{p, A})+[t_\lambda v_0]^q_{q, A} \ge M([t_\lambda v_0]^p_{p, A})[t_\lambda v_0]^p_{p, A}+[t_\lambda v_0]^q_{q, A}\nonumber \\&\quad = \int _{\Omega }\int _{\Omega } \frac{F(|t_\lambda v_0|^2)}{|x-y|^\mu }f(|t_\lambda v_0|^2)|t_\lambda v_0|^2\, \textrm{d}x\, \textrm{d}y\nonumber \\&\qquad +\int _{\Omega }\frac{{|t_\lambda v_0|}^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x+\lambda \int _{\Omega } k(x)|t_\lambda v_0|^{r}\textrm{d}x\nonumber \\&\quad \ge \frac{ \kappa }{4} \int _{\Omega }\int _{\Omega } \frac{F(|t_\lambda v_0 (x)|^2)F(|t_\lambda v_0 (y)|^2)}{|x-y|^\mu }\, \textrm{d}x\, \textrm{d}y\nonumber \\&\qquad +t^{p^*_\alpha }_\lambda \int _{\Omega }\frac{{|v_0|}^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x+\lambda \int _{\Omega } k(x)|t_\lambda v_0|^{r}\textrm{d}x\nonumber \\&\quad \ge t^{p^*_\alpha }_\lambda \int _{\Omega }\frac{{| v_0|}^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x. \end{aligned}$$

(3.27)

Without loss of generality, we assume that $t_\lambda \ge 1$ for all $\lambda >0$. Using $(M_2)$ again, (3.27) gives that

$$\begin{aligned} t^{p^*_\alpha }_\lambda \int _{\Omega }\frac{{|v_0|}^{p^*_\alpha }}{|x|^\alpha }\, \textrm{d}x\le \theta {\mathscr {M}}(1)t^{p\theta }_\lambda +{t^q_\lambda }[v_0]^q_{q, A}, \end{aligned}$$

with $q<\theta p<p^{*}_\alpha$, thus $\{t_\lambda \}$ is bounded. Thus, there exists $t_0>0$ and a sequence $\lambda _n\rightarrow \infty$ as $n\rightarrow \infty$ such that

$$\begin{aligned} t_{\lambda _n}\rightarrow t_0\quad \text {as } n\rightarrow \infty . \end{aligned}$$

By Lemma 2.6 and the Lebesgue dominated convergence theorem, we get

$$\begin{aligned}&\int _{\Omega }\int _{\Omega } \frac{F(|t_{\lambda _n} v_0(y)|^2)}{|x-y|^\mu }f(|t_{\lambda _n} v_0(x)|^2)|t_{\lambda _n} v_0(x)|^2\, \textrm{d}x\,\textrm{d}y\\&\quad \rightarrow \int _{\Omega }\int _{\Omega } \frac{F(|t_0 v_0(y)|^2)}{|x-y|^\mu }f(|t_0 v_0(x)|^2)|t_0 v_0(x)|^2\, \textrm{d}x\, \textrm{d}y, \end{aligned}$$

as $n\rightarrow \infty$. Thus

$$\begin{aligned} \lambda _n\int _{\Omega } k(x)|t_{\lambda _n} v_0(x)|^{r}\textrm{d}x \rightarrow \infty , \quad \text {as } n\rightarrow \infty . \end{aligned}$$

Hence, (3.26) implies that $M([t_0 v_0]^p_{p, A})[t_0 v_0]^p_{p, A}+[t_0 v_0]^q_{q, A}=\infty .$ This is impossible. Thus, $t_\lambda \rightarrow 0$ as $\lambda \rightarrow \infty$. From (3.26) again, we have

$$\begin{aligned} \lim _{\lambda \rightarrow \infty }\lambda \int _{\Omega } k(x)|t_{\lambda } v_0(x)|^{r}\textrm{d}x =0. \end{aligned}$$

This together with $(F_3)$ implies that

$$\begin{aligned} \lim _{\lambda \rightarrow \infty }\int _{\Omega }\int _{\Omega } \frac{F\left( |t_\lambda v_0 (x)|^2\right) F\left( |t_\lambda v_0 (y)|^2\right) }{|x-y|^\mu }\, \textrm{d}x\, \textrm{d}y=0. \end{aligned}$$

Therefore,

$$\begin{aligned} \lim _{\lambda \rightarrow \infty }\sup _{t\ge 0}{\mathcal {J}}_\lambda (t v_0)=\lim _{\lambda \rightarrow \infty }{\mathcal {J}}_\lambda (t_\lambda v_0)=0. \end{aligned}$$

Then, there exists $\lambda _*>0$ such that for any $\lambda \ge \lambda _*$,

$$\begin{aligned} \sup _{t\ge 0}{\mathcal {J}}_\lambda (t v_0)\le \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) (m_0 S_\alpha )^{\frac{p^*_\alpha }{p^*_\alpha -p}}. \end{aligned}$$

Taking $e=T v_0$ with T large enough to verify ${J}_\lambda (e)<0$, we obtain $c_\lambda \le \max _{t\in [0, 1]}{\mathcal {J}}_\lambda (\gamma (t)),$ with $\gamma (t)=tT {v_0}$. Therefore

$$\begin{aligned} c_\lambda \le \max _{t\in [0, 1]}{\mathcal {J}}_\lambda (\gamma (t))\le \left( \frac{1}{p\theta }-\frac{1}{p^*_\alpha }\right) (m_0 S_\alpha )^{\frac{p^*_\alpha }{p^*_\alpha -p}}, \end{aligned}$$

for $\lambda$ large enough. Thus (3.25) holds. $\square$

Proof of Theorem 1.1

Lemmas 3.1–3.3 and the mountain pass theorem guarantee that there exists $\lambda _*>0$ such that functional ${\mathcal {J}}_\lambda$ has a critical point for all $\lambda \ge \lambda _*$, so u is a solution of (1.1) with ${\mathcal {J}}_\lambda (u)>0$. $\square$